On 03/06/2025 1:42 pm, Anthony PERARD wrote: > From: Anthony PERARD <anthony.per...@vates.tech> > > We can't rely on an exit value from `run-tools-tests` since we only > have the console output. `console.exp` only look for success or it > times out. We could parse the console output, but the junit is more > concise. Also check if we have it or fail as well. > > Signed-off-by: Anthony PERARD <anthony.per...@vates.tech> > --- > automation/scripts/qubes-x86-64.sh | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/automation/scripts/qubes-x86-64.sh > b/automation/scripts/qubes-x86-64.sh > index 046137a4a6..7a4c5ae489 100755 > --- a/automation/scripts/qubes-x86-64.sh > +++ b/automation/scripts/qubes-x86-64.sh > @@ -298,6 +298,13 @@ TEST_RESULT=$? > > if [ -n "$retrieve_xml" ]; then > nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null > + # Findout if one of the test failed > + if ! grep -q '</testsuites>' tests-junit.xml; then > + echo "ERROR: tests-junit.xml is incomplete or missing." > + TEST_RESULT=1 > + elif grep -q '</failure>' tests-junit.xml; then > + TEST_RESULT=1 > + fi > fi > > exit "$TEST_RESULT"
A couple of things. >From my experimentation with junit, https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug we can also use </error> for classification. I'm also very disappointed in Gitlab classifying <warning> as success. Not for this patch, but for XTF I need to be able to express "tolerable failure". (All branches of Xen will run the same tests, and we don't have OSSTest to deem "fail never passed" as non-blocking.) Even if the job passes overall, I want tolerable failures to show up in the UI, so I have to use <failure> in junit.xml. But that means needing to be more selective, and I don't have a good idea of how to do this. (I have one terrible idea, which is </failure type=tolerable"> which will escape that grep, but it feels like (ab)buse of XML.) ~Andrew
