On 03/06/2025 2:44 pm, Marek Marczykowski-Górecki wrote: > On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote: >> On 03/06/2025 1:42 pm, Anthony PERARD wrote: >>> From: Anthony PERARD <anthony.per...@vates.tech> >>> >>> We can't rely on an exit value from `run-tools-tests` since we only >>> have the console output. `console.exp` only look for success or it >>> times out. We could parse the console output, but the junit is more >>> concise. Also check if we have it or fail as well. >>> >>> Signed-off-by: Anthony PERARD <anthony.per...@vates.tech> >>> --- >>> automation/scripts/qubes-x86-64.sh | 7 +++++++ >>> 1 file changed, 7 insertions(+) >>> >>> diff --git a/automation/scripts/qubes-x86-64.sh >>> b/automation/scripts/qubes-x86-64.sh >>> index 046137a4a6..7a4c5ae489 100755 >>> --- a/automation/scripts/qubes-x86-64.sh >>> +++ b/automation/scripts/qubes-x86-64.sh >>> @@ -298,6 +298,13 @@ TEST_RESULT=$? >>> >>> if [ -n "$retrieve_xml" ]; then >>> nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null >>> + # Findout if one of the test failed >>> + if ! grep -q '</testsuites>' tests-junit.xml; then >>> + echo "ERROR: tests-junit.xml is incomplete or missing." >>> + TEST_RESULT=1 >>> + elif grep -q '</failure>' tests-junit.xml; then >>> + TEST_RESULT=1 >>> + fi >>> fi >>> >>> exit "$TEST_RESULT" >> A couple of things. >> >> From my experimentation with junit, >> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug >> we can also use </error> for classification. I'm also very disappointed >> in Gitlab classifying <warning> as success. >> >> Not for this patch, but for XTF I need to be able to express "tolerable >> failure". (All branches of Xen will run the same tests, and we don't >> have OSSTest to deem "fail never passed" as non-blocking.) >> >> Even if the job passes overall, I want tolerable failures to show up in >> the UI, so I have to use <failure> in junit.xml. But that means needing >> to be more selective, and I don't have a good idea of how to do this. >> (I have one terrible idea, which is </failure type=tolerable"> which >> will escape that grep, but it feels like (ab)buse of XML.) > But that automation/ dir (including the run-tools-tests script) is > per-branch, so you can specify there which tests should be considered > failure and which just warning, no? It will require few more bits in the > script, but fundamentally shouldn't be a problem? >
XTF is in a separate repo and does not have branches. Now consider https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=d965e2ee07c56c341d8896852550914d87ea5374, and testing it. Anything I put into XTF to test it will pass on staging but fail on the older stable-* trees. In due course it will get backported to the bugfix branches, but it likely won't get fixed on the security-only branches. Furthermore, I need to not change xen.git to make this work, and older branches need to pick up newer XTF automatically. (XSA tests *are* expected to pass everywhere once the issue is public.) My current plan is to have logic of the form: if ( xenver >= $STAGING ) xtf_failure(...); else xtf_success("Expected Failure:" ...); where $STAGING moves when backports get done. I still want the failures to show up in the Tests UI. ~Andrew
