On 03/06/2025 3:09 pm, Andrew Cooper wrote: > On 03/06/2025 2:44 pm, Marek Marczykowski-Górecki wrote: >> On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote: >>> On 03/06/2025 1:42 pm, Anthony PERARD wrote: >>>> From: Anthony PERARD <anthony.per...@vates.tech> >>>> >>>> We can't rely on an exit value from `run-tools-tests` since we only >>>> have the console output. `console.exp` only look for success or it >>>> times out. We could parse the console output, but the junit is more >>>> concise. Also check if we have it or fail as well. >>>> >>>> Signed-off-by: Anthony PERARD <anthony.per...@vates.tech> >>>> --- >>>> automation/scripts/qubes-x86-64.sh | 7 +++++++ >>>> 1 file changed, 7 insertions(+) >>>> >>>> diff --git a/automation/scripts/qubes-x86-64.sh >>>> b/automation/scripts/qubes-x86-64.sh >>>> index 046137a4a6..7a4c5ae489 100755 >>>> --- a/automation/scripts/qubes-x86-64.sh >>>> +++ b/automation/scripts/qubes-x86-64.sh >>>> @@ -298,6 +298,13 @@ TEST_RESULT=$? >>>> >>>> if [ -n "$retrieve_xml" ]; then >>>> nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null >>>> + # Findout if one of the test failed >>>> + if ! grep -q '</testsuites>' tests-junit.xml; then >>>> + echo "ERROR: tests-junit.xml is incomplete or missing." >>>> + TEST_RESULT=1 >>>> + elif grep -q '</failure>' tests-junit.xml; then >>>> + TEST_RESULT=1 >>>> + fi >>>> fi >>>> >>>> exit "$TEST_RESULT" >>> A couple of things. >>> >>> From my experimentation with junit, >>> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug >>> we can also use </error> for classification. I'm also very disappointed >>> in Gitlab classifying <warning> as success. >>> >>> Not for this patch, but for XTF I need to be able to express "tolerable >>> failure". (All branches of Xen will run the same tests, and we don't >>> have OSSTest to deem "fail never passed" as non-blocking.) >>> >>> Even if the job passes overall, I want tolerable failures to show up in >>> the UI, so I have to use <failure> in junit.xml. But that means needing >>> to be more selective, and I don't have a good idea of how to do this. >>> (I have one terrible idea, which is </failure type=tolerable"> which >>> will escape that grep, but it feels like (ab)buse of XML.) >> But that automation/ dir (including the run-tools-tests script) is >> per-branch, so you can specify there which tests should be considered >> failure and which just warning, no? It will require few more bits in the >> script, but fundamentally shouldn't be a problem? >> > XTF is in a separate repo and does not have branches. > > Now consider > https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=d965e2ee07c56c341d8896852550914d87ea5374, > and testing it. > > Anything I put into XTF to test it will pass on staging but fail on the > older stable-* trees. In due course it will get backported to the > bugfix branches, but it likely won't get fixed on the security-only > branches. > > Furthermore, I need to not change xen.git to make this work, and older > branches need to pick up newer XTF automatically. (XSA tests *are* > expected to pass everywhere once the issue is public.) > > My current plan is to have logic of the form: > > if ( xenver >= $STAGING ) > xtf_failure(...); > else > xtf_success("Expected Failure:" ...); > > where $STAGING moves when backports get done. I still want the failures > to show up in the Tests UI.
P.S. I'm planning to teach ./xtf-runner how to write out a junit.xml directly. I'm not interested in interpreting python's stdout and writing xml in shell... ~Andrew
