On 03/06/2025 4:58 pm, Anthony PERARD wrote: > On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote: >> On 03/06/2025 1:42 pm, Anthony PERARD wrote: >>> if [ -n "$retrieve_xml" ]; then >>> nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null >>> + # Findout if one of the test failed >>> + if ! grep -q '</testsuites>' tests-junit.xml; then >>> + echo "ERROR: tests-junit.xml is incomplete or missing." >>> + TEST_RESULT=1 >>> + elif grep -q '</failure>' tests-junit.xml; then >>> + TEST_RESULT=1 >>> + fi >>> fi >>> >>> exit "$TEST_RESULT" >> A couple of things. >> >> From my experimentation with junit, >> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug >> we can also use </error> for classification. I'm also very disappointed >> in Gitlab classifying <warning> as success. > According to the documentation [1] which point to this junit xml format [2] > the only elements (and path) are: > testsuites.testsuite.testcase.failure > "error" or "warning" don't exist. > There's the attributes `type` in <failure> but this isn't explained how > it's used. > > But I guess if we follow the link in [2], go through web.archive.org, we > can find [3] which has "skipped", "error", "failure", but still no > "warning". > > > [1] > https://docs.gitlab.com/ci/testing/unit_test_reports/#unit-test-reporting-workflow > [2] > https://www.ibm.com/docs/en/developer-for-zos/16.0?topic=formats-junit-xml-format > [3] https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd
I was also very disappointed with the documentation, which seems to be almost non-existent. But after some source diving: https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/ci/parsers/test/junit.rb#L69-83 So anything which isn't recognised is deemed to be success. Lovely :( I'm also still none the wiser as to why Skip's system_out is a formatted json blob, unlike the others. > > >> Not for this patch, but for XTF I need to be able to express "tolerable >> failure". (All branches of Xen will run the same tests, and we don't >> have OSSTest to deem "fail never passed" as non-blocking.) > According to [1], there's a notion of "Existing failures", but that > might show up only on merge request. > >> Even if the job passes overall, I want tolerable failures to show up in >> the UI, so I have to use <failure> in junit.xml. But that means needing >> to be more selective, and I don't have a good idea of how to do this. >> (I have one terrible idea, which is </failure type=tolerable"> which >> will escape that grep, but it feels like (ab)buse of XML.) > At the moment, `run-tools-tests` write '<failure type="failure"' on > failure, so we could grep on that instead event if it is sligtly more > fragile. I've choosen to grep on '</failure>' at first because that's > much less likely to be written differently, while the attributes in the > tag could be written in a different order. > > Then, we can always use `sed` and extract the "type" to check it: > sed -n 's/.*<failure \(\|.* \)type="\([^"]\+\)" .*/\2/p' tests-junit.xml | > while read type; do > case $type in > failure) echo fail;; > tolerable) echo ok;; > *) echo error unknwon type $type;; > esac > done > But that maybe going a bit too far :-) There's xq which might be able to do this more nicely, and is packaged in alpine. ~Andrew
