Rob Savoye wrote:
On 10/31/18 7:34 AM, Jacob Bachmeyer wrote:
While browsing the DejaGnu sources, I noticed that sum2junit.sh appears
to miscount failing tests due to a possible typo on line 49. I am not
sufficiently familiar with JUnit to say with certainty or to be sure of
a correct patch, but this patch at least makes the code and comment
consistent.
I'm traveling right now, but as the author of that contrib script, it
looks ok. I can test it when I get home in a few days. That script is
for importing test results into Jenkins for CI, and not core DejaGnu
infrastructure.
The apparent typo fixed in that patch would only affect the summary
counts generated on the "testsuite" tag. Passes and expected failures
are counted as passed tests, while expected failures and unexpected
passes are counted as failures. Unexpected failures (DejaGnu result
"FAIL") do not appear to be counted at all. Expected failures (DejaGnu
result "XFAIL") are counted twice. This patch counts FAIL as failure
instead of counting XFAIL as both pass and failure.
Further examination finds another suspicious bit of code on line 93:
testcases are reported using only either "skipped" or "failure" tags,
even for tests that produced a PASS result, which just looks "fishy" to me.
I have not been able to find a DTD or even an actual specification for
this format. I found what appears to be an attempt to document the
format at <URL:http://llg.cubic.org/docs/junit/> but that page is
clearly not authoritative, as it asks for any additional information
anyone might have.
Apparently, JUnit XML does not allow any message to be recorded for a
test that passes? (This would explain the suspicious use of "failure"
tags in the sum2junit script.)
Perhaps Jenkins ignores these details in its XML input? It seems that
the sum2junit script groups test results into faked classes, such that a
failing test appears to be in class "${tool}-FAIL" and a passing test in
class "${tool}-PASS". This will probably work until someone tries to do
aggregation of test results across multiple runs or testing systems in
Jenkins where every test from a DejaGnu testsuite appears to fail if not
skipped. The information is displayed, but any statistics generated in
Jenkins will be wrong.
The best answer would probably be to add support for DejaGnu's own XML
format (which seems to have a far more sensible data model) to Jenkins,
but that is outside the scope of the DejaGnu project and I am not
currently using Jenkins, so I will leave that to someone else.
-- Jacob
_______________________________________________
Bug-dejagnu mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/bug-dejagnu
