|
||||||||
|
This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira |
||||||||
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
Hi Bruno,
something (probably on my side) seems to have changed recently. The original issue I was talking about is as follows: If you go to [1] and search for "test-results/t0061-run-command.sh" on that web page you'll see that in the table for test "t0061-run-command" subtest number 2 is missing. Instead, you see a number of comments in the table. But if you look at the slightly more recent test run at [2] there suddenly is subtest number 2, called "run_command can run a command", in the table. I'm not sure what caused the change.
I think you shoud simply ignore all lines
Anyway, even when looking at the (currently) most recent build at [3] there are a number of issues / inconsistencies:
1) That page contains the string "TAP Test Result (130 failures null)", not sure why "null" is in there. However, the total number of 130 failed tests is correct, that's the same number I'm calculating in my Groovy post-build script. The Groovy script also generates the output below "The following test(s) failed" on that page, which is my reference for the list of failing tests.
2) Slightly below, there's the text "This build contains 628 TAP test sets. Click here for details" and "1 parse error(s)". Again I'm not sure what causes the parse error, whether it's a bug in the TAP plugin or a bug it Git's test suite output. In any case it would be nice to see the name of the (sub-)test where the error occurred here.
3) The "TAP Test Result" from 1) and the "Click here for details" from 2) link to [4] and [5] respectively. At first I was confused that the TAP plugin provides two detailed results pages, and I'm still not sure whether this makes sense.
4) On [4], clicking on any if the ">>>" or test links under "All Failed Tests" results in "Status Code: 404". Also, the list of "All Failed Tests" does not seem to be correct as it includes the tests marked as "TODO known breakage". Maybe you could add an option to ignore results of tests marked as TODO? I'm also a bit confused why our total failed test count of 130 matches mine, although you count "TODO known breakage" as failures and I don't.
5) On [5], test marked as "TODO known breakage" are also marked red, i.e. as failures, but again the total number of failed tests oddly is correct. But the number of skipped tetsts is not. The header says "628 files 9549 tests, 8967 ok, 130 not ok, 1 skipped, 0 Bail Out!", but as you can see from the yellow entries in the table there are far more than 1 skipped tests. Also, whenever there's a comment in the raw TAP file, you make it part of the table. I think the plugin should simply ignore lines starting with "#" in the input files and not include them into the table. A good example on that page is at "test-results/t5801-remote-helpers.sh" where you see a lot of comments which should not be visible at all, IMHO. The bottom of the page finally contains the details about parse errors, which is great. Maybe the "1 parse error(s)" from the overview page could link here? In any case I like this result page much better than [4], and I'd like to suggest to completely drop [4] and just fix the issues with [5] instead. This will also lead to less confusion for the user which TAP result page to look at.
I hope all of this does not overwhelm you![]()
Thank in advance for any fixes!
[1] https://qa.nest-initiative.org/view/msysGit/job/msysgit-mingwGitDevEnv-test/96/tapResults/
[2] https://qa.nest-initiative.org/view/msysGit/job/msysgit-mingwGitDevEnv-test/99/tapResults/
[3] https://qa.nest-initiative.org/view/msysGit/job/msysgit-mingwGitDevEnv-test/125/
[4] https://qa.nest-initiative.org/view/msysGit/job/msysgit-mingwGitDevEnv-test/125/tapTestReport/
[5] https://qa.nest-initiative.org/view/msysGit/job/msysgit-mingwGitDevEnv-test/125/tapResults/