[
https://issues.apache.org/jira/browse/YETUS-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277536#comment-16277536
]
Allen Wittenauer commented on YETUS-587:
----------------------------------------
So some background on this:
During The Great HDFS Causing Build Node Crashes of 2017, it was discovered
that surefire doesn't report any data on any tests that it doesn't run. For
example, if surefire crashes before the mvn test is complete, then all of the
tests that were to be run post-crash don't effectively disappear.
>From a UI perspective, this is disastrous if any of the previous tests have
>failed. Users are led to believe that the only tests that failed are those
>that were reported.
FWIW, I filed SUREFIRE-1447 as to get a means to ask surefire how many tests
were expected to run.
Playing around with the idea presented above, I had a lot of trouble matching
the actual totals. However, it does provide a pretty good estimate for a
floor value. If we don't reach that floor value, we should throw out all of
the results and declare the whole test failed rather than reporting any other
results. This will cause end users to actually look at the log to see how bad
things are.
> Report if junit drops a significant amount of tests
> ---------------------------------------------------
>
> Key: YETUS-587
> URL: https://issues.apache.org/jira/browse/YETUS-587
> Project: Yetus
> Issue Type: New Feature
> Components: Test Patch
> Reporter: Allen Wittenauer
> Priority: Blocker
> Fix For: 0.7.0
>
>
> I believe precommit should be able to guesstimate how many tests results are
> expected from junit by taking the number @test lines subtracted by the number
> of @ignore lines. Precommit can then compare that to the actual numbers
> received. If that number is below some configurable percentage, precommit
> should throw a very significant warning probably in lieu of reporting the
> actual failed tests since things almost certainly went very very wrong.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
