[ 
https://issues.apache.org/jira/browse/SUREFIRE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118455#comment-15118455
 ] 

Sean Flanigan commented on SUREFIRE-1219:
-----------------------------------------

(Wow, I haven't used that [~seanf] account in years. I'd like to delete it, but 
that email address is dead and my password isn't accepted.)

[~tibor17], I certainly don't want to wait until all 100 tests have finished to 
implement fail-fast, or I wouldn't have submitted this bug!  In fact, it's when 
you are forced to use rerun that fail-fast becomes even more important. 
Otherwise your test times can double or triple when there is a common root 
cause breaking every test.

I'm sorry Tibor, but I'm finding your explanations very difficult to understand.

I don't understand what it is about forking and 
surefire-junit4/surefire-junit47 which prevents us from incrementing the 
counter only when a test has failed _all its runs_ (instead of each time there 
is a failure).

If we can only get the right behaviour in some cases, I think that's more 
important than having the same (wrong IMHO) behaviour in all cases.  Even if we 
risk running an extra couple of unnecessary tests due to concurrency issues 
when the fail counter is triggered, that's got to be better than running all 
the tests, or turning off the rerun feature.  But I haven't read that code, so 
I can't really suggest anything concrete.  I gather that this all has to do 
with the fact that there is currently a separate rerun phase after the main 
testing phase, so it might take a lot of refactoring to make the sort of change 
I have in mind.

In any case, what's implemented now is not good, because even though 
surefire-junit4 now in fact does run flaky tests multiple times, the XML file 
does not reflect this - it only reports the first run (a failure), and ignores 
the second run (which passed).  (This is inferred from the logging in my real 
project's tests.)

Not to mention the fact that the description here: 
https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
 doesn't seem to describe what actually happens when both options are set (as I 
said in the original report above). 

I have had to disable the fail-fast feature to get flaky reruns to work as they 
should, which is a real shame because these two options would really complement 
each other.


> skipAfterFailureCount should not count flaky tests as failures when using 
> rerunFailingTestsCount
> ------------------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1219
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1219
>             Project: Maven Surefire
>          Issue Type: Bug
>    Affects Versions: 2.19.1
>            Reporter: Sean Flanigan
>         Attachments: SUREFIRE-1219-tests.zip
>
>
> According to 
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
>  "failed tests within re-run phase are not included in 
> skipAfterFailureCount". For instance, if skipAfterFailureCount is 1, but 
> every test passes on the second attempt, the tests are "flaky" but not 
> "failures".  This should be reflected in summaries, in XML result files and 
> in fail fast (skipAfterFailureCount), but this is not the case.
> If I have 10 flaky-but-not-failing tests, set rerunFailingTestsCount=1 and 
> skipAfterFailureCount=5:
> 1. I would expect something like this to be printed on the console:
>   Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Flakes: 10
> 2. The XML report should list <flakyFailure>s but no <failures>.
> 3. All tests should be run, not just 5 of them, in other words 
> skipAfterFailureCount should not count the flaky tests as part of the failure 
> count.
> I have some tests demonstrating the problem, which I will attach to this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to