[ 
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505780#comment-15505780
 ] 

Julien Herr commented on SUREFIRE-1228:
---------------------------------------

I don't know how rerunFailingTestsCount is implemented in the testng provider 
(maybe it is not) but testng provides a similar feature out of the box named 
RetryAnaylzer. 
A workaround could be to replace rerunFailingTestsCount usage by it,  
rerunFailingTestsCount could use it. 
It is possible to rerun only a part of data provider tests but the feature was 
broken for a while and I fixed it few days/weeks ago (should be available in 
the latest version) 

> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> -------------------------------------------------------------------
>
>                 Key: SUREFIRE-1228
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: TestNG support
>    Affects Versions: 2.19.1
>            Reporter: Cam Morris
>            Assignee: Tibor Digana
>         Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one 
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data 
> and fails with others
> {code:title=TestNG example|language=java}
>   @Test(dataProvider = "succeed")
>   public void fail(boolean succeed) {
>     if (!succeed) {
>       Assert.fail("nope");
>     }
>   }
>   @DataProvider(name="succeed")
>   public Object[][] arrayIndex() {
>     return new Object[][]{ {true}, {false}};
>   }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
> <?xml version="1.0" encoding="UTF-8"?>
> <project xmlns="http://maven.apache.org/POM/4.0.0";
>          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
>     <modelVersion>4.0.0</modelVersion>
>     <groupId>cam.test</groupId>
>     <artifactId>testng-retry</artifactId>
>     <version>1.0-SNAPSHOT</version>
>     <dependencies>
>         <dependency>
>             <groupId>org.testng</groupId>
>             <artifactId>testng</artifactId>
>             <version>6.9.10</version>
>             <scope>test</scope>
>         </dependency>
>     </dependencies>
>     <build>
>         <plugins>
>             <plugin>
>                 <groupId>org.apache.maven.plugins</groupId>
>                 <artifactId>maven-surefire-plugin</artifactId>
>                 <version>2.19.1</version>
>                 <configuration>
>                   <rerunFailingTestsCount>1</rerunFailingTestsCount>
>                 </configuration>
>             </plugin>
>         </plugins>
>     </build>
> </project>
> {code}
> - *Expected results*:  "Tests run: 2, Failures: 1..."  The tests should not 
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, 
> Flakes: 1"  The rerun confuses test runs with different data for retry 
> attempts.  Just one of the data provider tests needs to succeed for the test 
> to be deemed a success.
> {code}mvn test
> ...
> -------------------------------------------------------
>  T E S T S
> -------------------------------------------------------
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< 
> FAILURE! - in TestRandomFail
> fail(TestRandomFail)  Time elapsed: 0.007 sec  <<< FAILURE!
> java.lang.AssertionError: nope
>         at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests: 
> TestRandomFail.fail(TestRandomFail)
>   Run 1: PASS
>   Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO] 
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] 
> ------------------------------------------------------------------------
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO] 
> ------------------------------------------------------------------------
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to