[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cam Morris updated SUREFIRE-1228:
---------------------------------
Description:
Surefire confuses tests run with a data-provider with rerun tests so only one
of the data-provided tests need to pass.
Steps to reproduce:
# Create a testNG test with a data provider that passes with some of the data
and fails with others
{code:title=TestNG example|language=java}
@Test(dataProvider = "succeed")
public void fail(boolean succeed) {
if (!succeed) {
Assert.fail("nope");
}
}
@DataProvider(name="succeed")
public Object[][] arrayIndex() {
return new Object[][]{ {true}, {false}};
}
{code}
# Configure surefire to rerun failed tests
{code:title=pom.xml example|language=xml}
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
<modelVersion>4.0.0</modelVersion>
<groupId>cam.test</groupId>
<artifactId>testng-retry</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.9.10</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<rerunFailingTestsCount>1</rerunFailingTestsCount>
</configuration>
</plugin>
</plugins>
</build>
</project>
{code}
- *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
pass.
- *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes:
1" The rerun confuses test runs with different data for retry attempts. Just
one of the data provider tests needs to succeed for the test to be deemed a
success.
{code}mvn test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestRandomFail
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
FAILURE! - in TestRandomFail
fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
java.lang.AssertionError: nope
at TestRandomFail.fail(TestRandomFail.java:14)
Results :
Flaked tests:
TestRandomFail.fail(TestRandomFail)
Run 1: PASS
Run 2: TestRandomFail.fail:14 nope
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.251 s
[INFO] Finished at: 2016-02-10T07:39:21-07:00
[INFO] Final Memory: 9M/244M
[INFO] ------------------------------------------------------------------------
{code}
was:
Surefire confuses tests run with a data-provider with rerun tests so only one
of the data-provided tests need to pass.
Steps to reproduce:
# Create a testNG test with a data provider that passes with some of the data
and fails with others
{code:title=TestNG example|language=java}
@Test(dataProvider = "succeed")
public void fail(boolean succeed) {
if (!succeed) {
Assert.fail("nope");
}
}
@DataProvider(name="succeed")
public Object[][] arrayIndex() {
return new Object[][]{ {true}, {false}};
}
{code}
# Configure surefire to rerun failed tests
{code:title=pom.xml example|language=xml}
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
<modelVersion>4.0.0</modelVersion>
<groupId>cam.test</groupId>
<artifactId>testng-retry</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.9.9</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<rerunFailingTestsCount>1</rerunFailingTestsCount>
</configuration>
</plugin>
</plugins>
</build>
</project>
{code}
- *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
pass.
- *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes:
1" The rerun confuses test runs with different data for retry attempts. Just
one of the data provider tests needs to succeed for the test to be deemed a
success.
{code}mvn test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestRandomFail
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
FAILURE! - in TestRandomFail
fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
java.lang.AssertionError: nope
at TestRandomFail.fail(TestRandomFail.java:14)
Results :
Flaked tests:
TestRandomFail.fail(TestRandomFail)
Run 1: PASS
Run 2: TestRandomFail.fail:14 nope
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.251 s
[INFO] Finished at: 2016-02-10T07:39:21-07:00
[INFO] Final Memory: 9M/244M
[INFO] ------------------------------------------------------------------------
{code}
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> -------------------------------------------------------------------
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
> Affects Versions: 2.19.1
> Reporter: Cam Morris
> Attachments: TestRandomFail.java, testng-retry.iml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
> <?xml version="1.0" encoding="UTF-8"?>
> <project xmlns="http://maven.apache.org/POM/4.0.0";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
> <modelVersion>4.0.0</modelVersion>
> <groupId>cam.test</groupId>
> <artifactId>testng-retry</artifactId>
> <version>1.0-SNAPSHOT</version>
> <dependencies>
> <dependency>
> <groupId>org.testng</groupId>
> <artifactId>testng</artifactId>
> <version>6.9.10</version>
> <scope>test</scope>
> </dependency>
> </dependencies>
> <build>
> <plugins>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.19.1</version>
> <configuration>
> <rerunFailingTestsCount>1</rerunFailingTestsCount>
> </configuration>
> </plugin>
> </plugins>
> </build>
> </project>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
> ------------------------------------------------------------------------
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
