Javier A. Ortiz created SUREFIRE-2180:
-----------------------------------------
Summary: Surefire not invoking tests with TestNG DataProvider
Key: SUREFIRE-2180
URL: https://issues.apache.org/jira/browse/SUREFIRE-2180
Project: Maven Surefire
Issue Type: Bug
Components: TestNG support
Affects Versions: 3.1.2
Reporter: Javier A. Ortiz
I'm having a weird issue I can't figure out. It's driving me insane.
I have a Maven project for which I'm running the same command both locally and
on Jenkins. The environment is the same as far as I can tell:
- Maven 3.9.2 (On Jenkins: Docker image maven:3.9.2-eclipse-temurin-11)
- JDK 11 (On Jenkins: Docker image maven:3.9.2-eclipse-temurin-11)
- TestNG 7.8.0
Command:
{code}
mvn -Ddependency.surefire.verbose=10 -Pcoverage verify -B -U -Dgroup=PLATFORM
{code}
Relevant POM section:
{code}
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.10</version>
</plugin>
...
</plugins>
</pluginManagement>
...
</build>
...
<profiles>
<profile>
<id>coverage</id>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>com/XXX/framework/dataproviders/**/*</exclude>
<exclude>com/XXX/v2/basetest/**/*</exclude>
<exclude>**/*Exception*</exclude>
</excludes>
</configuration>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<!-- attached to Maven test phase -->
<execution>
<id>report</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
<!-- Add this checking -->
<execution>
<id>jacoco-check</id>
<goals>
<goal>check</goal>
</goals>
<configuration>
<rules>
<rule>
<element>BUNDLE</element>
<limits>
<limit>
<counter>INSTRUCTION</counter>
<value>COVEREDRATIO</value>
<minimum>60%</minimum>
</limit>
<limit>
<counter>CLASS</counter>
<value>MISSEDCOUNT</value>
<maximum>0</maximum>
</limit>
</limits>
</rule>
<rule>
<element>PACKAGE</element>
<limits>
<limit>
<counter>LINE</counter>
<value>COVEREDRATIO</value>
<minimum>60%</minimum>
</limit>
</limits>
</rule>
</rules>
<skip>${skipTests}</skip>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
{code}
One of the tests that have the issue (only tests with DataProviders) show this
pattern in the logs. Enabled verbose to level 10 in hopes that something would
show up.
Locally:
{code}
[2023-06-06 16:55:46.795] [INFO] [TestClass] Creating TestClass for [ClassImpl
class=com.XXX.framework.utils.TimeUtilsTest]
[2023-06-06 16:55:46.796] [INFO] Method public java.lang.Object[][]
com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation but
also a return value: ignoring it. Use <suite allow-return-values="true"> to fix
this
[2023-06-06 16:55:46.797] [INFO] [TestClass] Adding method
TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest
{code}
...
{code}
[2023-06-06 16:55:47.316] [INFO] ===== Test class
com.XXX.framework.utils.TimeUtilsTest
[2023-06-06 16:55:47.316] [INFO] @Test
TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
instance:com.XXX.framework.utils.TimeUtilsTest@283eb984]
[2023-06-06 16:55:47.316] [INFO] ======
{code}
...
{code}
[TestNG] INVOKING: "Surefire test" -
com.XXX.framework.utils.TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)(value(s):
NANOSECONDS, 60, "60ns", TestNameParameter(customName=60 NANOSECONDS))
{code}
...
Test executes and results, etc.
On Jenkins:
{code}
[2023-06-06 21:02:29.270] [INFO] [TestClass] Creating TestClass for [ClassImpl
class=com.XXX.framework.utils.TimeUtilsTest]
[2023-06-06 21:02:29.271] [INFO] Method public java.lang.Object[][]
com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation but
also a return value: ignoring it. Use <suite allow-return-values="true"> to fix
this
[2023-06-06 21:02:29.271] [INFO] [TestClass] Adding method
TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest
{code}
...
{code}
[2023-06-06 21:02:29.529] [INFO] ===== Test class
com.XXX.framework.utils.TimeUtilsTest
[2023-06-06 21:02:29.529] [INFO] @Test
TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
instance:com.XXX.framework.utils.TimeUtilsTest@52f9a620]
[2023-06-06 21:02:29.529] [INFO] ======
{code}
I see no other references about the test class in the rest of the trace.
Basically, all tests with a DataProvider don't run.
Maybe someone has a suggestion on what else to look for.
I've tried the following:
- Making sure the proper Test annotation is used (TestNG vs. JUnit
- Remove test groups or any other listener that could be interfering.
- Downgrade the TestNG version.
- Pray.
Here's the investigation on the TestNG side:
https://github.com/testng-team/testng/issues/2924
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
