[jira] [Commented] (SUREFIRE-1849) Cucumber/JUnit5 Test Execution Regression in M4 and M5
[
https://issues.apache.org/jira/browse/SUREFIRE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387640#comment-17387640
]
M.P. Korstanje commented on SUREFIRE-1849:
--
So adding {{isRootContainer}} to the conditional does work for Cucumber but for
JUnit Jupiter it results in an additional empty xml report. The things it could
contain are all reported elsewhere.
https://github.com/apache/maven-surefire/pull/385
> Cucumber/JUnit5 Test Execution Regression in M4 and M5
> --
>
> Key: SUREFIRE-1849
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1849
> Project: Maven Surefire
> Issue Type: Bug
>Reporter: Jacques Burns
>Priority: Major
>
> My JUnit 5 Cucumber tests fail to execute properly when in Surefire version
> 3.0.0-M4 and 3.0.0-M5. Downgrading to 3.0.0-M3 solves the issue.
> I've built a small, complete example project demostrating the issue
> [https://github.com/jmathewburns/surefire-cucumber-junit5-sscce|https://github.com/jmathewburns/surefire-cucumber-junit5-sscce].
> You can play with the version property in the `pom.xml` in my example and
> run the `test` goal
>
> In M4 and M5, the Cucumber glue is recognised and the step definitions are
> run, but failed assertions do not affect the outcome of the build. Also, true
> assertions do not count towards the "Tests run" statistic
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (SUREFIRE-1849) Cucumber/JUnit5 Test Execution Regression in M4 and M5
[
https://issues.apache.org/jira/browse/SUREFIRE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387633#comment-17387633
]
M.P. Korstanje edited comment on SUREFIRE-1849 at 7/26/21, 10:27 PM:
-
[~tibordigana] it looks like the {{RunListenerAdapter}} is now making
assumptions about classes. These do not hold for JUnit5. Tests do not have to
be located in classes.
I can't really work out why the {{RunListenerAdapter}} is doing this though. I
guess this is an attempt at squashing the JUnit results tree back into a
class/method report.
If so then I guess
{code:java}
if ( failed || isClass || isTest ){code}
Should be:
{code:java}
if ( failed || isClass || isTest || isRootContainer ){code}
So that the {{runListener.testSetCompleted}} is always invoked for the root
container. But it all looks very adhoc right now. So I can't quite say what
that would break.
[https://github.com/apache/maven-surefire/blob/670dbe49855133947fbb436c97d7aa46e81d4d2c/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java]
was (Author: mpkorstanje):
[~tibordigana] it looks like the {{RunListenerAdapter}} is now making
assumptions about classes. These do not hold for JUnit4. Tests do not have to
be located in classes.
I can't really work out why the {{RunListenerAdapter}} is doing this though. I
guess this is an attempt at squashing the JUnit results tree back into a
class/method report.
If so then I guess
{code:java}
if ( failed || isClass || isTest ){code}
Should be:
{code:java}
if ( failed || isClass || isTest || isRootContainer ){code}
So that the {{runListener.testSetCompleted}} is always invoked for the root
container. But it all looks very adhoc right now. So I can't quite say what
that would break.
[https://github.com/apache/maven-surefire/blob/670dbe49855133947fbb436c97d7aa46e81d4d2c/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java]
> Cucumber/JUnit5 Test Execution Regression in M4 and M5
> --
>
> Key: SUREFIRE-1849
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1849
> Project: Maven Surefire
> Issue Type: Bug
>Reporter: Jacques Burns
>Priority: Major
>
> My JUnit 5 Cucumber tests fail to execute properly when in Surefire version
> 3.0.0-M4 and 3.0.0-M5. Downgrading to 3.0.0-M3 solves the issue.
> I've built a small, complete example project demostrating the issue
> [https://github.com/jmathewburns/surefire-cucumber-junit5-sscce|https://github.com/jmathewburns/surefire-cucumber-junit5-sscce].
> You can play with the version property in the `pom.xml` in my example and
> run the `test` goal
>
> In M4 and M5, the Cucumber glue is recognised and the step definitions are
> run, but failed assertions do not affect the outcome of the build. Also, true
> assertions do not count towards the "Tests run" statistic
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (SUREFIRE-1849) Cucumber/JUnit5 Test Execution Regression in M4 and M5
[
https://issues.apache.org/jira/browse/SUREFIRE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387633#comment-17387633
]
M.P. Korstanje edited comment on SUREFIRE-1849 at 7/26/21, 10:03 PM:
-
[~tibordigana] it looks like the \{{RunListenerAdapter}} is now making
assumptions about classes. I can't really work out why though. I guess this is
an attempt at squashing the JUnit results tree back into a class/method report.
If so then I guess
{code:java}
if ( failed || isClass || isTest ){code}
Should be:
{code:java}
if ( failed || isClass || isTest || isRootContainer ){code}
So that the {{runListener.testSetCompleted}} is always invoked for the root
container. But it all looks very adhoc right now. So I can't quite say what
that would break.
[https://github.com/apache/maven-surefire/blob/670dbe49855133947fbb436c97d7aa46e81d4d2c/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java]
was (Author: mpkorstanje):
[~tibordigana] it looks like the \{{RunListenerAdapter }}is now making
assumptions about classes. I can't really work out why though. I guess this is
an attempt at squashing the JUnit results tree back into a class/method report.
If so then I guess
{code:java}
if ( failed || isClass || isTest ){code}
Should be:
{code:java}
if ( failed || isClass || isTest || isRootContainer ){code}
So that the \{{runListener.testSetCompleted}} is always invoked for the root
container. But it all looks very adhoc right now. So I can't quite say what
that would break.
[https://github.com/apache/maven-surefire/blob/670dbe49855133947fbb436c97d7aa46e81d4d2c/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java]
> Cucumber/JUnit5 Test Execution Regression in M4 and M5
> --
>
> Key: SUREFIRE-1849
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1849
> Project: Maven Surefire
> Issue Type: Bug
>Reporter: Jacques Burns
>Priority: Major
>
> My JUnit 5 Cucumber tests fail to execute properly when in Surefire version
> 3.0.0-M4 and 3.0.0-M5. Downgrading to 3.0.0-M3 solves the issue.
> I've built a small, complete example project demostrating the issue
> [https://github.com/jmathewburns/surefire-cucumber-junit5-sscce|https://github.com/jmathewburns/surefire-cucumber-junit5-sscce].
> You can play with the version property in the `pom.xml` in my example and
> run the `test` goal
>
> In M4 and M5, the Cucumber glue is recognised and the step definitions are
> run, but failed assertions do not affect the outcome of the build. Also, true
> assertions do not count towards the "Tests run" statistic
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (SUREFIRE-1849) Cucumber/JUnit5 Test Execution Regression in M4 and M5
[
https://issues.apache.org/jira/browse/SUREFIRE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387633#comment-17387633
]
M.P. Korstanje edited comment on SUREFIRE-1849 at 7/26/21, 10:03 PM:
-
[~tibordigana] it looks like the {{RunListenerAdapter}} is now making
assumptions about classes. These do not hold for JUnit4. Tests do not have to
be located in classes.
I can't really work out why the {{RunListenerAdapter}} is doing this though. I
guess this is an attempt at squashing the JUnit results tree back into a
class/method report.
If so then I guess
{code:java}
if ( failed || isClass || isTest ){code}
Should be:
{code:java}
if ( failed || isClass || isTest || isRootContainer ){code}
So that the {{runListener.testSetCompleted}} is always invoked for the root
container. But it all looks very adhoc right now. So I can't quite say what
that would break.
[https://github.com/apache/maven-surefire/blob/670dbe49855133947fbb436c97d7aa46e81d4d2c/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java]
was (Author: mpkorstanje):
[~tibordigana] it looks like the \{{RunListenerAdapter}} is now making
assumptions about classes. I can't really work out why though. I guess this is
an attempt at squashing the JUnit results tree back into a class/method report.
If so then I guess
{code:java}
if ( failed || isClass || isTest ){code}
Should be:
{code:java}
if ( failed || isClass || isTest || isRootContainer ){code}
So that the {{runListener.testSetCompleted}} is always invoked for the root
container. But it all looks very adhoc right now. So I can't quite say what
that would break.
[https://github.com/apache/maven-surefire/blob/670dbe49855133947fbb436c97d7aa46e81d4d2c/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java]
> Cucumber/JUnit5 Test Execution Regression in M4 and M5
> --
>
> Key: SUREFIRE-1849
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1849
> Project: Maven Surefire
> Issue Type: Bug
>Reporter: Jacques Burns
>Priority: Major
>
> My JUnit 5 Cucumber tests fail to execute properly when in Surefire version
> 3.0.0-M4 and 3.0.0-M5. Downgrading to 3.0.0-M3 solves the issue.
> I've built a small, complete example project demostrating the issue
> [https://github.com/jmathewburns/surefire-cucumber-junit5-sscce|https://github.com/jmathewburns/surefire-cucumber-junit5-sscce].
> You can play with the version property in the `pom.xml` in my example and
> run the `test` goal
>
> In M4 and M5, the Cucumber glue is recognised and the step definitions are
> run, but failed assertions do not affect the outcome of the build. Also, true
> assertions do not count towards the "Tests run" statistic
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1849) Cucumber/JUnit5 Test Execution Regression in M4 and M5
[
https://issues.apache.org/jira/browse/SUREFIRE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387633#comment-17387633
]
M.P. Korstanje commented on SUREFIRE-1849:
--
[~tibordigana] it looks like the {{RunListenerAdapter }}is now making
assumptions about classes. I can't really work out why though. I guess this is
an attempt at squashing the JUnit results tree back into a class/method report.
If so then I guess
{code:java}
if ( failed || isClass || isTest ){code}
Should be:
{code:java}
if ( failed || isClass || isTest || isRootContainer ){code}
So that the {{runListener.testSetCompleted }}is always invoked for the root
container. But it all looks very adhoc right now. So I can't quite say what
that would break.
https://github.com/apache/maven-surefire/blob/670dbe49855133947fbb436c97d7aa46e81d4d2c/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java
> Cucumber/JUnit5 Test Execution Regression in M4 and M5
> --
>
> Key: SUREFIRE-1849
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1849
> Project: Maven Surefire
> Issue Type: Bug
>Reporter: Jacques Burns
>Priority: Major
>
> My JUnit 5 Cucumber tests fail to execute properly when in Surefire version
> 3.0.0-M4 and 3.0.0-M5. Downgrading to 3.0.0-M3 solves the issue.
> I've built a small, complete example project demostrating the issue
> [https://github.com/jmathewburns/surefire-cucumber-junit5-sscce|https://github.com/jmathewburns/surefire-cucumber-junit5-sscce].
> You can play with the version property in the `pom.xml` in my example and
> run the `test` goal
>
> In M4 and M5, the Cucumber glue is recognised and the step definitions are
> run, but failed assertions do not affect the outcome of the build. Also, true
> assertions do not count towards the "Tests run" statistic
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (SUREFIRE-1849) Cucumber/JUnit5 Test Execution Regression in M4 and M5
[
https://issues.apache.org/jira/browse/SUREFIRE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387633#comment-17387633
]
M.P. Korstanje edited comment on SUREFIRE-1849 at 7/26/21, 9:58 PM:
[~tibordigana] it looks like the \{{RunListenerAdapter }}is now making
assumptions about classes. I can't really work out why though. I guess this is
an attempt at squashing the JUnit results tree back into a class/method report.
If so then I guess
{code:java}
if ( failed || isClass || isTest ){code}
Should be:
{code:java}
if ( failed || isClass || isTest || isRootContainer ){code}
So that the \{{runListener.testSetCompleted}} is always invoked for the root
container. But it all looks very adhoc right now. So I can't quite say what
that would break.
[https://github.com/apache/maven-surefire/blob/670dbe49855133947fbb436c97d7aa46e81d4d2c/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java]
was (Author: mpkorstanje):
[~tibordigana] it looks like the {{RunListenerAdapter }}is now making
assumptions about classes. I can't really work out why though. I guess this is
an attempt at squashing the JUnit results tree back into a class/method report.
If so then I guess
{code:java}
if ( failed || isClass || isTest ){code}
Should be:
{code:java}
if ( failed || isClass || isTest || isRootContainer ){code}
So that the {{runListener.testSetCompleted }}is always invoked for the root
container. But it all looks very adhoc right now. So I can't quite say what
that would break.
https://github.com/apache/maven-surefire/blob/670dbe49855133947fbb436c97d7aa46e81d4d2c/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java
> Cucumber/JUnit5 Test Execution Regression in M4 and M5
> --
>
> Key: SUREFIRE-1849
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1849
> Project: Maven Surefire
> Issue Type: Bug
>Reporter: Jacques Burns
>Priority: Major
>
> My JUnit 5 Cucumber tests fail to execute properly when in Surefire version
> 3.0.0-M4 and 3.0.0-M5. Downgrading to 3.0.0-M3 solves the issue.
> I've built a small, complete example project demostrating the issue
> [https://github.com/jmathewburns/surefire-cucumber-junit5-sscce|https://github.com/jmathewburns/surefire-cucumber-junit5-sscce].
> You can play with the version property in the `pom.xml` in my example and
> run the `test` goal
>
> In M4 and M5, the Cucumber glue is recognised and the step definitions are
> run, but failed assertions do not affect the outcome of the build. Also, true
> assertions do not count towards the "Tests run" statistic
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (SUREFIRE-1869) Classloader.getResource() doesn't encode blanks with forkCount=0
[
https://issues.apache.org/jira/browse/SUREFIRE-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256221#comment-17256221
]
M.P. Korstanje edited comment on SUREFIRE-1869 at 12/30/20, 1:34 AM:
-
Worth nothing that:
1. The test case only fails when running tests in-process. The test case does
not fail when forking the jvm. It is the inconsistency that is surprising. I
would consider the later result to be the more correct one.
2. This happens because the isolated class loader urls are constructed from the
absolute path here:
[https://github.com/apache/maven-surefire/blob/master/surefire-booter/src/main/java/org/apache/maven/surefire/booter/Classpath.java#L149]
{code:java}
classLoader.addURL( new File( classPathElement ).toURL() );{code}
And then turned into URLS here:
[https://github.com/apache/maven-surefire/blob/master/surefire-booter/src/main/java/org/apache/maven/surefire/booter/Classpath.java#L149]
{code:java}
classLoader.addURL( new File( classPathElement ).toURL() );{code}
Following [https://bugs.openjdk.java.net/browse/JDK-6227069] this should
probably be:
{code:java}
classLoader.addURL( new File( classPathElement ).toURI().toURL() );{code}
was (Author: mpkorstanje):
Worth nothing that:
1. The test case only fails when running tests in-process. The test case does
not fail when forking the jvm.
2. This happens because the isolated class loader urls are constructed from the
absolute path here:
[https://github.com/apache/maven-surefire/blob/master/surefire-booter/src/main/java/org/apache/maven/surefire/booter/Classpath.java#L149]
{code:java}
classLoader.addURL( new File( classPathElement ).toURL() );{code}
And then turned into URLS here:
[https://github.com/apache/maven-surefire/blob/master/surefire-booter/src/main/java/org/apache/maven/surefire/booter/Classpath.java#L149]
{code:java}
classLoader.addURL( new File( classPathElement ).toURL() );{code}
Following [https://bugs.openjdk.java.net/browse/JDK-6227069] this should
probably be:
{code:java}
classLoader.addURL( new File( classPathElement ).toURI().toURL() );{code}
> Classloader.getResource() doesn't encode blanks with forkCount=0
>
>
> Key: SUREFIRE-1869
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1869
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.20, 2.22.2, 3.0.0-M5
>Reporter: René Kraneis
>Priority: Major
>
> We found this bug when running cucumber tests, having project paths with
> spaces and forkCount set to 0 (see
> [https://github.com/cucumber/cucumber-jvm/issues/2196] for the but report
> there). A minimal reproducing example would be
> {code:java}
> $ pwd
> /home/rene/Projects/path with space/classpath test{code}
> {code:java}
> $ tree
> .
> ├── pom.xml
> └── src
> └── test
> └── java
> └── ClassPathTest.java{code}
> {code:java}
> $ cat src/test/java/ClassPathTest.java
> import java.io.IOException;
> import java.net.URI;
> import java.net.URISyntaxException;
> public class ClassPathTest {
> public void testClassPath() throws IOException, URISyntaxException {
> URI uri = ClassPathTest.class.getClassLoader()
> .getResources(".")
> .nextElement()
> .toURI();
> System.out.println(uri);
> }
> }{code}
> {code:java}
> $ mvn clean test -DforkCount=0
> ...
> [INFO] --- maven-surefire-plugin:2.20:test (default-test) @ ClassPathTest ---
> [WARNING] useSystemClassloader setting has no effect when not forking
> [INFO] Running ClassPathTest
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.002
> s <<< FAILURE! - in ClassPathTest
> [ERROR] ClassPathTest.testClassPath() Time elapsed: 0.001 s <<< FAILURE!
> java.net.URISyntaxException: Illegal character in path at index 29:
> file:/home/rene/Projects/path with space/classpath test/target/test-classes/
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parseHierarchical(URI.java:3105)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at java.net.URL.toURI(URL.java:964)
> at ClassPathTest.testClassPath(ClassPathTest.java:13)
> ...
> {code}
> Last working version of surefire is 2.19.1. The build does not contain any
> additional dependencies and relies on pojo test execution. The same problem
> also happens when running tests with junit, junit vintage or junit platform.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1869) Classloader.getResource() doesn't encode blanks with forkCount=0
[
https://issues.apache.org/jira/browse/SUREFIRE-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256221#comment-17256221
]
M.P. Korstanje commented on SUREFIRE-1869:
--
Worth nothing that:
1. The test case only fails when running tests in-process. The test case does
not fail when forking the jvm.
2. This happens because the isolated class loader urls are constructed from the
absolute path here:
[https://github.com/apache/maven-surefire/blob/master/surefire-booter/src/main/java/org/apache/maven/surefire/booter/Classpath.java#L149]
{code:java}
classLoader.addURL( new File( classPathElement ).toURL() );{code}
And then turned into URLS here:
[https://github.com/apache/maven-surefire/blob/master/surefire-booter/src/main/java/org/apache/maven/surefire/booter/Classpath.java#L149]
{code:java}
classLoader.addURL( new File( classPathElement ).toURL() );{code}
Following [https://bugs.openjdk.java.net/browse/JDK-6227069] this should
probably be:
{code:java}
classLoader.addURL( new File( classPathElement ).toURI().toURL() );{code}
> Classloader.getResource() doesn't encode blanks with forkCount=0
>
>
> Key: SUREFIRE-1869
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1869
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.20, 2.22.2, 3.0.0-M5
>Reporter: René Kraneis
>Priority: Major
>
> We found this bug when running cucumber tests, having project paths with
> spaces and forkCount set to 0 (see
> [https://github.com/cucumber/cucumber-jvm/issues/2196] for the but report
> there). A minimal reproducing example would be
> {code:java}
> $ pwd
> /home/rene/Projects/path with space/classpath test{code}
> {code:java}
> $ tree
> .
> ├── pom.xml
> └── src
> └── test
> └── java
> └── ClassPathTest.java{code}
> {code:java}
> $ cat src/test/java/ClassPathTest.java
> import java.io.IOException;
> import java.net.URI;
> import java.net.URISyntaxException;
> public class ClassPathTest {
> public void testClassPath() throws IOException, URISyntaxException {
> URI uri = ClassPathTest.class.getClassLoader()
> .getResources(".")
> .nextElement()
> .toURI();
> System.out.println(uri);
> }
> }{code}
> {code:java}
> $ mvn clean test -DforkCount=0
> ...
> [INFO] --- maven-surefire-plugin:2.20:test (default-test) @ ClassPathTest ---
> [WARNING] useSystemClassloader setting has no effect when not forking
> [INFO] Running ClassPathTest
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.002
> s <<< FAILURE! - in ClassPathTest
> [ERROR] ClassPathTest.testClassPath() Time elapsed: 0.001 s <<< FAILURE!
> java.net.URISyntaxException: Illegal character in path at index 29:
> file:/home/rene/Projects/path with space/classpath test/target/test-classes/
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parseHierarchical(URI.java:3105)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at java.net.URL.toURI(URL.java:964)
> at ClassPathTest.testClassPath(ClassPathTest.java:13)
> ...
> {code}
> Last working version of surefire is 2.19.1. The build does not contain any
> additional dependencies and relies on pojo test execution. The same problem
> also happens when running tests with junit, junit vintage or junit platform.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1743) Allow custom listeners to request stop
[ https://issues.apache.org/jira/browse/SUREFIRE-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012128#comment-17012128 ] M.P. Korstanje commented on SUREFIRE-1743: -- I don't see a simple way to do that with the JUnit Platform. The `ExecutionCondition` only works when JUnit Jupiter (one specific implementation of a test engine) is used. Is there anything opposed to splitting the JUnit5 Platform into versions in the feature matrix like we split Junit 4 into JUnit 4 and Junit 47? > Allow custom listeners to request stop > -- > > Key: SUREFIRE-1743 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1743 > Project: Maven Surefire > Issue Type: New Feature > Components: documentation, JUnit 5.x support, Maven Failsafe Plugin, > Maven Surefire Plugin >Reporter: Andrew Neeson >Priority: Major > > Currently Surefire gives you the ability to [add custom listeners and > reporters|https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit.html#Using_Custom_Listeners_and_Reporters]. > However, those listeners do not have a hook on the Notifier which created > them. > I would like to be able to create listeners which have the ability to request > that the test run is stopped (RunNotifier.pleaseStop) under specific > conditions. These conditions are not currently covered by existing > mechanisms such as > [skipAfterFailure|https://maven.apache.org/surefire/maven-surefire-plugin/examples/skip-after-failure.html]. > Note: I originally considered this as [a change to Cucumber > JVM|https://github.com/cucumber/cucumber-jvm/issues/1854], but I realise that > it's more generic than that (hence this ticket). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[ https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981275#comment-16981275 ] M.P. Korstanje commented on SUREFIRE-1724: -- Sounds good. Lets see what dragons I encounter. > Support JUnit Platform DiscoverySelectors > -- > > Key: SUREFIRE-1724 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1724 > Project: Maven Surefire > Issue Type: Improvement >Reporter: M.P. Korstanje >Priority: Major > > The JUnit Platform defines multiple > [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] > to discover tests in different locations. This allows TestEngines to > discover tests in classes, the class path root, resource root, files, ect. > Currently Surefire supports TestEngine implementations through the > JUnitPlatformProvider. However the JUnitPlatformProvider only issues > discovery requests for classes that have already been discovered by Surefire. > Additionally if no tests were discovered by Surefire then no test discovery > requests will be issued at all. > This means that Surefire does not fully utilize the discovery capabilities of > the JUnit Platform and makes it impossible to use the JUnitPlatformProvider > for test frameworks that do not have class based tests such as Cucumber. > To make Cucumber work at the very least the class path root discovery > selector would have to be supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[
https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981270#comment-16981270
]
M.P. Korstanje commented on SUREFIRE-1724:
--
> You mean that the Cucumber has its own selector classes?
No. Cucumber does not have it's own selectors.
While it is possible for a test engine to implement custom selectors. I think
we should stick to those provided by the JUnit Platform.
> Is it fast to look up in every Jar file?
This would be TestEngine specific. I have not benchmarked this but I would
expect Cucumber to be reasonably fast.
Though in practice I don't think we'll scan every jar on the class path. We
have to provide an explicit set of class path roots. I can see some use cases
for adding all jar file dependencies, but I think that for most cases using
{{${build.testOutputDirectory}}} as the classpath root should be sufficient.
> I guess that the user would not code the selector by himself in
> {{target/classes}}.
Yeah. This would be configured in the pom file. Marc Philipp did a
suggestion for the syntax in
https://issues.apache.org/jira/browse/SUREFIRE-1337
Based on that we could start with something like for the classpath root and
maybe add something a little bit nicer later on:
{code:java}
selector>
${build.testOutputDirectory}
{code}
> Support JUnit Platform DiscoverySelectors
> --
>
> Key: SUREFIRE-1724
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1724
> Project: Maven Surefire
> Issue Type: Improvement
>Reporter: M.P. Korstanje
>Priority: Major
>
> The JUnit Platform defines multiple
> [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html]
> to discover tests in different locations. This allows TestEngines to
> discover tests in classes, the class path root, resource root, files, ect.
> Currently Surefire supports TestEngine implementations through the
> JUnitPlatformProvider. However the JUnitPlatformProvider only issues
> discovery requests for classes that have already been discovered by Surefire.
> Additionally if no tests were discovered by Surefire then no test discovery
> requests will be issued at all.
> This means that Surefire does not fully utilize the discovery capabilities of
> the JUnit Platform and makes it impossible to use the JUnitPlatformProvider
> for test frameworks that do not have class based tests such as Cucumber.
> To make Cucumber work at the very least the class path root discovery
> selector would have to be supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[
https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981247#comment-16981247
]
M.P. Korstanje edited comment on SUREFIRE-1724 at 11/25/19 2:21 AM:
> Why you use two providers? You added junit5 api to your POM dependencies, so
> it will be junit5 provider.
Demonstration purposes. I wanted to show how the API would work with a provider
that didn't implement discovery and with one that did. The important bit here
is that the JUnit4 Provider falls back to the test classes provided by surefire
and that JUnitPlatform Provider does it's own discovery.
> Since {{discoverTestsInFork}} returns multiple classes, it woun't be verry
> effective forking process because the loaded classes must be executed in the
> same fork jvm. Another alternative is to use isolated CL in plugin and
> discover them there.
That would be more efficient. We may have to take some care and use separate
launchers for discovery and execution though.
> Why you have to {{selectClasspathRoots}} if DiscoverySelectors already exist
> on the classpath? They are not found by SPI?
Indeed they are not load via SPI. The JUnit platform uses a builder pattern to
create a discovery request. There is a nicely comprehensive example here:
[https://junit.org/junit5/docs/5.0.0/api/org/junit/platform/launcher/core/LauncherDiscoveryRequestBuilder.html]
The discovery request is executed by all available TestEngine implementations.
The test engine implementations are loaded via SPI. The test engines then
query the discovery request for selectors that they implement and they use
those selectors to discover tests.
was (Author: mpkorstanje):
> Why you use two providers? You added junit5 api to your POM dependencies, so
> it will be junit5 provider.
Demonstration purposes. I wanted to show how the API would work with a provider
that didn't implement discovery and with one that did. The important bit here
is that the JUnit4 Provider falls back to the test classes provided by surefire
and that JUnitPlatform Provider does it's own discovery.
> Since {{discoverTestsInFork}} returns multiple classes, it woun't be verry
> effective forking process because the loaded classes must be executed in the
> same fork jvm. Another alternative is to use isolated CL in plugin and
> discover them there.
That would be more efficient. We may have to take some care and use separate
launchers for discovery and execution though.
> Why you have to {{selectClasspathRoots}} if DiscoverySelectors already exist
> on the classpath? They are not found by SPI?
Indeed they are not load via SPI. The JUnit platform uses a builder pattern to
create a discovery request. There is a nicely comprehensive example here:
[https://junit.org/junit5/docs/5.0.0/api/org/junit/platform/launcher/core/LauncherDiscoveryRequestBuilder.html]
The discovery request is executed by all available TestEngine implementations.
The test engine implementations are loaded via SPI.
> Support JUnit Platform DiscoverySelectors
> --
>
> Key: SUREFIRE-1724
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1724
> Project: Maven Surefire
> Issue Type: Improvement
>Reporter: M.P. Korstanje
>Priority: Major
>
> The JUnit Platform defines multiple
> [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html]
> to discover tests in different locations. This allows TestEngines to
> discover tests in classes, the class path root, resource root, files, ect.
> Currently Surefire supports TestEngine implementations through the
> JUnitPlatformProvider. However the JUnitPlatformProvider only issues
> discovery requests for classes that have already been discovered by Surefire.
> Additionally if no tests were discovered by Surefire then no test discovery
> requests will be issued at all.
> This means that Surefire does not fully utilize the discovery capabilities of
> the JUnit Platform and makes it impossible to use the JUnitPlatformProvider
> for test frameworks that do not have class based tests such as Cucumber.
> To make Cucumber work at the very least the class path root discovery
> selector would have to be supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[
https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981247#comment-16981247
]
M.P. Korstanje edited comment on SUREFIRE-1724 at 11/25/19 2:16 AM:
> Why you use two providers? You added junit5 api to your POM dependencies, so
> it will be junit5 provider.
Demonstration purposes. I wanted to show how the API would work with a provider
that didn't implement discovery and with one that did. The important bit here
is that the JUnit4 Provider falls back to the test classes provided by surefire
and that JUnitPlatform Provider does it's own discovery.
> Since {{discoverTestsInFork}} returns multiple classes, it woun't be verry
> effective forking process because the loaded classes must be executed in the
> same fork jvm. Another alternative is to use isolated CL in plugin and
> discover them there.
That would be more efficient. We may have to take some care and use separate
launchers for discovery and execution though.
> Why you have to {{selectClasspathRoots}} if DiscoverySelectors already exist
> on the classpath? They are not found by SPI?
Indeed they are not load via SPI. The JUnit platform uses a builder pattern to
create a discovery request. There is a nicely comprehensive example here:
[https://junit.org/junit5/docs/5.0.0/api/org/junit/platform/launcher/core/LauncherDiscoveryRequestBuilder.html]
The discovery request is executed by all available TestEngine implementations.
The test engine implementations are loaded via SPI.
was (Author: mpkorstanje):
> Why you use two providers? You added junit5 api to your POM dependencies, so
> it will be junit5 provider.
Demonstration purposes. I wanted to show how the API would work with a provider
that didn't implement discovery and with one that did. The important bit here
is that the JUnit4 Provider falls back to the test classes provided by surefire
and that JUnitPlatform Provider does it's own discovery.
> Since {{discoverTestsInFork}} returns multiple classes, it woun't be verry
> effective forking process because the loaded classes must be executed in the
> same fork jvm. Another alternative is to use isolated CL in plugin and
> discover them there.
That would be more efficient.
> Why you have to {{selectClasspathRoots}} if DiscoverySelectors already exist
> on the classpath? They are not found by SPI?
Indeed they are not load via SPI. The JUnit platform uses a builder pattern to
create a discovery request. There is a nicely comprehensive example here:
[https://junit.org/junit5/docs/5.0.0/api/org/junit/platform/launcher/core/LauncherDiscoveryRequestBuilder.html]
The discovery request is executed by all available TestEngine implementations.
The test engine implementations are loaded via SPI.
> Support JUnit Platform DiscoverySelectors
> --
>
> Key: SUREFIRE-1724
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1724
> Project: Maven Surefire
> Issue Type: Improvement
>Reporter: M.P. Korstanje
>Priority: Major
>
> The JUnit Platform defines multiple
> [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html]
> to discover tests in different locations. This allows TestEngines to
> discover tests in classes, the class path root, resource root, files, ect.
> Currently Surefire supports TestEngine implementations through the
> JUnitPlatformProvider. However the JUnitPlatformProvider only issues
> discovery requests for classes that have already been discovered by Surefire.
> Additionally if no tests were discovered by Surefire then no test discovery
> requests will be issued at all.
> This means that Surefire does not fully utilize the discovery capabilities of
> the JUnit Platform and makes it impossible to use the JUnitPlatformProvider
> for test frameworks that do not have class based tests such as Cucumber.
> To make Cucumber work at the very least the class path root discovery
> selector would have to be supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[
https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981247#comment-16981247
]
M.P. Korstanje edited comment on SUREFIRE-1724 at 11/25/19 2:15 AM:
> Why you use two providers? You added junit5 api to your POM dependencies, so
> it will be junit5 provider.
Demonstration purposes. I wanted to show how the API would work with a provider
that didn't implement discovery and with one that did. The important bit here
is that the JUnit4 Provider falls back to the test classes provided by surefire
and that JUnitPlatform Provider does it's own discovery.
> Since {{discoverTestsInFork}} returns multiple classes, it woun't be verry
> effective forking process because the loaded classes must be executed in the
> same fork jvm. Another alternative is to use isolated CL in plugin and
> discover them there.
That would be more efficient.
> Why you have to {{selectClasspathRoots}} if DiscoverySelectors already exist
> on the classpath? They are not found by SPI?
Indeed they are not load via SPI. The JUnit platform uses a builder pattern to
create a discovery request. There is a nicely comprehensive example here:
[https://junit.org/junit5/docs/5.0.0/api/org/junit/platform/launcher/core/LauncherDiscoveryRequestBuilder.html]
The discovery request is executed by all available TestEngine implementations.
The test engine implementations are loaded via SPI.
was (Author: mpkorstanje):
> Why you use two providers? You added junit5 api to your POM dependencies, so
> it will be junit5 provider.
Demonstration purposes. I wanted to show how the API would work with a provider
that didn't implement discovery and with one that did. The important bit here
is that the JUnit4 Provider falls back to the test classes provided by surefire
and that JUnitPlatform Provider does it's own discovery.
> Since {{discoverTestsInFork}} returns multiple classes, it woun't be verry
> effective forking process because the loaded classes must be executed in the
> same fork jvm. Another alternative is to use isolated CL in plugin and
> discover them there.
That would be more efficient.
> Why you have to {{selectClasspathRoots}} if DiscoverySelectors already exist
> on the classpath? They are not found by SPI?
Indeed they are not load via SPI. The JUnit platform uses a builder pattern to
create a discovery request. There is a nicely comprehensive example here:
[https://junit.org/junit5/docs/5.0.0/api/org/junit/platform/launcher/core/LauncherDiscoveryRequestBuilder.html]
The discovery request is executed by all available TestEngine implementations.
These test engine implementations are loaded via SPI.
> Support JUnit Platform DiscoverySelectors
> --
>
> Key: SUREFIRE-1724
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1724
> Project: Maven Surefire
> Issue Type: Improvement
>Reporter: M.P. Korstanje
>Priority: Major
>
> The JUnit Platform defines multiple
> [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html]
> to discover tests in different locations. This allows TestEngines to
> discover tests in classes, the class path root, resource root, files, ect.
> Currently Surefire supports TestEngine implementations through the
> JUnitPlatformProvider. However the JUnitPlatformProvider only issues
> discovery requests for classes that have already been discovered by Surefire.
> Additionally if no tests were discovered by Surefire then no test discovery
> requests will be issued at all.
> This means that Surefire does not fully utilize the discovery capabilities of
> the JUnit Platform and makes it impossible to use the JUnitPlatformProvider
> for test frameworks that do not have class based tests such as Cucumber.
> To make Cucumber work at the very least the class path root discovery
> selector would have to be supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[
https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981247#comment-16981247
]
M.P. Korstanje commented on SUREFIRE-1724:
--
> Why you use two providers? You added junit5 api to your POM dependencies, so
> it will be junit5 provider.
Demonstration purposes. I wanted to show how the API would work with a provider
that didn't implement discovery and with one that did. The important bit here
is that the JUnit4 Provider falls back to the test classes provided by surefire
and that JUnitPlatform Provider does it's own discovery.
> Since {{discoverTestsInFork}} returns multiple classes, it woun't be verry
> effective forking process because the loaded classes must be executed in the
> same fork jvm. Another alternative is to use isolated CL in plugin and
> discover them there.
That would be more efficient.
> Why you have to {{selectClasspathRoots}} if DiscoverySelectors already exist
> on the classpath? They are not found by SPI?
Indeed they are not load via SPI. The JUnit platform uses a builder pattern to
create a discovery request. There is a nicely comprehensive example here:
[https://junit.org/junit5/docs/5.0.0/api/org/junit/platform/launcher/core/LauncherDiscoveryRequestBuilder.html]
The discovery request is executed by all available TestEngine implementations.
These test engine implementations are loaded via SPI.
> Support JUnit Platform DiscoverySelectors
> --
>
> Key: SUREFIRE-1724
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1724
> Project: Maven Surefire
> Issue Type: Improvement
>Reporter: M.P. Korstanje
>Priority: Major
>
> The JUnit Platform defines multiple
> [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html]
> to discover tests in different locations. This allows TestEngines to
> discover tests in classes, the class path root, resource root, files, ect.
> Currently Surefire supports TestEngine implementations through the
> JUnitPlatformProvider. However the JUnitPlatformProvider only issues
> discovery requests for classes that have already been discovered by Surefire.
> Additionally if no tests were discovered by Surefire then no test discovery
> requests will be issued at all.
> This means that Surefire does not fully utilize the discovery capabilities of
> the JUnit Platform and makes it impossible to use the JUnitPlatformProvider
> for test frameworks that do not have class based tests such as Cucumber.
> To make Cucumber work at the very least the class path root discovery
> selector would have to be supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[
https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981226#comment-16981226
]
M.P. Korstanje commented on SUREFIRE-1724:
--
> The interface {{org.junit.platform.engine.DiscoverySelector}} v1.3.2 is
> implemented by 10 default implementations.
That will be allot of typing to map the Junit Platform configuration to
classes. :D
> It would be worth to write a trivial GitHub project with your proposal
Have a look at: [https://github.com/mpkorstanje/surefire-1724-proposal] Main
class is the point to start.
It's a demo with Junit4, Junit5 and Cucumber. The latter two use the JUnit
Platform and all tests are discovered with a single discovery requests composed
of classes discovered by surefire and a class path root selector.
> I guess this might be a problem for some users but we are not there yet with
> our analysis.
Tests would be executed in a different fork so those effects should be gone. I
guess you may also not want to pollute surefire and run discovery in a fork too.
> Support JUnit Platform DiscoverySelectors
> --
>
> Key: SUREFIRE-1724
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1724
> Project: Maven Surefire
> Issue Type: Improvement
>Reporter: M.P. Korstanje
>Priority: Major
>
> The JUnit Platform defines multiple
> [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html]
> to discover tests in different locations. This allows TestEngines to
> discover tests in classes, the class path root, resource root, files, ect.
> Currently Surefire supports TestEngine implementations through the
> JUnitPlatformProvider. However the JUnitPlatformProvider only issues
> discovery requests for classes that have already been discovered by Surefire.
> Additionally if no tests were discovered by Surefire then no test discovery
> requests will be issued at all.
> This means that Surefire does not fully utilize the discovery capabilities of
> the JUnit Platform and makes it impossible to use the JUnitPlatformProvider
> for test frameworks that do not have class based tests such as Cucumber.
> To make Cucumber work at the very least the class path root discovery
> selector would have to be supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[ https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981035#comment-16981035 ] M.P. Korstanje commented on SUREFIRE-1724: -- I think I'm closer to #2 but I would say neither. Terms: * Discovery request: A request to discover test cases. A discovery requests is composed of multiple selectors. When executed it yields a collection of test identifiers. * Discovery selector: A specific location to look for tests. These take many forms most of which are provider specific. For example for Junit5 we can discover tests in a classpath root, package, class, file folder or uniqueId. Surefire currently implements an implicit discovery selector that I would describe as _classes discovered by surefire_. * Test Identifier: Some string that a provider can use to execute a specific test. I would imagine the following process: *1. Surefire issues a discovery request to a provider.* By default the discovery request will only contain _classes discovered by surefire_. However if provider specific selectors are configured these selector will be added to the request. A challenge here would be to find a nice way to disable the default discovery request by surefire when it is unwanted. *2. The provider resolves this discovery request and replies with list of test identifiers.* Providers that do not have the ability to discover tests will fall back to Surefires build in _exclusively_ classes discovery functionality. Otherwise they'll use their own. For Junit4 for example this would mean returning the list of _classes discovered by surefire_. Providers that can discover tests will do so. If requested to discovery classes exclusively Junit5 would be issued with a discovery selector for the classes discovered by surefire. If any other selectors were provided they would be added to the discovery request. The discovered tests would be provided to surefire by their uniqueId. *3. If no tests are discovered by any provider surefire exits with the usual warning.* This changes from the current situation where surefire exits when surefire wasn't able to find tests. *4. Surefire distributes the test identifiers between forks.* This doesn't change much from the existing ForkedBooter/Runner setup. We distribute the test identifiers rather then classes between the different forks. *5. The forked provider executes the requested tests.* The provider converts the test identifier back into something it can execute. Either a class name for JUnit4 or a UniqueId for Junit 5. > Support JUnit Platform DiscoverySelectors > -- > > Key: SUREFIRE-1724 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1724 > Project: Maven Surefire > Issue Type: Improvement >Reporter: M.P. Korstanje >Priority: Major > > The JUnit Platform defines multiple > [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] > to discover tests in different locations. This allows TestEngines to > discover tests in classes, the class path root, resource root, files, ect. > Currently Surefire supports TestEngine implementations through the > JUnitPlatformProvider. However the JUnitPlatformProvider only issues > discovery requests for classes that have already been discovered by Surefire. > Additionally if no tests were discovered by Surefire then no test discovery > requests will be issued at all. > This means that Surefire does not fully utilize the discovery capabilities of > the JUnit Platform and makes it impossible to use the JUnitPlatformProvider > for test frameworks that do not have class based tests such as Cucumber. > To make Cucumber work at the very least the class path root discovery > selector would have to be supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[ https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980576#comment-16980576 ] M.P. Korstanje commented on SUREFIRE-1724: -- > You can maybe see that JUnit5 is one of our provider and we have older users > e.g. Karaf/Felix/OSGi who are waiting longer to fix the blocker issue which > can be perfectly fixed with new TCP/IP interprocess communication. > This way i want to say that we have given order of the tasks as the users > requests come to JIRA. > In the next step would be the event-based report interface because this is a > prerequisite of Provider API and it solves the current activities in > JUnit5/opentest4j. I don't envy you. :D I'll use the class based work around for now. > Support JUnit Platform DiscoverySelectors > -- > > Key: SUREFIRE-1724 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1724 > Project: Maven Surefire > Issue Type: Improvement >Reporter: M.P. Korstanje >Priority: Major > > The JUnit Platform defines multiple > [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] > to discover tests in different locations. This allows TestEngines to > discover tests in classes, the class path root, resource root, files, ect. > Currently Surefire supports TestEngine implementations through the > JUnitPlatformProvider. However the JUnitPlatformProvider only issues > discovery requests for classes that have already been discovered by Surefire. > Additionally if no tests were discovered by Surefire then no test discovery > requests will be issued at all. > This means that Surefire does not fully utilize the discovery capabilities of > the JUnit Platform and makes it impossible to use the JUnitPlatformProvider > for test frameworks that do not have class based tests such as Cucumber. > To make Cucumber work at the very least the class path root discovery > selector would have to be supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[
https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980575#comment-16980575
]
M.P. Korstanje commented on SUREFIRE-1724:
--
> Do you think that {{LauncherDiscoveryRequestBuilder}} can operate over a list
> of JAR file, means their Path-s of location?
Yes. That is possible. Using the class path root selector you can provide a
list of class path roots. These can be either jar files or directories.
{code:java}
DiscoverySelectors.selectClasspathRoots(Set.of(
Paths.get("file:/path/to1.jar"),
Paths.get("file:/path/to2.jar"),
Paths.get("file:/path/to3.jar")
));
{code}
> I guess this implementation of {{LauncherDiscoveryRequestBuilder}} takes a
> look in the classpath or the classloader.
When the discover request is executed by the JUnit Platform Launcher a
TestEngine implementation may potentially scan for classes. JUnit Jupiters
implementation of the TestEngine for example will scan for classes with @Test
annotated methods. Cucumbers implementation on the other hand doesn't - it only
looks for feature files.
> I want to use the discovery via {{LauncherDiscoveryRequestBuilder}}
excluding the system classloader. I don't want this because i don't want the
classes been loaded while performing the discovery phase.
That is clear and makes allot of sense.
> Doing it in IsolatedClassLoader is feasible for the forkCount > 1 but you may
> have noticed that we are reworking several implementations (TCP/IP
> interprocess communication, Reports API to support future JUnit5 reports,
> Provider interface) and these selectors would also require splitting Provider
> interface in two at least.
>
> This effectivelly removes the IsolatedClassLoader in Maven process and
> enables the subprocess to perform filtering of test file within the same JVM
> where it would be executed and we solved the old issue requested by the
> Arquillian. I guess that DiscoverySelectors may fit here as well.
That sounds about right. Though I'm not into all the details.
Aside from splitting the Provider interface into to a discovery and execution
part, it can also no longer use classes to identify tests. The JUnit platform
identifies tests by a UniqueId. Which can be serialized to string and parsed
again. So to distribute tests across forks the fork starter and booter would
have to work with strings (maybe wrap them in something nice) rather then
classes.
> Support JUnit Platform DiscoverySelectors
> --
>
> Key: SUREFIRE-1724
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1724
> Project: Maven Surefire
> Issue Type: Improvement
>Reporter: M.P. Korstanje
>Priority: Major
>
> The JUnit Platform defines multiple
> [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html]
> to discover tests in different locations. This allows TestEngines to
> discover tests in classes, the class path root, resource root, files, ect.
> Currently Surefire supports TestEngine implementations through the
> JUnitPlatformProvider. However the JUnitPlatformProvider only issues
> discovery requests for classes that have already been discovered by Surefire.
> Additionally if no tests were discovered by Surefire then no test discovery
> requests will be issued at all.
> This means that Surefire does not fully utilize the discovery capabilities of
> the JUnit Platform and makes it impossible to use the JUnitPlatformProvider
> for test frameworks that do not have class based tests such as Cucumber.
> To make Cucumber work at the very least the class path root discovery
> selector would have to be supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[
https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980548#comment-16980548
]
M.P. Korstanje commented on SUREFIRE-1724:
--
Heya thanks for getting back so quickly. I am going to reply piecemeal. There
are quite a few points to touch upon.
> AFAIK Cucumber works with classes
It is a common misconception - classes are involved but they are not the test.
In simple terms cucumber test consists of two parts: a feature file and glue
code.
The feature file is a plain text file with the .feature suffix. For example:
{code:java}
Feature: Eating too many cucumbers may not be good for you
Eating too much of anything may not be good for you.
Scenario: Eating a few is no problem
Given Alice is hungry
When she eats 3 cucumbers
Then she will be full
And still hungry{code}
The Given/When/Then lines are called steps and part of a scenario. A scenario
would be comparable to a single test annotated method in JUnit.
To execute a scenario there must be some Java code to make these steps
executable. For example:
{code:java}
public class StepDefinitions {
private Person person
@Given("Alice is hungry")
public void aliceIsHungry(){
person = new Person("Alice")
person.setHungry(true);
}
@When("she eats {int} Cucumbers")
public void sheEatsCucumbers(int cucumbers){
person.eat(cucumbers);
}
{code}
To execute the Given line Cucumber will invoke the method annotated with "Alice
is hungry". To invoke the When line Cucumber will invoke the method annotated
with "she eats \{int} Cucumbers", ect. And as you can see the step definition
classes are not the tests. The feature file is.
> the way how Surefire discovers the classes should be satisfactory
It was with with JUnit4. Cucumber would implement a JUnit4 Runner class. For
example:
{code:java}
package io.cucumber.example;
import io.cucumber.junit.Cucumber;
import org.junit.runner.RunWith;
@RunWith(Cucumber.class)
public class RunCucumberTest {
}
{code}
However with the Junit Platform there is no need for this class any more. The
JUnit Platform implementation of Cucumber will use the ClassPathRoot and other
discovery selectors to discover feature files in the selected locations.
For now as a work around I'm adding a marker class that will be discovered by
Surefire. Cucumber will then scan the package of the annotated class for
features. But this is less then ideal - we have this superfluous class that
must be present for all the magic to work.
{code:java}
@Cucumber
public class RunCucumberTest {
}
{code}
> Support JUnit Platform DiscoverySelectors
> --
>
> Key: SUREFIRE-1724
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1724
> Project: Maven Surefire
> Issue Type: Improvement
>Reporter: M.P. Korstanje
>Priority: Major
>
> The JUnit Platform defines multiple
> [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html]
> to discover tests in different locations. This allows TestEngines to
> discover tests in classes, the class path root, resource root, files, ect.
> Currently Surefire supports TestEngine implementations through the
> JUnitPlatformProvider. However the JUnitPlatformProvider only issues
> discovery requests for classes that have already been discovered by Surefire.
> Additionally if no tests were discovered by Surefire then no test discovery
> requests will be issued at all.
> This means that Surefire does not fully utilize the discovery capabilities of
> the JUnit Platform and makes it impossible to use the JUnitPlatformProvider
> for test frameworks that do not have class based tests such as Cucumber.
> To make Cucumber work at the very least the class path root discovery
> selector would have to be supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[ https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] M.P. Korstanje updated SUREFIRE-1724: - Description: The JUnit Platform defines multiple [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] to discover tests in different locations. This allows TestEngines to discover tests in classes, the class path root, resource root, files, ect. Currently Surefire supports TestEngine implementations through the JUnitPlatformProvider. However the JUnitPlatformProvider only issues discovery requests for classes that have already been discovered by Surefire. Additionally if no tests were discovered by Surefire then no test discovery requests will be issued at all. This means that Surefire does not fully utilize the discovery capabilities of the JUnit Platform and makes it impossible to use the JUnitPlatformProvider for test frameworks that do not have class based tests such as Cucumber. To make Cucumber work at the very least the class path root discovery selector would have to be supported. was: The JUnit Platform defines multiple [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] to discover tests in different locations. This allows TestEngines to discover tests in classes, the class path root, resource root, files, ect. Currently Surefire supports TestEngine implementations through the JUnitPlatformProvider. However the JUnitPlatformProvider only issues discovery requests for classes that have already been discovered by Surefire. Additionally if no tests were discovered by Surefire then no test discovery requests will be issued at all. This means that Surefire does not fully utilize the discovery capabilities of the JUnit Platform and makes it impossible to use the JUnitPlatformProvider for test frameworks that do not have class based tests such as Cucumber. > Support JUnit Platform DiscoverySelectors > -- > > Key: SUREFIRE-1724 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1724 > Project: Maven Surefire > Issue Type: Improvement >Reporter: M.P. Korstanje >Priority: Major > > The JUnit Platform defines multiple > [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] > to discover tests in different locations. This allows TestEngines to > discover tests in classes, the class path root, resource root, files, ect. > Currently Surefire supports TestEngine implementations through the > JUnitPlatformProvider. However the JUnitPlatformProvider only issues > discovery requests for classes that have already been discovered by Surefire. > Additionally if no tests were discovered by Surefire then no test discovery > requests will be issued at all. > This means that Surefire does not fully utilize the discovery capabilities of > the JUnit Platform and makes it impossible to use the JUnitPlatformProvider > for test frameworks that do not have class based tests such as Cucumber. > To make Cucumber work at the very least the class path root discovery > selector would have to be supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[ https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980058#comment-16980058 ] M.P. Korstanje commented on SUREFIRE-1724: -- Note that this functionality was also requested in https://issues.apache.org/jira/browse/SUREFIRE-1337 however depending on the implementation this may not result in the ability to execute non-class based tests. > Support JUnit Platform DiscoverySelectors > -- > > Key: SUREFIRE-1724 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1724 > Project: Maven Surefire > Issue Type: Improvement >Reporter: M.P. Korstanje >Priority: Major > > The JUnit Platform defines multiple > [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] > to discover tests in different locations. This allows TestEngines to > discover tests in classes, the class path root, resource root, files, ect. > Currently Surefire supports TestEngine implementations through the > JUnitPlatformProvider. However the JUnitPlatformProvider only issues > discovery requests for classes that have already been discovered by Surefire. > Additionally if no tests were discovered by Surefire then no test discovery > requests will be issued at all. > This means that Surefire does not fully utilize the discovery capabilities of > the JUnit Platform and makes it impossible to use the JUnitPlatformProvider > for test frameworks that do not have class based tests such as Cucumber. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[ https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] M.P. Korstanje updated SUREFIRE-1724: - Description: The JUnit Platform defines multiple [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] to discover tests in different locations. This allows TestEngines to discover tests in classes, the class path root, resource root, files, ect. Currently Surefire supports TestEngine implementations through the JUnitPlatformProvider. However the JUnitPlatformProvider only issues discovery requests for classes that have already been discovered by Surefire. Additionally if no tests were discovered by Surefire then no test discovery requests will be issued at all. This means that Surefire does not fully utilize the discovery capabilities of the JUnit Platform and makes it impossible to use the JUnitPlatformProvider for test frameworks that do not have class based tests such as Cucumber. was: The JUnit Platform defines multiple [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] to discover tests in different locations. This allows TestEngines to discover tests in classes, the class path root, resource root, files, ect. Currently Surefire supports TestEngine implementations through the JUnitPlatformProvider. However the JUnitPlatformProvider only issues test discovery requests for classes that have already been discovered by Surefire. Additionally if no tests were discovered by Surefire then no test discovery requests will be issued at all. This means that Surefire does not fully utilize the discovery capabilities of the JUnit Platform and makes it impossible to use the JUnitPlatformProvider for test frameworks that do not have class based tests such as Cucumber. > Support JUnit Platform DiscoverySelectors > -- > > Key: SUREFIRE-1724 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1724 > Project: Maven Surefire > Issue Type: Improvement >Reporter: M.P. Korstanje >Priority: Major > > The JUnit Platform defines multiple > [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] > to discover tests in different locations. This allows TestEngines to > discover tests in classes, the class path root, resource root, files, ect. > Currently Surefire supports TestEngine implementations through the > JUnitPlatformProvider. However the JUnitPlatformProvider only issues > discovery requests for classes that have already been discovered by Surefire. > Additionally if no tests were discovered by Surefire then no test discovery > requests will be issued at all. > This means that Surefire does not fully utilize the discovery capabilities of > the JUnit Platform and makes it impossible to use the JUnitPlatformProvider > for test frameworks that do not have class based tests such as Cucumber. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
[ https://issues.apache.org/jira/browse/SUREFIRE-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] M.P. Korstanje updated SUREFIRE-1724: - Description: The JUnit Platform defines multiple [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] to discover tests in different locations. This allows TestEngines to discover tests in classes, the class path root, resource root, files, ect. Currently Surefire supports TestEngine implementations through the JUnitPlatformProvider. However the JUnitPlatformProvider only issues test discovery requests for classes that have already been discovered by Surefire. Additionally if no tests were discovered by Surefire then no test discovery requests will be issued at all. This means that Surefire does not fully utilize the discovery capabilities of the JUnit Platform and makes it impossible to use the JUnitPlatformProvider for test frameworks that do not have class based tests such as Cucumber. was: The JUnit Platform defines multiple [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] to discover different kinds of tests. This allows TestEngines to discover tests in classes, the class path root, resource root, files, ect. Currently Surefire supports TestEngine implementations through the JUnitPlatformProvider. However the JUnitPlatformProvider only issues test discovery requests for classes that have already been discovered by Surefire. Additionally if no tests were discovered by Surefire then no test discovery requests will be issued at all. This means that Surefire does not fully utilize the discovery capabilities of the JUnit Platform and makes it impossible to use the JUnitPlatformProvider for test frameworks that do not have class based tests such as Cucumber. > Support JUnit Platform DiscoverySelectors > -- > > Key: SUREFIRE-1724 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1724 > Project: Maven Surefire > Issue Type: Improvement >Reporter: M.P. Korstanje >Priority: Major > > The JUnit Platform defines multiple > [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] > to discover tests in different locations. This allows TestEngines to > discover tests in classes, the class path root, resource root, files, ect. > Currently Surefire supports TestEngine implementations through the > JUnitPlatformProvider. However the JUnitPlatformProvider only issues test > discovery requests for classes that have already been discovered by Surefire. > Additionally if no tests were discovered by Surefire then no test discovery > requests will be issued at all. > This means that Surefire does not fully utilize the discovery capabilities of > the JUnit Platform and makes it impossible to use the JUnitPlatformProvider > for test frameworks that do not have class based tests such as Cucumber. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SUREFIRE-1724) Support JUnit Platform DiscoverySelectors
M.P. Korstanje created SUREFIRE-1724: Summary: Support JUnit Platform DiscoverySelectors Key: SUREFIRE-1724 URL: https://issues.apache.org/jira/browse/SUREFIRE-1724 Project: Maven Surefire Issue Type: Improvement Reporter: M.P. Korstanje The JUnit Platform defines multiple [DiscoverySelectors|https://junit.org/junit5/docs/5.0.2/api/org/junit/platform/engine/DiscoverySelector.html] to discover different kinds of tests. This allows TestEngines to discover tests in classes, the class path root, resource root, files, ect. Currently Surefire supports TestEngine implementations through the JUnitPlatformProvider. However the JUnitPlatformProvider only issues test discovery requests for classes that have already been discovered by Surefire. Additionally if no tests were discovered by Surefire then no test discovery requests will be issued at all. This means that Surefire does not fully utilize the discovery capabilities of the JUnit Platform and makes it impossible to use the JUnitPlatformProvider for test frameworks that do not have class based tests such as Cucumber. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SUREFIRE-1691) surefire-junit47 reports results against incorrect test
[
https://issues.apache.org/jira/browse/SUREFIRE-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
M.P. Korstanje updated SUREFIRE-1691:
-
Description:
This is a follow up on:
- https://issues.apache.org/jira/browse/SUREFIRE-1156
- [https://github.com/cucumber/cucumber-jvm/issues/865]
The latest reproducer can be found here:
-
[https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2]
In short Cucumber has a hierarchy of tests that looks at follows:
{noformat}
Class annotated with @RunWith(Cucumber.class)
|- Feature File 1
| |- Scenario 1a
| |- Scenario 1b
|- Feature File 2
| |- Scenario 2a
| |- Scenario 2b
{noformat}
Surefire tries to group output in tests sets, either per class or per test
suite. As Cucumber currently only emits test started/finished events for
Scenarios sure fire can not group scenarios into a test set. They are instead
grouped with the previous test set. However even after adding test
started/finished events for features, sure fire groups all results in a single
test set.
I believe this is caused by an error in
{{NonConcurrentRunListener.describesNewTestSet}}.
{code:java}
private boolean describesNewTestSet( Description description )
{
if ( currentTestSetDescription != null )
{
if ( null != description.getTestClass() )
{
return !description.getTestClass().equals(
currentTestSetDescription.getTestClass() );
}
else if ( description.isSuite() )
{
return description.getChildren().equals(
currentTestSetDescription.getChildren() );
}
return false;
}
return true;
}
{code}
The value of {{description.getChildren().equals(
currentTestSetDescription.getChildren() );}} should be negated so so different
test suits are properly recognized as test sets.
Fixing this locally works and can be reproduced by:
1. Negating the line and building sure fire locally
2. Building [https://github.com/cucumber/cucumber-jvm/pull/1765] with `mvn
clean install -DskipTests`
3. Running `mvn clean test -Pjunit47` on
[https://github.com/mpkorstanje/surefire-test]
It does however does result in another unwanted result: A feature file with two
scenarios is now reported as having 3 results. One for each scenario and one
for the feature.
{noformat}
---
Test set: Some Feature 1
---
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in
Some_Feature_1
{noformat}
was:
This is a follow up on:
- https://issues.apache.org/jira/browse/SUREFIRE-1156
- [https://github.com/cucumber/cucumber-jvm/issues/865]
The latest reproducer can be found here:
-
[https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2]
In short Cucumber has a hierarchy of tests that looks at follows:
{noformat}
Class annotated with @RunWith(Cucumber.class)
|- Feature File 1
| |- Scenario 1a
| |- Scenario 1b
|- Feature File 2
| |- Scenario 2a
| |- Scenario 2b
{noformat}
Surefire tries to group output in tests sets, either per class or per test
suite. As Cucumber currently only emits test started/finished events for
Scenarios sure fire can not group scenarios into a test set. They are instead
group with the previous test set. However even after adding test
started/finished events for features results in a single test set.
I believe there is an error in {{NonConcurrentRunListener.describesNewTestSet}}.
{code:java}
private boolean describesNewTestSet( Description description )
{
if ( currentTestSetDescription != null )
{
if ( null != description.getTestClass() )
{
return !description.getTestClass().equals(
currentTestSetDescription.getTestClass() );
}
else if ( description.isSuite() )
{
return description.getChildren().equals(
currentTestSetDescription.getChildren() );
}
return false;
}
return true;
}
{code}
The value of {{description.getChildren().equals(
currentTestSetDescription.getChildren() );}} should be negated so so different
test suits are properly recognized as test sets.
Fixing this locally works and can be reproduced by:
1. Negating the line and building sure fire locally
2. Building [https://github.com/cucumber/cucumber-jvm/pull/1765] with `mvn
clean install -DskipTests`
3. Running `mvn clean test -Pjunit47` on
[https://github.com/mpkorstanje/surefire-test]
It does however does result in another unwanted result: A feature file with two
scenarios is now reported as having 3 results. One for each scenario and one
for the feature.
{noformat}
[jira] [Updated] (SUREFIRE-1691) surefire-junit47 reports results against incorrect test
[
https://issues.apache.org/jira/browse/SUREFIRE-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
M.P. Korstanje updated SUREFIRE-1691:
-
Description:
This is a follow up on:
- https://issues.apache.org/jira/browse/SUREFIRE-1156
- [https://github.com/cucumber/cucumber-jvm/issues/865]
The latest reproducer can be found here:
-
[https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2]
In short Cucumber has a hierarchy of tests that looks at follows:
{noformat}
Class annotated with @RunWith(Cucumber.class)
|- Feature File 1
| |- Scenario 1a
| |- Scenario 1b
|- Feature File 2
| |- Scenario 2a
| |- Scenario 2b
{noformat}
Surefire tries to group output in tests sets, either per class or per test
suite. As Cucumber currently only emits test started/finished events for
Scenarios sure fire can not group scenarios into a test set. They are instead
group with the previous test set. However even after adding test
started/finished events for features results in a single test set.
I believe there is an error in {{NonConcurrentRunListener.describesNewTestSet}}.
{code:java}
private boolean describesNewTestSet( Description description )
{
if ( currentTestSetDescription != null )
{
if ( null != description.getTestClass() )
{
return !description.getTestClass().equals(
currentTestSetDescription.getTestClass() );
}
else if ( description.isSuite() )
{
return description.getChildren().equals(
currentTestSetDescription.getChildren() );
}
return false;
}
return true;
}
{code}
The value of {{description.getChildren().equals(
currentTestSetDescription.getChildren() );}} should be negated so so different
test suits are properly recognized as test sets.
Fixing this locally works and can be reproduced by:
1. Negating the line and building sure fire locally
2. Building [https://github.com/cucumber/cucumber-jvm/pull/1765] with `mvn
clean install -DskipTests`
3. Running `mvn clean test -Pjunit47` on
[https://github.com/mpkorstanje/surefire-test]
It does however does result in another unwanted result: A feature file with two
scenarios is now reported as having 3 results. One for each scenario and one
for the feature.
{noformat}
---
Test set: Some Feature 1
---
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in
Some_Feature_1
{noformat}
was:
This is a follow up on:
- https://issues.apache.org/jira/browse/SUREFIRE-1156
- [https://github.com/cucumber/cucumber-jvm/issues/865]
The latest reproducer can be found here:
-
[https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2]
In short Cucumber has a hierarchy of tests that looks at follows:
{noformat}
Class annotated with @RunWith(Cucumber.class)
|- Feature File 1
| |- Scenario 1a
| |- Scenario 1b
|- Feature File 2
| |- Scenario 2a
| |- Scenario 2b
{noformat}
Surefire tries to group output in tests sets, either per class or per test
suite. As Cucumber currently only emits test started/finished events for
Scenarios sure fire can not group scenarios into a test set. They are instead
group with the previous test set. However even after adding test
started/finished events for features results in a single test set.
I believe there is an error in {{NonConcurrentRunListener.describesNewTestSet}}.
{code:java}
private boolean describesNewTestSet( Description description )
{
if ( currentTestSetDescription != null )
{
if ( null != description.getTestClass() )
{
return !description.getTestClass().equals(
currentTestSetDescription.getTestClass() );
}
else if ( description.isSuite() )
{
return description.getChildren().equals(
currentTestSetDescription.getChildren() );
}
return false;
}
return true;
}
{code}
The value of {{description.getChildren().equals(
currentTestSetDescription.getChildren() );}} should be negated so so different
test sets are properly recognized.
Fixing this locally works and can be reproduced by:
1. Negating the line and building sure fire locally
2. Building [https://github.com/cucumber/cucumber-jvm/pull/1765] with `mvn
clean install -DskipTests`
3. Running `mvn clean test -Pjunit47` on
[https://github.com/mpkorstanje/surefire-test]
It does however does result in another unwanted result: A feature file with two
scenarios is now reported as having 3 results. One for each scenario and one
for the feature.
{noformat}
[jira] [Created] (SUREFIRE-1691) surefire-junit47 reports results against incorrect test
M.P. Korstanje created SUREFIRE-1691:
Summary: surefire-junit47 reports results against incorrect test
Key: SUREFIRE-1691
URL: https://issues.apache.org/jira/browse/SUREFIRE-1691
Project: Maven Surefire
Issue Type: Bug
Components: Junit 4.x support
Affects Versions: 3.0.0-M3
Reporter: M.P. Korstanje
This is a follow up on:
- https://issues.apache.org/jira/browse/SUREFIRE-1156
- [https://github.com/cucumber/cucumber-jvm/issues/865]
The latest reproducer can be found here:
-
[https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2]
In short Cucumber has a hierarchy of tests that looks at follows:
{noformat}
Class annotated with @RunWith(Cucumber.class)
|- Feature File 1
| |- Scenario 1a
| |- Scenario 1b
|- Feature File 2
| |- Scenario 2a
| |- Scenario 2b
{noformat}
Surefire tries to group output in tests sets, either per class or per test
suite. As Cucumber currently only emits test started/finished events for
Scenarios sure fire can not group scenarios into a test set. They are instead
group with the previous test set. However even after adding test
started/finished events for features results in a single test set.
I believe there is an error in {{NonConcurrentRunListener.describesNewTestSet}}.
{code:java}
private boolean describesNewTestSet( Description description )
{
if ( currentTestSetDescription != null )
{
if ( null != description.getTestClass() )
{
return !description.getTestClass().equals(
currentTestSetDescription.getTestClass() );
}
else if ( description.isSuite() )
{
return description.getChildren().equals(
currentTestSetDescription.getChildren() );
}
return false;
}
return true;
}
{code}
The value of {{description.getChildren().equals(
currentTestSetDescription.getChildren() );}} should be negated so so different
test sets are properly recognized.
Fixing this locally works and can be reproduced by:
1. Negating the line and building sure fire locally
2. Building [https://github.com/cucumber/cucumber-jvm/pull/1765] with `mvn
clean install -DskipTests`
3. Running `mvn clean test -Pjunit47` on
[https://github.com/mpkorstanje/surefire-test]
It does however does result in another unwanted result: A feature file with two
scenarios is now reported as having 3 results. One for each scenario and one
for the feature.
{noformat}
---
Test set: Some Feature 1
---
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in
Some_Feature_1
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106258#comment-16106258
]
M.P. Korstanje commented on SUREFIRE-1372:
--
[~tibor17]
Regarding the NonConcurrentRunListener. My solution broke several tests in
non-obvious ways. I'd need to invest some time in creating a proper minimal
reproducer and report it as a separate issue. However the
NonConcurrentRunListener works as expected which is good enough for me right
now.
I've setup the integration tests to run against cucumber:2.0.0-SNAPSHOT. So you
can go ahead and merge it.
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074081#comment-16074081
]
M.P. Korstanje commented on SUREFIRE-1372:
--
This issue can not be resolved until cucumber 2.0.0 is done. We're wrapping up.
However at the moment it is two people and their spare time so it could be
anywhere between a month or six.
Please do not wait for cucumber.
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037207#comment-16037207
]
M.P. Korstanje edited comment on SUREFIRE-1372 at 6/5/17 6:33 PM:
--
Certainly. I'll add those.
Mean while would it be possible to use
[Description#getClassName()|https://github.com/junit-team/junit4/blob/master/src/main/java/org/junit/runner/Description.java#L310]
in
[NonConcurrentRunListener#describesNewTestSet|https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/NonConcurrentRunListener.java#L120]?
The current implementation assumes the class is loadable which may not always
be the case when dealing with suites of generated tests.
It'd look like this:
https://github.com/mpkorstanje/maven-surefire/commit/73814de8694b9c40e7636b480a91d936d224f974
was (Author: mpkorstanje):
Certainly. I'll add those.
Mean while would it be possible to use
[Description#getClassName()|https://github.com/junit-team/junit4/blob/master/src/main/java/org/junit/runner/Description.java#L310]
in
[NonConcurrentRunListener#describesNewTestSet|https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/NonConcurrentRunListener.java#L120]?
The current implementation assumes the class is loadable which may not always
be the case when dealing with suites of generated tests.
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037207#comment-16037207
]
M.P. Korstanje commented on SUREFIRE-1372:
--
Certainly. I'll add those.
Mean while would it be possible to use
[Description#getClassName()|https://github.com/junit-team/junit4/blob/master/src/main/java/org/junit/runner/Description.java#L310]
in
[NonConcurrentRunListener#describesNewTestSet|https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/NonConcurrentRunListener.java#L120]?
The current implementation assumes the class is loadable which may not always
be the case when dealing with suites of generated tests.
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027395#comment-16027395
]
M.P. Korstanje edited comment on SUREFIRE-1372 at 5/27/17 10:59 AM:
bq. Ok, lets try to use latest cucumber release version without having
2.0.0-SNAPSHOT and run the build of surefire mvn install -P run-its. Is the
build successful?
No. {{cucumber-jvm:1.2.5}} does not provided proper id objects in Descriptions.
They don't implement equals and hashcode making it impossible to rerun failing
tests. This causes the the failure of JUnit47RerunFailingTestWithCucumberIT and
JUnit4RerunFailingTestWithCucumberIT.
Additionally I'm getting incorrect runcounts for JUnit4RerunFailingTestsIT
because it does not create a new {{Result}} object for the rerun session. This
did not bother the old implementation because by going straight to the
execution of the class method a testFinished event was never fired. Currently
considering how to resolve this, but at the moment I reckon it'd be best to
only support the Junit47Provider. Not unreasonable as cucumber requires Junit
4.12.
bq. Is there any way of using CucumberOptions with re-run in the old release
versions of cucumber?
No. Cucumber.Options was deprecated in favor of CucumberOptions and has been
removed in {{cucumber-jvm:1.2.0}}. This unfortunately did not result in a major
release.
bq. Any drawback?
CucumberOptions does not control or influence the rerun. It is not required.
However not using {{@CucumberOptions(junit = \{"--no-step-notifications"\})}}
means cucumber will report cucumber steps (the line in a test) as tests rather
then cucumber scenarios (the actual tests). This makes the test count and
reports look odd.
Additionally it will not be possible to rerun older versions of cucumber
because cucumber does not provide a way by which the tests to rerun can be
identified.
As shown in:
https://github.com/apache/maven-surefire/blob/cba4adb1b93002c5b4bb2d2f22f461cc53bd8738/surefire-integration-tests/src/test/java/org/apache/maven/surefire/its/JUnit47WithCucumberIT.java#L57-L59
bq. Maybe the users of surefire and Cucubmer would prefer some documentation in
surefire if current CucumberOptions are possible.
Do you have any suggestion where I might submit such documentation?
was (Author: mpkorstanje):
bq. Ok, lets try to use latest cucumber release version without having
2.0.0-SNAPSHOT and run the build of surefire mvn install -P run-its. Is the
build successful?
No. {{cucumber-jvm:1.2.5}} does not provided proper id objects in Descriptions.
They don't implement equals and hashcode making it impossible to rerun failing
tests. This causes the the failure of JUnit47RerunFailingTestWithCucumberIT and
JUnit4RerunFailingTestWithCucumberIT.
Additionally I'm getting incorrect runcounts for JUnit4RerunFailingTestsIT
because it does not create a new {{Result}} object for the rerun session. This
did not bother the old implementation because by going straight to the
execution of the class method a testFinished event was never fired. Currently
considering how to resolve this, but at the moment I reckon it'd be best to
only support the Junit47Provider. Not unreasonable as cucumber requires Junit
4.12.
bq. Is there any way of using CucumberOptions with re-run in the old release
versions of cucumber?
No. Cucumber.Options was deprecated in favor of CucumberOptions and has been
removed in {{cucumber-jvm:1.2.0}}. This unfortunately did not result in a major
release.
bq. Any drawback?
CucumberOptions does not control or influence the rerun. It is not required.
However not using {{@CucumberOptions(junit = \{"--no-step-notifications"\})}}
means cucumber will report cucumber steps (the line in a test) as tests rather
then cucumber scenarios (the actual tests). This makes the test count and
reports look odd.
As shown in:
https://github.com/apache/maven-surefire/blob/cba4adb1b93002c5b4bb2d2f22f461cc53bd8738/surefire-integration-tests/src/test/java/org/apache/maven/surefire/its/JUnit47WithCucumberIT.java#L57-L59
bq. Maybe the users of surefire and Cucubmer would prefer some documentation in
surefire if current CucumberOptions are possible.
Do you have any suggestion where I might submit such documentation?
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
>
[jira] [Commented] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027395#comment-16027395
]
M.P. Korstanje commented on SUREFIRE-1372:
--
> Ok, lets try to use latest cucumber release version without having
> 2.0.0-SNAPSHOT and run the build of surefire mvn install -P run-its. Is the
> build successful?
No. {{cucumber-jvm:1.2.5}} does not provided proper id objects in Descriptions.
They don't implement equals and hashcode making it impossible to rerun failing
tests. This causes the the failure of JUnit47RerunFailingTestWithCucumberIT and
JUnit4RerunFailingTestWithCucumberIT.
Additionally I'm getting incorrect runcounts for JUnit4RerunFailingTestsIT
because it does not create a new {{Result}} object for the rerun session. This
did not bother the old implementation because by going straight to the
execution of the class method a testFinished event was never fired. Currently
considering how to resolve this, but at the moment I reckon it'd be best to
only support the Junit47Provider. Not unreasonable as cucumber requires Junit
4.12.
> Is there any way of using CucumberOptions with re-run in the old release
> versions of cucumber?
No. Cucumber.Options was deprecated in favor of CucumberOptions and has been
removed in {{cucumber-jvm:1.2.0}}. This unfortunately did not result in a major
release.
> Any drawback?
CucumberOptions does not control or influence the rerun. It is not required.
However not using {{@CucumberOptions(junit = \{"--no-step-notifications"\})}}
means cucumber will report cucumber steps (the line in a test) as tests rather
then cucumber scenarios (the actual tests). This makes the test count and
reports look odd.
As shown in:
https://github.com/apache/maven-surefire/blob/cba4adb1b93002c5b4bb2d2f22f461cc53bd8738/surefire-integration-tests/src/test/java/org/apache/maven/surefire/its/JUnit47WithCucumberIT.java#L57-L59
> Maybe the users of surefire and Cucubmer would prefer some documentation in
> surefire if current CucumberOptions are possible.
Do you have any suggestion where I might submit such documentation?
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027395#comment-16027395
]
M.P. Korstanje edited comment on SUREFIRE-1372 at 5/27/17 10:41 AM:
bq. Ok, lets try to use latest cucumber release version without having
2.0.0-SNAPSHOT and run the build of surefire mvn install -P run-its. Is the
build successful?
No. {{cucumber-jvm:1.2.5}} does not provided proper id objects in Descriptions.
They don't implement equals and hashcode making it impossible to rerun failing
tests. This causes the the failure of JUnit47RerunFailingTestWithCucumberIT and
JUnit4RerunFailingTestWithCucumberIT.
Additionally I'm getting incorrect runcounts for JUnit4RerunFailingTestsIT
because it does not create a new {{Result}} object for the rerun session. This
did not bother the old implementation because by going straight to the
execution of the class method a testFinished event was never fired. Currently
considering how to resolve this, but at the moment I reckon it'd be best to
only support the Junit47Provider. Not unreasonable as cucumber requires Junit
4.12.
bq. Is there any way of using CucumberOptions with re-run in the old release
versions of cucumber?
No. Cucumber.Options was deprecated in favor of CucumberOptions and has been
removed in {{cucumber-jvm:1.2.0}}. This unfortunately did not result in a major
release.
bq. Any drawback?
CucumberOptions does not control or influence the rerun. It is not required.
However not using {{@CucumberOptions(junit = \{"--no-step-notifications"\})}}
means cucumber will report cucumber steps (the line in a test) as tests rather
then cucumber scenarios (the actual tests). This makes the test count and
reports look odd.
As shown in:
https://github.com/apache/maven-surefire/blob/cba4adb1b93002c5b4bb2d2f22f461cc53bd8738/surefire-integration-tests/src/test/java/org/apache/maven/surefire/its/JUnit47WithCucumberIT.java#L57-L59
bq. Maybe the users of surefire and Cucubmer would prefer some documentation in
surefire if current CucumberOptions are possible.
Do you have any suggestion where I might submit such documentation?
was (Author: mpkorstanje):
> Ok, lets try to use latest cucumber release version without having
> 2.0.0-SNAPSHOT and run the build of surefire mvn install -P run-its. Is the
> build successful?
No. {{cucumber-jvm:1.2.5}} does not provided proper id objects in Descriptions.
They don't implement equals and hashcode making it impossible to rerun failing
tests. This causes the the failure of JUnit47RerunFailingTestWithCucumberIT and
JUnit4RerunFailingTestWithCucumberIT.
Additionally I'm getting incorrect runcounts for JUnit4RerunFailingTestsIT
because it does not create a new {{Result}} object for the rerun session. This
did not bother the old implementation because by going straight to the
execution of the class method a testFinished event was never fired. Currently
considering how to resolve this, but at the moment I reckon it'd be best to
only support the Junit47Provider. Not unreasonable as cucumber requires Junit
4.12.
> Is there any way of using CucumberOptions with re-run in the old release
> versions of cucumber?
No. Cucumber.Options was deprecated in favor of CucumberOptions and has been
removed in {{cucumber-jvm:1.2.0}}. This unfortunately did not result in a major
release.
> Any drawback?
CucumberOptions does not control or influence the rerun. It is not required.
However not using {{@CucumberOptions(junit = \{"--no-step-notifications"\})}}
means cucumber will report cucumber steps (the line in a test) as tests rather
then cucumber scenarios (the actual tests). This makes the test count and
reports look odd.
As shown in:
https://github.com/apache/maven-surefire/blob/cba4adb1b93002c5b4bb2d2f22f461cc53bd8738/surefire-integration-tests/src/test/java/org/apache/maven/surefire/its/JUnit47WithCucumberIT.java#L57-L59
> Maybe the users of surefire and Cucubmer would prefer some documentation in
> surefire if current CucumberOptions are possible.
Do you have any suggestion where I might submit such documentation?
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
>
[jira] [Comment Edited] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026814#comment-16026814
]
M.P. Korstanje edited comment on SUREFIRE-1372 at 5/26/17 8:58 PM:
---
There will be no patch version of {{cucumber-jvm}}. I'll ask for an e.t.a. on
the 2.0.0 release but as far as I am concerned there is no need for surefire to
wait for {{cucumber-jvm:2.0.0}}. I'm not in a rush.
I've added integration tests and created a PR against maven-surefire:
https://github.com/apache/maven-surefire/pull/150
However note that this uses {{cucumber-jvm:2.0.0-SNAPSHOT}} and depends on
features that have yet to be merged.
was (Author: mpkorstanje):
There will be no patch version of {{cucumber-jvm}}. I'll ask for an e.t.a. on
the 2.0.0 release but as far as I am concerned there is no need for surefire to
wait for {{cucumber-jvm:2.0.0}}. I'm not in a rush.
I've added integration tests and created a PR against maven-surefire:
https://github.com/apache/maven-surefire/pull/150
However note that this uses {{cucumber-jvm:2.0.0-SNAPSHOT}}.
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026814#comment-16026814
]
M.P. Korstanje edited comment on SUREFIRE-1372 at 5/26/17 8:47 PM:
---
There will be no patch version of {{cucumber-jvm}}. I'll ask for an e.t.a. on
the 2.0.0 release but as far as I am concerned there is no need for surefire to
wait for {{cucumber-jvm:2.0.0}}. I'm not in a rush.
I've added integration tests and created a PR against maven-surefire:
https://github.com/apache/maven-surefire/pull/150
However note that this uses {{cucumber-jvm:2.0.0-SNAPSHOT}}.
was (Author: mpkorstanje):
There will be no patch version of {{cucumber-jvm}}. I'll ask for an e.t.a. on
the 2.0.0 release but as far as I am concerned there is no need for surefire to
wait for {{cucumber-jvm:2.0.0}}. I'm not in a rush.
I've added integration tests and created a PR against maven-surefire:
https://github.com/apache/maven-surefire/pull/150
However not that this uses {{cucumber-jvm:2.0.0-SNAPSHOT}}.
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026814#comment-16026814
]
M.P. Korstanje commented on SUREFIRE-1372:
--
There will be no patch version of {{cucumber-jvm}}. I'll ask for an e.t.a. on
the 2.0.0 release but as far as I am concerned there is no need for surefire to
wait for {{cucumber-jvm:2.0.0}}. I'm not in a rush.
I've added integration tests and created a PR against maven-surefire:
https://github.com/apache/maven-surefire/pull/150
However not that this uses {{cucumber-jvm:2.0.0-SNAPSHOT}}.
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026016#comment-16026016
]
M.P. Korstanje commented on SUREFIRE-1372:
--
Maintainer of cucumber-jvm has been located. Currently waiting for
[#1134|https://github.com/cucumber/cucumber-jvm/pull/1134] and the release of
cucumber-jvm 2.0.0.
Still not expecting to make the deadline for 2.20.1.
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007357#comment-16007357
]
M.P. Korstanje commented on SUREFIRE-1372:
--
I've hit a roadblock. I need
[#1121|https://github.com/cucumber/cucumber-jvm/pull/1121] merged into
cucumber-jvm to make surefire work but it appears the maintainer is awol.
So I don't expect this will make it into 2.20.1. Should things change I'll
finish this up.
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006760#comment-16006760
]
M.P. Korstanje commented on SUREFIRE-1372:
--
On what time-frame do you want to finish 2.20.1?
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006751#comment-16006751
]
M.P. Korstanje commented on SUREFIRE-1372:
--
No problem. I'll add integration tests.
I'm not cucumber staff. Acting as contributor of cucumber-jvm-parallel-plugin.
Yes this is cucumber related. Cucumber doesn't use test classes and provides
human readable names where surefire expects a class.
I will also have to fix this issue for junit47 provider.
I'm not aware of any other cucumber-surefire related issues.
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>Assignee: Tibor Digana
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005561#comment-16005561
]
M.P. Korstanje commented on SUREFIRE-1372:
--
Found a solution. Just gotta write the tests.
https://github.com/mpkorstanje/maven-surefire/commit/1ad7af080eb732adabfcec702f1b8f3743736316
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
M.P. Korstanje updated SUREFIRE-1372:
-
Description:
When using surefire to rerun failing tests created by a Runner that uses
{noformat}Description#createSuiteDescription{noformat} with a human readable
name rather then a class name the following stack trace occurs:
{code}
org.apache.maven.surefire.testset.TestSetFailedException: Unable to create test
class 'Scenario: Fail when running'
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
{code}
was:
Using surefire to rerun failing tests created by a Runner that uses
{noformat}Description#createSuiteDescription{noformat} with a human readable
name rather then a class name the following stack trace occurs:
{code}
org.apache.maven.surefire.testset.TestSetFailedException: Unable to create test
class 'Scenario: Fail when running'
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
{code}
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>
> When using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005492#comment-16005492
]
M.P. Korstanje edited comment on SUREFIRE-1372 at 5/10/17 9:38 PM:
---
It appears that JUnit4Provider#executeWithRerun uses a combination of class and
method names to rerun the tests. It could make use of the Filterable interface
of the Runner and use the Filter#matchMethodDescription to create a filter from
the descriptions of the failing tests to avoid the problems above.
was (Author: mpkorstanje):
It appears that JUnit4Provider#executeWithRerun uses a combination of class and
method names to rerun the tests. It could however make use of the Filterable
interface of the Runner and use the Filter#matchMethodDescription to create a
filter from the descriptions of the failing tests.
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>
> Using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005492#comment-16005492
]
M.P. Korstanje commented on SUREFIRE-1372:
--
It appears that JUnit4Provider#executeWithRerun uses a combination of class and
method names to rerun the tests. It could however make use of the Filterable
interface of the Runner and use the Filter#matchMethodDescription to create a
filter from the descriptions of the failing tests.
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>
> Using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (SUREFIRE-1372) Rerunning failing tests fails in combination with Description#createSuiteDescription
[
https://issues.apache.org/jira/browse/SUREFIRE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
M.P. Korstanje updated SUREFIRE-1372:
-
Summary: Rerunning failing tests fails in combination with
Description#createSuiteDescription (was: Rerunning failing tests fails when
@RunWith is used)
> Rerunning failing tests fails in combination with
> Description#createSuiteDescription
>
>
> Key: SUREFIRE-1372
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
>Affects Versions: 2.20
>Reporter: M.P. Korstanje
>
> Using surefire to rerun failing tests created by a Runner that uses
> {noformat}Description#createSuiteDescription{noformat} with a human readable
> name rather then a class name the following stack trace occurs:
> {code}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create
> test class 'Scenario: Fail when running'
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (SUREFIRE-1372) Rerunning failing tests fails when @RunWith is used
M.P. Korstanje created SUREFIRE-1372:
Summary: Rerunning failing tests fails when @RunWith is used
Key: SUREFIRE-1372
URL: https://issues.apache.org/jira/browse/SUREFIRE-1372
Project: Maven Surefire
Issue Type: Bug
Components: Maven Surefire Plugin
Affects Versions: 2.20
Reporter: M.P. Korstanje
Using surefire to rerun failing tests created by a Runner that uses
{noformat}Description#createSuiteDescription{noformat} with a human readable
name rather then a class name the following stack trace occurs:
{code}
org.apache.maven.surefire.testset.TestSetFailedException: Unable to create test
class 'Scenario: Fail when running'
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:385)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
Caused by: java.lang.ClassNotFoundException: Scenario: Fail when running
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeFailedMethod(JUnit4Provider.java:379)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:292)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
