M.P. Korstanje created SUREFIRE-1691: ----------------------------------------
Summary: surefire-junit47 reports results against incorrect test Key: SUREFIRE-1691 URL: https://issues.apache.org/jira/browse/SUREFIRE-1691 Project: Maven Surefire Issue Type: Bug Components: Junit 4.x support Affects Versions: 3.0.0-M3 Reporter: M.P. Korstanje This is a follow up on: - https://issues.apache.org/jira/browse/SUREFIRE-1156 - [https://github.com/cucumber/cucumber-jvm/issues/865] The latest reproducer can be found here: - [https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2] In short Cucumber has a hierarchy of tests that looks at follows: {noformat} Class annotated with @RunWith(Cucumber.class) |- Feature File 1 | |- Scenario 1a | |- Scenario 1b |- Feature File 2 | |- Scenario 2a | |- Scenario 2b {noformat} Surefire tries to group output in tests sets, either per class or per test suite. As Cucumber currently only emits test started/finished events for Scenarios sure fire can not group scenarios into a test set. They are instead group with the previous test set. However even after adding test started/finished events for features results in a single test set. I believe there is an error in {{NonConcurrentRunListener.describesNewTestSet}}. {code:java} private boolean describesNewTestSet( Description description ) { if ( currentTestSetDescription != null ) { if ( null != description.getTestClass() ) { return !description.getTestClass().equals( currentTestSetDescription.getTestClass() ); } else if ( description.isSuite() ) { return description.getChildren().equals( currentTestSetDescription.getChildren() ); } return false; } return true; } {code} The value of {{description.getChildren().equals( currentTestSetDescription.getChildren() );}} should be negated so so different test sets are properly recognized. Fixing this locally works and can be reproduced by: 1. Negating the line and building sure fire locally 2. Building [https://github.com/cucumber/cucumber-jvm/pull/1765] with `mvn clean install -DskipTests` 3. Running `mvn clean test -Pjunit47` on [https://github.com/mpkorstanje/surefire-test] It does however does result in another unwanted result: A feature file with two scenarios is now reported as having 3 results. One for each scenario and one for the feature. {noformat} ------------------------------------------------------------------------------- Test set: Some Feature 1 ------------------------------------------------------------------------------- Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in Some_Feature_1 {noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003)