[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 Not at this moment. Perhaps the weekend. ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 @Tibor17 It appears this branch is still up to date with master. Is the removal of the class the problem or the underlying fundamentals? If it is just the removal I can restore `FailingMethodFilter.java`. It went unused after my changes. If it is the underlying fundamentals please let me know how I can help you. ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue:
https://github.com/apache/maven-surefire/pull/150
> How is Description constructed now by Cucumber?
Executing cucumber while using JUnit creates a tree of Descriptions like so:
```
Cucumber
|- Feature 1
| | - Scenario 1
| | - Scenario 2
| | - Scenario 3
|- Feature 2
| | - Scenario 4
| | - Scenario 5
| | - Scenario 6
```
The root description describes the [Cucumber JUnit
runner](https://github.com/cucumber/cucumber-jvm/blob/v2.2.0/junit/src/main/java/cucumber/api/junit/Cucumber.java)
and is created in [JUnits
ParentRunner](https://github.com/junit-team/junit4/blob/r4.12/src/main/java/org/junit/runners/ParentRunner.java#L349).
using
[Description.createSuitDescription(String)](https://github.com/junit-team/junit4/blob/r4.12/src/main/java/org/junit/runner/Description.java#L44).
This results in a `Description` with `fUniqueId` equal to the name of the
runner. This is always the fqn of the JUnit test annotated with
`@RunWith(Cucumber.class)`.
The Feature Descriptions are created in the
[FeatureRunner](https://github.com/cucumber/cucumber-jvm/blob/v2.2.0/junit/src/main/java/cucumber/runtime/junit/FeatureRunner.java#L44)
using [Description.createSuitDescription(String,
Serializable)](https://github.com/junit-team/junit4/blob/r4.12/src/main/java/org/junit/runner/Description.java#L57).
The name is name of the, which is used for display purposes but otherwise
irrelevant. For the extra `Serializable` parameter we provide a
[FeatureId](https://github.com/cucumber/cucumber-jvm/blob/v2.2.0/junit/src/main/java/cucumber/runtime/junit/FeatureRunner.java#L105).
This results in a `Description` with `fUniqueId` equal to the `FeatureId` of
that feature. This `FeatureId` contains the features uri which will satisfy
equality between different (repeated) executions of the same test class.
The same reasoning applies to the creation of the scenario description.
Each `Description` is created with a `PickleId` that contains the uri of that
scenario. The actual implementation is slightly more complex so I'd like to
omit it for brevity.
As such we end up with:
```
Cucumber Description.fUniqueId =
"com.example.package.of.my.RunnerTest"
|- Feature 1 Description.fUniqueId =
FeatureId("path/to/my/cucumber.feature1")
| | - Scenario 1 Description.fUniqueId =
PickleId("path/to/my/cucumber.feature1:1")
| | - Scenario 2 Description.fUniqueId =
PickleId("path/to/my/cucumber.feature1:2")
| | - Scenario 3 Description.fUniqueId =
PickleId("path/to/my/cucumber.feature1:3")
|- Feature 2 Description.fUniqueId =
FeatureId("path/to/my/cucumber.feature2")
| | - Scenario 4 Description.fUniqueId =
PickleId("path/to/my/cucumber.feature2:1")
| | - Scenario 5 Description.fUniqueId =
PickleId("path/to/my/cucumber.feature2:2")
| | - Scenario 6 Description.fUniqueId =
PickleId("path/to/my/cucumber.feature2:3")
```
> Does it contain the real method name or it is a scenario text?
No. It does not contain a method name. A scenario does not have a
one-to-one mapping with any methods. Instead we use uri's to reference a
scenario. Note that if we were to rewrite the above suite of features to a
suite of JUnit tests, the underlying `Description` tree would have a similar
identity.
```
TestSuite Description.fUniqueId =
"com.example.package.of.my.TestSuite"
|- Feature 1 Description.fUniqueId =
"com.example.package.of.my.Feature1"
| | - Scenario 1 Description.fUniqueId =
"scenario1(com.example.package.of.my.Feature1)"
| | - Scenario 2 Description.fUniqueId =
"scenario2(com.example.package.of.my.Feature1)"
| | - Scenario 3 Description.fUniqueId =
"scenario3(com.example.package.of.my.Feature1)"
|- Feature 2 Description.fUniqueId =
"com.example.package.of.my.TestSuite"
| | - Scenario 4 Description.fUniqueId =
"scenario1(com.example.package.of.my.Feature2)"
| | - Scenario 5 Description.fUniqueId =
"scenario2(com.example.package.of.my.Feature2)"
| | - Scenario 6 Description.fUniqueId =
"scenario3(com.example.package.of.my.Feature2)"
```
---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 > Additionally, can you write a separate documentation file cucumber.apt.vm for Cucumber with your best practices to setup and use cucumber in terms of maven-surefire-plugin and maven-failsafe-plugin tests, annotations like CucumberOptions, dependencies, versions, configuration if any, and re-run notices? Sure! ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 @Tibor17 please let me know if you have more questions. ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 I haven't answered yet is why cucumber creates descriptions with a unique Id and doesn't just use the method names. The reason for that is that scenarios in cucumber don't need to have unique names. While we could use a URI instead this would result in a rather hideous display name. We like to keep things friendly so we use the Id. Other use cases include frameworks that repeatedly execute the same test method with different parameters. Each invocation would have it's own id while the method and test name would be the same. ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 > Why you have chosen provider surefire-junit47 and not also the surefire-junit4? While it would be possible to change surefire-junit4 to filter by description there is no need to. Prior to JUnit 4.11 descriptions use their display name to test for equality. The display name was composed of a method name and a class name. As such filtering by a description created from the method and class name of a failed test and filtering by the description of that failed test will yield the same result. ```java Description failedTestDescription = Description.createTestDescription(Test.class, "shouldTestSomething"); Description descriptionFromFailedMethodAndClassName = Description.createTestDescription(Test.class, "shouldTestSomething"); assertEquals(failedTestDescription, descriptionFromFailedMethodAndClassName); assertEquals(failedTestDescription.getDisplayName(), descriptionFromFailedMethodAndClassName.getDisplayName()); ``` > Why the method generateFailingTestDescriptions has to return Set and not the previous Map<Class, Set>? With the introduction of the `fUniqueId` property to test for equality between descriptions in JUnit 4.11 filtering by a description created from the method and class name of a failed test and filtering by the description of that failed test are no longer the same. Hence the need to use the description of the failed test instead of its method and class name. For example: ```java Description failedTestDescription = Description.createTestDescription(Test.class.getCanonicalName(), "shouldTestSomething", "my-test-id"); Description descriptionFromFailedMethodAndClassName = Description.createTestDescription(Test.class, "shouldTestSomething"); assertNotEquals(failedTestDescription, descriptionFromFailedMethodAndClassName); assertEquals(failedTestDescription.getDisplayName(), descriptionFromFailedMethodAndClassName.getDisplayName()); ``` ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 You're welcome! ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 Merged in 0a81c48971e3ee4b79b0858be7b8f4b13ece7287. ---
[GitHub] maven-surefire pull request #150: SUREFIRE-1372 Filter tests to be rerun by ...
Github user mpkorstanje closed the pull request at: https://github.com/apache/maven-surefire/pull/150 ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 > Then try to alter the behavior of the integration test with Maven profile (junit4 and junit47). We are already doing it, see addGoal( "-P junit4" ), addGoal( "-P junit47" ). > This would lead to @Parameterized IT which exists in our code already, see FailFastJUnitIT as a hint. Rerunning failed tests based on their description is not possible with the junit4 provider. Only the junit47 provider and up would allow us to rerun tests based on their description rather then methods. So I am unsure about the utility of creating a parameterized test. ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 @Tibor17 `MatchDescriptions` uses `Filter.matchMethodDescription` which does not exist in junit4. It would make the junit 4 version of `MatchDescriptions` divergent enough to duplicate its functionality. I've added an extra commit onto this PR that shows just that. ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 > Can you write the documentation? Done. >Fix the checkstyle in your integration test and the classes FlakeCucumberTest. I seem to be unable to get checkstyle to scan these tests and classes. I've visually checked these files. ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue:
https://github.com/apache/maven-surefire/pull/150
@Tibor17 that should be doable. Replacing the inner most loop in the
`JUnit4Provider.executeWithRerun` with the code below should do it. This would
require the `FilterFactory` to be moved to another common module. I can't
quite tell what the the nock-on effects of that would be.
```java
for ( int i = 0; i < rerunFailingTestsCount &&
!failureListener.getAllFailures().isEmpty(); i++ )
{
Set failures = generateFailingTestDescriptions(
failureListener.getAllFailures() );
failureListener.reset();
FilterFactory filterFactory = new FilterFactory( testClassLoader );
Filter failingMethodsFilter =
filterFactory.createMatchMethodDescriptionsFilter( failures );
Request.aClass( clazz ).filterWith( failingMethodsFilter
).getRunner().run( rerunNotifier );
}
```
---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 > The build passed successfully. I will refactor little details, I will squash our commits and then I will push it to master. Cheers. > I could not find a documentation for configuring annotation I've changed the layout of the project to match the default used by cucumber. When using the default layout `@CucumberOptions` is not needed. Cucumber uses the location of the runner class to figure out where the features and glue are. All in all `@CucumberOptions` influences which features are included but doesn't change how cucumber presents itself to surefire. From surefires perspective it should be yet another junit test suite. > One more question I have is regarding artifacts of Cucumber. Which one is right to use: `cucumber-junit` provides integration with junit and should be used when junit is to be used. `cucumber-java` provides java annotations to denote step definitions. `cucumber-java8` depends on `cucumber-java` and adds lambda based step definitions. Using `cucumber-java` is sufficient. ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 Bit hard to say without the full stack trace, components and version numbers involved. To fix this bug I also had to fix cucumber to ensure that the it provided the proper ids to [Description.createSuiteDescription](https://github.com/junit-team/junit4/blob/r4.12/src/main/java/org/junit/runner/Description.java#L57). The ids had to be unique in a single run but when a suite was recreated they had to be the same such as to satisfy the equals and hashcode method. ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 In that case yes, you only need to wait for the release of surefire. https://github.com/cucumber/cucumber-jvm/issues/1120 was fixed by https://github.com/cucumber/cucumber-jvm/pull/1134 and is part of cucumber-jvm 2.x.x. ---
[GitHub] maven-surefire issue #150: SUREFIRE-1372 Filter tests to be rerun by descrip...
Github user mpkorstanje commented on the issue: https://github.com/apache/maven-surefire/pull/150 The path: ``` F:\jenkins\jenkins-slave\workspace\fire-pipeline_SUREFIRE-1463-5KZ5VLS45OW347227XHB7NPF5FLJFXODCMZF4ZDVYJEJZVION6UQ\build\surefire-integration-tests\target\TestMultipleMethodPatternsTestNGIT_shouldMatchSimpleClassNameAndMethodWithJavaExtWildcardPkg_testng-test ``` Contains 261 characters. MS allows for a maximum of 260 characters including the drive designation. Assuming all the other parts are not under your control you might want to shorten the method name a bit. ---
