colinwerner opened a new issue, #3510: URL: https://github.com/apache/maven-surefire/issues/3510
Upgrading 3.5.6 → 3.6.0 changed two runOrder behaviours that neither the 3.6.0 What's New page nor #3179 mentions. 1. runOrder now applies to methods within a class, not only to classes. In 3.5.6, the JUnit Platform provider ordered only classes, via RunOrderCalculator.orderTestClasses(). Methods kept JUnit Jupiter's default order. In 3.6.0, JUnitPlatformProvider.setupRunOrder() also sets junit.jupiter.testmethod.order.default: ``` ┌─────────────────────┬────────────────────────────────────┐ │ runOrder │ method orderer set by 3.6.0 │ ├─────────────────────┼────────────────────────────────────┤ │ alphabetical │ MethodOrderer$MethodName │ ├─────────────────────┼────────────────────────────────────┤ │ reversealphabetical │ ReverseOrdering$ReverseMethodOrder │ ├─────────────────────┼────────────────────────────────────┤ │ random │ MethodOrderer$Random │ └─────────────────────┴────────────────────────────────────┘ ``` As a result, a suite that passed on 3.5.6 with -DrunOrder=alphabetical can fail on 3.6.0 with the same setting and the same dependencies. That happens whenever its test methods quietly depend on the old default method order, for example @TestInstance(PER_CLASS) classes with shared fixtures. In our case, a 4,608-test suite went from 0 failures to 4–5 failures per run. The change itself is reasonable, but it is silent and breaking for users who set runOrder explicitly. 2. hourly was removed. RunOrder no longer has HOURLY, so builds that set <runOrder>hourly</runOrder> now fail at the test phase: There's no RunOrder with the name hourly. Use one of the following RunOrders: alphabetical, filesystem, random, reversealphabetical, balanced, failedfirst. Requests: - Add both changes to the 3.6.0 What's New / migration notes. - Update the runOrder parameter docs to say it controls method order too for JUnit Platform, and how to opt out. For example: can a user's configurationParameters or junit-platform.properties override junit.jupiter.testmethod.order.default? - If removing hourly was deliberate, mention what replaces it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
