Hi, reading this discussion, it made me think of one of the (I think very important) missing features when using junit5 and surefire: you can't run a single parameterized test on the command line (-Dtest=pattern)
For junit4, there was a syntax like this (see https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#test "The Parameterized JUnit runner describes test methods using an index in brackets, so the non-regex method pattern would become: #testMethod[*]. If using @Parameters(name="{index}: fib({0})={1}") and selecting the index e.g. 5 in pattern, the non-regex method pattern would become #testMethod[5:*]." But last time I looked, the parameterized patterns were just never implemented for junit5, surefire just checks the pattern against the methodsource name: https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/TestMethodFilter.java#L51-L53 . Personally this forces me to load the project in eclipse just to be able to rightclick on one of the parameterized tests. Do you think this problem would go away as part of this migration ? I never found an issue for this but maybe someone knows if there's one already ? Otherwise I can create one if that helps. (By the way I see that the docs are outdated "-Dtest=MyTest#myMethod. This is supported for junit 4.x and TestNg." but this is also supported by junit5 tests currently. Do you want me to do a PR for the docs, or can someone else quickly udpate them ? Mentionning that junit5 doesn't support the parameterized tests pattern explicitly would also help I think, I remember losing a few hours desperately trying to get the junit4 patterns to work with junit5 parameterized tests out of ignorance) Cheers, Jon On Wed, Oct 15, 2025 at 4:24 AM Olivier Lamy <[email protected]> wrote: > > Thanks, Delany. Your message perfectly reflects my own opinion, even > as a long-time committer on the project. > > We never really closed this thread with a clear conclusion or next > steps, so I’d like to summarize the current state and propose a path > forward. > > From the discussion so far, there seems to be a general consensus on > simplifying Surefire’s codebase by migrating test execution to the > JUnit Platform engine. This would be done in a moderately conservative > way, as described earlier in this thread: > https://lists.apache.org/thread/vovj7srtpjlk67z7ocowzbz3g3z3878t > > Quick Summary of the current situation: > Surefire currently runs tests through multiple provider modules (JUnit > 3, 4, 5, TestNG, etc.). Each provider implements similar interfaces > and classes, which means that any change to reporters or listeners > requires parallel updates across all providers. > This architecture made sense in the past, when there wasn’t a unified > way to run JUnit 3, JUnit 4, TestNG, and JUnit 5 tests. However, it > now results in significant code duplication and added complexity > whenever changes are made. > > Proposed approach: > A few years ago, the JUnit team introduced the JUnit Platform, which > can run not only JUnit 5 tests (as used by the junit-platform > provider) but also JUnit 3, 4, and TestNG tests. > To simplify the codebase, we propose removing the dedicated providers > for JUnit 3, 4, and TestNG, and using the junit-platform provider as > the single unified test runner. Please note the provider architecture > is still supported so everybody is free to implement/run their own but > we need to stop maintaining so many. > > > This change would mean that JUnit versions older than 4.12 will no > longer be supported. > However, existing JUnit 3 and JUnit 4 tests (≤ 4.11) can still be > executed without modification by simply upgrading the JUnit dependency > to at least version 4.12. > > Reference: > A work-in-progress PR is available here: > https://github.com/apache/maven-surefire/pull/3179 > > Even if it’s a significant PR with still work to do (Thanks, > Sebastian, for his help), which removes a considerable amount of > legacy code, it seems to be the way to go and represents some > consensus reached with this thread. > > Onward > Olivier > > > On Thu, 9 Oct 2025 at 17:23, Delany <[email protected]> wrote: > > > > Hi Tibor, > > > > Another popular principle out there is do one thing well. > > The provider principle is already in play with Maven's plugin architecture. > > > > I would much rather have a test plugin purpose built for the test framework > > I use. > > Its fine until you want to do something a bit extra, > > https://github.com/apache/maven-surefire/issues/3146 > > > > Without knowing anything about the internals, my perception is that > > surefire is hugely complicated with a lot of needless ceremony. > > I don't feel that way about any other plugin. > > You talk about separation of concerns, but at least from the outside it > > seems so many concerns are overlapping. > > Configuration, test discovery. Threading is configurable in all three, > > junit, surefire, and maven, not to mention the unit under test. > > > > The gold standard is if test frameworks provide their own plugin so there > > is clear responsibility for it, and maybe a plugin that shares its namesake > > will encourage some buy-in. > > If the lineage of developers maintaining the plugin is patchy, what hope is > > there for supporting a particular test framework integration, > > or looking into deeper issues, or just having some time to respond? > > > > I probably shouldn't talk about contributing code, but I think designing a > > plugin that people would want to contribute should be pretty high on the > > list of priorities. > > Keeping it lean and fit for purpose should keep the number of releases down > > too. Its a busy plugin. Probably the busiest? > > > > Delany > > > > On Wed, 8 Oct 2025 at 21:30, Tibor Digana <[email protected]> wrote: > > > > > @Martin Desruisseaux, > > > > > > >> Even if Maven wanted to be independent of JUnit 5... > > > I know Surefire and JUnit5 quite well, no need to tell me, but I was not > > > talking about technical things. I am talking about totally different > > > topics, especially about good Surefire purpose and design with Providers, > > > risk management and the whole globe of users. > > > JUnit5 is not a principal shift. > > > > > > - Supporting only JUnit5 f/w, deleting Provider API, deleting JUnit3 + > > > 4 + TestNG is intentionally a bad idea. > > > The thing happening now goes against the historical continuity because > > > previous two developers left ASF. > > > - The Surefire originators who designed the architecture had a good > > > reasons to introduce Provider API, it is a good design. It separates > > > concerns of the plugin from the concerns of test frameworks. Although > > > the > > > test frameworks have very similar Listeners, they are cohesive with > > > Surefire Listener and this enables Maven + plugin to be independent of > > > test > > > framework implementation across vendors and versions and third party > > > policies (pricing, license, SDLC) and this provides us with certain > > > freedom > > > meaning no coupling (technical coupling and team coupling) between > > > Maven > > > plugin and third party frameworks. > > > > > > >> Keeping direct support of JUnit 3 (as opposed to indirect support > > > through JUnit 5 API) has a cost. > > > No, this is not true. > > > I was a developer of Surefire for many years and JUnit3 + POJO *Provider* > > > was stable which means I did not need to touch it for quite a long time. > > > This Provider is pretty simple for simple tests, and the positive is that > > > the POJO does not require JUnit, this is good for lightweight test > > > projects. > > > > > > > > > >> Of course there is a risk that we break something. ... difficult to > > > implement. > > > It is not difficult to implement. I know Surefire like my shoes. > > > My problem is that I left during Covid and the historical continuity with > > > new developers was broken, that's my fault, I am honest now. If this did > > > not happen then most probably Slawomir would continue straight ahead and > > > others maybe too. > > > > > > Cheers > > > Tibor > > > > > > On Tue, Oct 7, 2025 at 8:22 PM Martin Desruisseaux > > > <[email protected]> wrote: > > > > > > > Hello Tibor > > > > > > > > Even if Maven wanted to be independent of JUnit 5, Maven does not need > > > > to invent its own test API. JUnit5 has a separation between API and > > > > implementations, so it would be possible to use only the API part if > > > > necessary (I don't think that it would be necessary, just mentioning > > > > that we have this possibility). Therefore, a full switch to JUnit 5 > > > > would not necessarily make us fully prisoner of JUnit 5. > > > > > > > > Keeping direct support of JUnit 3 (as opposed to indirect support > > > > through JUnit 5 API) has a cost. For example, I saw requests on this > > > > mailing list for testing the different versions of a multi-release JAR > > > > file. Currently, any classes in META-INF/versions/* will be ignored at > > > > testing time, because Surefire runs the tests with the `*.class` files > > > > on the class-path while multi-release works only with JAR files. Last > > > > Saturday, I wanted to start looking for the possibility of adding an > > > > option for running the tests many times with different > > > > `META-INF/versions/*` directories added to the class-path. It was not > > > > easy to find my way in the Surefire code, and I rely a lot on the > > > > simplification proposed by Romain before to make any other attempt. > > > > > > > > Of course there is a risk that we break something. But the example given > > > > in the previous email (tests as POJO) should be easy to port. The > > > > alternative (not removing direct JUnit 3 support) is more like freezing > > > > the Surefire plugin: no accidental lost of feature, but also less new > > > > features (e.g. easier testing of multi-release projects) because they > > > > would be more difficult to implement. > > > > > > > > Martin > > > > > > > > > > > > Le 07/10/2025 à 19:16, Tibor Digana a écrit : > > > > > Romain, > > > > > > > > > > to be honest the JUnit guys always wanted to make their own business > > > and > > > > > monopoly over the testing phase in Maven, which means using JUnit5 > > > only, > > > > > that's it. > > > > > > > > > > I have been a JUnit4 developer since cca 2011 or so, and we know each > > > > other. > > > > > If I say business then I really mean business and not only the word. > > > > > Defining report schema and forking mechanism JVMs in JUnit5, these are > > > > > still the same competition problems between us, and I say it's about > > > the > > > > > monopoly, which means who takes the control over these things takes > > > full > > > > > control over Maven testing and then Maven becomes completely dependent > > > on > > > > > some test framework which is the risk for Maven. > > > > > > > > > > This is the main problem but you wouldn't see this unless you have > > > spent > > > > > years and years developing both parties as I have. > > > > > > > > > > Making Maven strictly dependent only on JUnit5 is the worst decision > > > > > ever, I am telling you! > > > > > Btw, experienced Maven guys must know what a strict dependency means > > > and > > > > > the consequences too. > > > > > We are still in the loop for years with deleting something because > > > > somebody > > > > > has got a crazy idea in the morning, found out github/dev-factory or > > > > > anything else, some tool or technology as many other, those tools > > > > > which > > > > > come and go every year. > > > > > > > > > > On Mon, Sep 29, 2025 at 9:09 AM Romain Manni-Bucau < > > > > [email protected]> > > > > > wrote: > > > > > > > > > >> Hi all, > > > > >> > > > > >> I'd like to start a thread about potentially dropping surefire > > > totally. > > > > >> The rational is that surefire (and failsafe) are mainly an > > > > >> abstraction > > > > >> layer on top of main test providers. > > > > >> However, since JUnit5 the platform/engine is itself such an > > > abstraction > > > > >> layer and a runner. > > > > >> > > > > >> On another side, testng and junit4 are slowly getting abandonned - > > > even > > > > EE > > > > >> TCK started to move. > > > > >> > > > > >> In terms of additional features we do have the maven site integratoin > > > - > > > > but > > > > >> I doubt it is much used outside and to be honest it can be replaced > > > > with a > > > > >> github/dev-factory link with more benefit these days. > > > > >> > > > > >> So overall I think we can converge by dropping surefire plugin in > > > favor > > > > of > > > > >> a thin wrapper of junit5 console runner ([1]). > > > > >> > > > > >> Short terms I'm sure Christian could help us getting something fast > > > > based > > > > >> on its implementation ([2] - including a small surefire compatibility > > > > mode) > > > > >> and long term it will reduce the maintenance cost we do have for a > > > very > > > > >> poor gain in current world (site and remoting are no more key > > > > >> features > > > > >> thanks the CI and doc evolution). > > > > >> > > > > >> Wdyt? Is maven 4 the mometum to do it? > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
