On Fri, Sep 10, 2021 at 5:06 PM Tibor Digana <tibordig...@apache.org> wrote:
> If you use forkCount > 1, the Surefire loads test classes via load > balancer. > If you use default forkCount = 0, all the classes are run eagerly as a > suite via JUnit5 Launcher in one shot. > Yes, this is what I saw in the JUnitPlatformProvider. The cause of my issues lies in the difference in events triggered by Arquillian due to the classes executed one by one. The demo-project attached to the ticket demonstrates this in just a few lines of code. If you are aiming for Arquillian, testing the applications in the > application server, you should use maven-failsafe-plugin which has another > testing model. > See the plugin goals of Failsafe plugin. Maven Failsafe Plugin – Plugin > Documentation (apache.org) > <https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html> > https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html > There are Maven phases: > pre-integration-test > integration-test > post-integration-test > verify > > Therefore you should start the application server in the phase > pre-integration-test. > Accordingly, you should stop it in the phase post-integration-test. > > Then use Failsafe plugin in the phases integration-test and verify. > Unfortunately, this is not how arquillian is setup. You can run tests against running servers (which you can then start in pre-integration-test), but using arquillian cube, these containers can be started as part of the test lifecycle. Also, Arquillian is just a single incarnation of the underlying issue. Every extension that takes more than a few seconds to start and is built to only initialize once for a suite will trigger this same problem (as can be seen by the demo project). As surefire and failsafe use the same JUnitPlatformProvider, the problem will also be the same. What do you think about the solution of using a LauncherSession to bind the multiple executions together? This will require JUnit Platform 1.8 and changes in the JUnitPlatformProvider. However, I'm not sure what the intended audience of LauncherSessionListener is. Best regards, Emond On Fri, Sep 10, 2021 at 12:42 PM Emond Papegaaij <emond.papega...@gmail.com> > wrote: > > > Hi all, > > > > First of all, sorry for the lengthy post. I decided to add some context > to > > explain things a bit, but it resulted in quite a long e-mail. For the > past > > few weeks I've been trying to come up with a solution for the issue I > > filled under SUREFIRE-1935, but I'm getting stuck and starting to feel > like > > the issue cannot be solved with the current JUnit Platform API. To give > > some perspective into why this issue is important for us, I first have to > > explain a bit about our setup. > > > > We write our tests in the Spock framework and use Arquillian to run the > > tests in the application container. Some of our tests, especially the > > Selenium based tests, require quite some setup and tear down. Prior to > > starting the test, Arquillian cube builds and starts several docker > > containers and the tests are run against these containers. We currently > use > > the JUnit 4 based Spock 1.3 with Groovy 2.5, but we like to upgrade to > > Spock 2 and Groovy 3, which runs on top of the JUnit Platform. For this, > > I've already started working on a Spock extension that integrates Spock 2 > > in the Arquillian test life cycle [1]. This extension is inspired by the > > (currently Alpha) JUnit5 module for Arquillian [2]. Both use a global > > registration to keep track of the state managed by Arquilllian. This will > > end up somewhere at the root of the TestPlan (for example see [3]). > > > > Because our tests are quite extensive, with a total run time of 6 hours, > we > > run them with a forkCount of 8, greatly reducing the total duration. > > However, this is where SUREFIRE-1935 comes into play. With a forkCount > > 1, > > the entire test life cycle is started over and over again for every test > > class. This happens in JUnitPlatformProvider at line 197 [4]. This > results > > in the entire Arquillian suite being torn down and setup for every class, > > in our case adding several minutes to the execution of every test class > > because the docker setup is done over and over again. > > > > To overcome this issue, the state from one test class execution has to be > > carried to the next. It seems the LauncherSession (introduced in JUnit > 5.8) > > is meant to close this gap. However, this would mean my extension would > > also need to implement a LauncherSessionListener, and I'm not sure if > > extensions are supposed to integrate with the launcher as well. Also, for > > this surefire would need to start a session prior to the tests, and close > > it when done. I think this is a good idea anyway when running on platform > > 1.8 or higher. > > > > Another solution could be a (sort of) dynamic test that produces the > tests > > to be run one by one. However, here my knowledge of JUnit really falls > > short. I've got no idea of this is even possible. > > > > I hope someone can help me out on this one and point me in the right > > direction, as we like to upgrade our test frameworks and this is blocking > > us at the moment. > > > > Best regards, > > Emond Papegaaij > > > > [1] Arquillian extension for Spock Framework 2: > > > > > https://github.com/topicusonderwijs/arquillian-testrunner-spock/tree/spock-2.0-junit5 > > [2] Arquillian module for JUnit 5: > > https://github.com/arquillian/arquillian-core/tree/master/junit5 > > [3] Registration in root store: > > > > > https://github.com/arquillian/arquillian-core/blob/master/junit5/core/src/main/java/org/jboss/arquillian/junit5/JUnitJupiterTestClassLifecycleManager.java#L20 > > [4] JUnitPlatformProvider launching the tests: > > > > > https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProvider.java#L197 > > >
