Sorry for typos, I sent it from my mobile. T On Fri, Sep 10, 2021 at 6:18 PM Tibor Digana <tibordig...@apache.org> wrote:
> We have to dig into Junit 5. Surefire is a streamer of classes across the > forks which is our load balancer. Therefore each class is running > separately, pre and post fixtures. If the Junit 5 used Java Streamer > including dome kind of control of fixtures then web would have this issue. > > Dňa pi 10. 9. 2021, 18:03 Emond Papegaaij <emond.papega...@gmail.com> > napísal(a): > >> On Fri, Sep 10, 2021 at 5:06 PM Tibor Digana <tibordig...@apache.org> >> wrote: >> >> > If you use forkCount > 1, the Surefire loads test classes via load >> > balancer. >> > If you use default forkCount = 0, all the classes are run eagerly as a >> > suite via JUnit5 Launcher in one shot. >> > >> >> Yes, this is what I saw in the JUnitPlatformProvider. The cause of my >> issues lies in the difference in events triggered by Arquillian due to the >> classes executed one by one. The demo-project attached to the ticket >> demonstrates this in just a few lines of code. >> >> If you are aiming for Arquillian, testing the applications in the >> > application server, you should use maven-failsafe-plugin which has >> another >> > testing model. >> > See the plugin goals of Failsafe plugin. Maven Failsafe Plugin – Plugin >> > Documentation (apache.org) >> > < >> https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html> >> > >> https://maven.apache.org/surefire/maven-failsafe-plugin/plugin-info.html >> > There are Maven phases: >> > pre-integration-test >> > integration-test >> > post-integration-test >> > verify >> > >> > Therefore you should start the application server in the phase >> > pre-integration-test. >> > Accordingly, you should stop it in the phase post-integration-test. >> > >> > Then use Failsafe plugin in the phases integration-test and verify. >> > >> >> Unfortunately, this is not how arquillian is setup. You can run tests >> against running servers (which you can then start in >> pre-integration-test), >> but using arquillian cube, these containers can be started as part of the >> test lifecycle. Also, Arquillian is just a single incarnation of the >> underlying issue. Every extension that takes more than a few seconds to >> start and is built to only initialize once for a suite will trigger this >> same problem (as can be seen by the demo project). As surefire and >> failsafe >> use the same JUnitPlatformProvider, the problem will also be the same. >> >> What do you think about the solution of using a LauncherSession to bind >> the >> multiple executions together? This will require JUnit Platform 1.8 and >> changes in the JUnitPlatformProvider. However, I'm not sure what the >> intended audience of LauncherSessionListener is. >> >> Best regards, >> Emond >> >> On Fri, Sep 10, 2021 at 12:42 PM Emond Papegaaij < >> emond.papega...@gmail.com> >> > wrote: >> > >> > > Hi all, >> > > >> > > First of all, sorry for the lengthy post. I decided to add some >> context >> > to >> > > explain things a bit, but it resulted in quite a long e-mail. For the >> > past >> > > few weeks I've been trying to come up with a solution for the issue I >> > > filled under SUREFIRE-1935, but I'm getting stuck and starting to feel >> > like >> > > the issue cannot be solved with the current JUnit Platform API. To >> give >> > > some perspective into why this issue is important for us, I first >> have to >> > > explain a bit about our setup. >> > > >> > > We write our tests in the Spock framework and use Arquillian to run >> the >> > > tests in the application container. Some of our tests, especially the >> > > Selenium based tests, require quite some setup and tear down. Prior to >> > > starting the test, Arquillian cube builds and starts several docker >> > > containers and the tests are run against these containers. We >> currently >> > use >> > > the JUnit 4 based Spock 1.3 with Groovy 2.5, but we like to upgrade to >> > > Spock 2 and Groovy 3, which runs on top of the JUnit Platform. For >> this, >> > > I've already started working on a Spock extension that integrates >> Spock 2 >> > > in the Arquillian test life cycle [1]. This extension is inspired by >> the >> > > (currently Alpha) JUnit5 module for Arquillian [2]. Both use a global >> > > registration to keep track of the state managed by Arquilllian. This >> will >> > > end up somewhere at the root of the TestPlan (for example see [3]). >> > > >> > > Because our tests are quite extensive, with a total run time of 6 >> hours, >> > we >> > > run them with a forkCount of 8, greatly reducing the total duration. >> > > However, this is where SUREFIRE-1935 comes into play. With a >> forkCount > >> > 1, >> > > the entire test life cycle is started over and over again for every >> test >> > > class. This happens in JUnitPlatformProvider at line 197 [4]. This >> > results >> > > in the entire Arquillian suite being torn down and setup for every >> class, >> > > in our case adding several minutes to the execution of every test >> class >> > > because the docker setup is done over and over again. >> > > >> > > To overcome this issue, the state from one test class execution has >> to be >> > > carried to the next. It seems the LauncherSession (introduced in JUnit >> > 5.8) >> > > is meant to close this gap. However, this would mean my extension >> would >> > > also need to implement a LauncherSessionListener, and I'm not sure if >> > > extensions are supposed to integrate with the launcher as well. Also, >> for >> > > this surefire would need to start a session prior to the tests, and >> close >> > > it when done. I think this is a good idea anyway when running on >> platform >> > > 1.8 or higher. >> > > >> > > Another solution could be a (sort of) dynamic test that produces the >> > tests >> > > to be run one by one. However, here my knowledge of JUnit really falls >> > > short. I've got no idea of this is even possible. >> > > >> > > I hope someone can help me out on this one and point me in the right >> > > direction, as we like to upgrade our test frameworks and this is >> blocking >> > > us at the moment. >> > > >> > > Best regards, >> > > Emond Papegaaij >> > > >> > > [1] Arquillian extension for Spock Framework 2: >> > > >> > > >> > >> https://github.com/topicusonderwijs/arquillian-testrunner-spock/tree/spock-2.0-junit5 >> > > [2] Arquillian module for JUnit 5: >> > > https://github.com/arquillian/arquillian-core/tree/master/junit5 >> > > [3] Registration in root store: >> > > >> > > >> > >> https://github.com/arquillian/arquillian-core/blob/master/junit5/core/src/main/java/org/jboss/arquillian/junit5/JUnitJupiterTestClassLifecycleManager.java#L20 >> > > [4] JUnitPlatformProvider launching the tests: >> > > >> > > >> > >> https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProvider.java#L197 >> > > >> > >> >
