So we took a look at this. First we wanted to know why the forked goals were executed multiple times, while MNG-6566 [1] should have taken care of that. But it seems that that only optimizes duplicate goals in the context of one project. I attempted to explain that [2]. We created a ticket for optimizing this for multi module projects [3], but we think that it's not straightforward to fix this.
Back to the problem at hand. IIUC your suggestion is to pause project builds halfway, resume them after the required child projects goals have been executed (instead of forking), and finally finish the child project builds afterwards. I think it's elegant that in this solution, no duplicate goals will be executed, since no real forking has to be done. I do have the feeling that it would become quite a drastic change. I am also not sure whether we can assume that switching halfway through project builds works fine in every scenario. For the user it would also become hard to follow the build, I guess... Perhaps it would be easier to ensure projects can only be built once at a time (e.g. using locks or something). It wouldn't be the most efficient, but I guess it does not involve big changes to both the inner workings and user perspective. Martin [1] https://issues.apache.org/jira/browse/MNG-6566 [2] https://issues.apache.org/jira/browse/MNG-6566?focusedCommentId=17353384&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17353384 [3] https://issues.apache.org/jira/browse/MNG-7163 Op di 18 mei 2021 om 21:45 schreef Martin Kanters <martinkant...@apache.org >: > Ah, too bad. I thought it was relevant. Perhaps I can take a look at this > with Maarten Mulders, but that will not be in the coming week at least. > > Op di 18 mei 2021 om 21:34 schreef Guillaume Nodet <gno...@apache.org>: > >> I just tried with master and the problem is the same. >> The reason is that in my use case, the forked goals are executed before >> the >> standard executions, and not even in the same project. >> >> Le mar. 18 mai 2021 à 20:41, Martin Kanters <martinkant...@apache.org> a >> écrit : >> >> > Not sure how I've missed this post. Have you tried this build with the >> > master build of Maven? >> > MNG-6566 [1] should prevent any unnecessary double executions, thus >> > optimizing the buildplan. >> > I'm interested if this will solve the problem at hand. >> > >> > Martin >> > >> > [1] https://issues.apache.org/jira/browse/MNG-6566 >> > >> > Op di 18 mei 2021 om 14:57 schreef Guillaume Nodet <gno...@apache.org>: >> > >> > > I'm looking a bit at aggregator goals. >> > > At first glance, it seems most of the problem comes from the fact that >> > the >> > > build ordering is done mostly per-project. This causes obvious >> problems >> > > for aggregators when a mojo needs a given phase to be executed for all >> > > children before it can run (for example the compilation phase before >> > > creating the javadoc). Another problem comes from the clean task >> which >> > has >> > > obvious side effects. Calling clean with an aggregator goal involved >> in >> > > the build seems like a recipe for problems... >> > > >> > > Would you see bad side effects to split the build in smaller chunks >> > > (basically down to a mojo execution) so that ordering can be changed >> in a >> > > more meaningful way ? For example when running an aggregate javadoc, >> the >> > > top level project would be built first until before the javadoc >> aggregate >> > > goal. The build would then go to all other projects (like the forked >> > > lifecycle) until all required goals have been run, then resume from >> the >> > > javadoc goal. >> > > Note that with the current behavior, when you run "clean verify", the >> > > execution is the following >> > > >> > > [INFO] Scanning for projects... >> > > [INFO] >> > > >> ------------------------------------------------------------------------ >> > > [INFO] Reactor Build Order: >> > > [INFO] >> > > [INFO] forked >> > > [pom] >> > > [INFO] forked-mod1 >> > > [jar] >> > > [INFO] forked-mod2 >> > > [jar] >> > > [INFO] >> > > [INFO] ---------------< org.mvndaemon.mvnd.test.forked:forked >> > > >---------------- >> > > [INFO] Building forked 0.0.1-SNAPSHOT >> > > [1/3] >> > > [INFO] --------------------------------[ pom >> > > ]--------------------------------- >> > > [INFO] >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked --- >> > > [INFO] >>> maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) > >> > > compile @ forked >>> >> > > [INFO] >> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > > [INFO] Forking forked-mod1 0.0.1-SNAPSHOT >> > > [INFO] >> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ >> > > forked-mod1 --- >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> > > forked-mod1 --- >> > > [INFO] >> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > > [INFO] Forking forked-mod2 0.0.1-SNAPSHOT >> > > [INFO] >> > > >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ >> > > forked-mod2 --- >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> > > forked-mod2 --- >> > > [INFO] <<< maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) < >> > > compile @ forked <<< >> > > [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) @ >> > > forked --- >> > > [INFO] >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod1 >> > > >------------- >> > > [INFO] Building forked-mod1 0.0.1-SNAPSHOT >> > > [2/3] >> > > [INFO] --------------------------------[ jar >> > > ]--------------------------------- >> > > [INFO] >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod1 >> --- >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ >> > > forked-mod1 --- >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> > > forked-mod1 --- >> > > [INFO] --- maven-resources-plugin:2.6:testResources >> > (default-testResources) >> > > @ forked-mod1 --- >> > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile >> (default-testCompile) >> > @ >> > > forked-mod1 --- >> > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ >> forked-mod1 >> > > --- >> > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod1 --- >> > > [INFO] >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod2 >> > > >------------- >> > > [INFO] Building forked-mod2 0.0.1-SNAPSHOT >> > > [3/3] >> > > [INFO] --------------------------------[ jar >> > > ]--------------------------------- >> > > [INFO] >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod2 >> --- >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ >> > > forked-mod2 --- >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> > > forked-mod2 --- >> > > [INFO] --- maven-resources-plugin:2.6:testResources >> > (default-testResources) >> > > @ forked-mod2 --- >> > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile >> (default-testCompile) >> > @ >> > > forked-mod2 --- >> > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ >> forked-mod2 >> > > --- >> > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod2 --- >> > > [INFO] >> > > >> ------------------------------------------------------------------------ >> > > [INFO] Reactor Summary for forked 0.0.1-SNAPSHOT: >> > > [INFO] >> > > [INFO] forked ............................................. SUCCESS [ >> > > 1.921 s] >> > > [INFO] forked-mod1 ........................................ SUCCESS [ >> > > 0.498 s] >> > > [INFO] forked-mod2 ........................................ SUCCESS [ >> > > 0.046 s] >> > > [INFO] >> > > >> ------------------------------------------------------------------------ >> > > [INFO] BUILD SUCCESS >> > > [INFO] >> > > >> ------------------------------------------------------------------------ >> > > [INFO] Total time: 2.535 s >> > > [INFO] Finished at: 2021-05-18T14:32:09+02:00 >> > > [INFO] >> > > >> ------------------------------------------------------------------------ >> > > >> > > As you can see, the resources and compile goals are run twice for >> each >> > > project. Furthermore, given the clean goal is in the loop, the >> compiler >> > > actually compiles things twice. >> > > >> > > The build plan could be rewritten as >> > > >> > > [INFO] Scanning for projects... >> > > [INFO] >> > > >> ------------------------------------------------------------------------ >> > > [INFO] Reactor Build Order: >> > > [INFO] >> > > [INFO] forked >> > > [pom] >> > > [INFO] forked-mod1 >> > > [jar] >> > > [INFO] forked-mod2 >> > > [jar] >> > > [INFO] >> > > [INFO] ---------------< org.mvndaemon.mvnd.test.forked:forked >> > > >---------------- >> > > [INFO] Building forked 0.0.1-SNAPSHOT >> > > [1/3] >> > > [INFO] --------------------------------[ pom >> > > ]--------------------------------- >> > > [INFO] >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked --- >> > > ... normal lifecycle interrupted because the next goal is the >> aggregator >> > > goal ... >> > > [INFO] >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod1 >> > > >------------- >> > > [INFO] Building forked-mod1 0.0.1-SNAPSHOT >> > > [2/3] >> > > [INFO] --------------------------------[ jar >> > > ]--------------------------------- >> > > [INFO] >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod1 >> --- >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ >> > > forked-mod1 --- >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> > > forked-mod1 --- >> > > ... interrupting forked-mod1 build as it reached the required point >> > > execution ... >> > > [INFO] >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod2 >> > > >------------- >> > > [INFO] Building forked-mod2 0.0.1-SNAPSHOT >> > > [3/3] >> > > [INFO] --------------------------------[ jar >> > > ]--------------------------------- >> > > [INFO] >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod2 >> --- >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ >> > > forked-mod2 --- >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> > > forked-mod2 --- >> > > ... interrupting forked-mod2 build as it reached the required point >> > > execution ... >> > > ... resuming the top level project... >> > > [INFO] >> > > [INFO] ---------------< org.mvndaemon.mvnd.test.forked:forked >> > > >---------------- >> > > [INFO] Building forked 0.0.1-SNAPSHOT >> > > [1/3] >> > > [INFO] --------------------------------[ pom >> > > ]--------------------------------- >> > > [INFO] >> > > [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) @ >> > > forked --- >> > > ... then resuming the forked-mod1 project... >> > > [INFO] >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod1 >> > > >------------- >> > > [INFO] Building forked-mod1 0.0.1-SNAPSHOT >> > > [2/3] >> > > [INFO] --------------------------------[ jar >> > > ]--------------------------------- >> > > [INFO] >> > > [INFO] --- maven-resources-plugin:2.6:testResources >> > (default-testResources) >> > > @ forked-mod1 --- >> > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile >> (default-testCompile) >> > @ >> > > forked-mod1 --- >> > > [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) @ >> > > forked --- >> > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ >> forked-mod1 >> > > --- >> > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod1 --- >> > > ... and resuming the forked-mod2 project... >> > > [INFO] >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod2 >> > > >------------- >> > > [INFO] Building forked-mod2 0.0.1-SNAPSHOT >> > > [3/3] >> > > [INFO] --------------------------------[ jar >> > > ]--------------------------------- >> > > [INFO] >> > > [INFO] --- maven-resources-plugin:2.6:testResources >> > (default-testResources) >> > > @ forked-mod2 --- >> > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile >> (default-testCompile) >> > @ >> > > forked-mod2 --- >> > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ >> forked-mod2 >> > > --- >> > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod2 --- >> > > [INFO] >> > > >> ------------------------------------------------------------------------ >> > > [INFO] Reactor Summary for forked 0.0.1-SNAPSHOT: >> > > [INFO] >> > > [INFO] forked ............................................. SUCCESS [ >> > > 1.921 s] >> > > [INFO] forked-mod1 ........................................ SUCCESS [ >> > > 0.498 s] >> > > [INFO] forked-mod2 ........................................ SUCCESS [ >> > > 0.046 s] >> > > [INFO] >> > > >> ------------------------------------------------------------------------ >> > > [INFO] BUILD SUCCESS >> > > [INFO] >> > > >> ------------------------------------------------------------------------ >> > > [INFO] Total time: 2.535 s >> > > [INFO] Finished at: 2021-05-18T14:32:09+02:00 >> > > [INFO] >> > > >> ------------------------------------------------------------------------ >> > > >> > > This would also solve the parallel build problem because each project >> > would >> > > not be able to be both "cleaned" and "compiled" concurrently, which is >> > > currently possible in the current state. >> > > >> > > >> > > Le lun. 17 mai 2021 à 14:06, Guillaume Nodet <gno...@apache.org> a >> > écrit : >> > > >> > > > I've raised https://issues.apache.org/jira/browse/MNG-7156 >> > > > >> > > > Le mer. 12 mai 2021 à 17:57, Falko Modler <f.mod...@gmx.net> a >> écrit : >> > > > >> > > >> Hi Guillaume, >> > > >> >> > > >> aggregation goals and parallel builds in combination are a bit of a >> > > >> mess, e.g.: >> > > >> >> > > >> - https://issues.apache.org/jira/browse/MNG-6843 >> > > >> - https://github.com/apache/maven/pull/413 >> > > >> - https://www.mail-archive.com/dev@maven.apache.org/msg123439.html >> > > >> >> > > >> Cheers, >> > > >> >> > > >> Falko >> > > >> >> > > >> Am 12.05.2021 um 17:25 schrieb Guillaume Nodet: >> > > >> > Hi >> > > >> > >> > > >> > I've analyzed a bug reported on mvnd this afternoon ( >> > > >> > https://github.com/mvndaemon/mvnd/issues/408). It appears that >> the >> > > >> parent >> > > >> > pom executes the javadoc aggregate goal, which forks the >> lifecycle >> > of >> > > >> the >> > > >> > children modules in order to compile the sources. In a >> traditional >> > > >> build, >> > > >> > this does not cause any real problem, but in a parallel build, a >> > > clean >> > > >> > verify can definitely cause issues if the forked lifecycle and >> the >> > > >> normal >> > > >> > project build (and especially the clean) are run concurrently. >> > > >> > This definitely looks like an issue to me. Any idea where I >> should >> > > >> look at >> > > >> > how to solve the problem ? I wonder if the MojoExecutor should >> > > somehow >> > > >> > delegate to the Builder which is responsible for synchronizing >> the >> > > >> > executions in the case of a multithreaded build... >> > > >> > Thoughts ? >> > > >> > >> > > >> >> > > >> >> > > >> >> --------------------------------------------------------------------- >> > > >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> > > >> For additional commands, e-mail: dev-h...@maven.apache.org >> > > >> >> > > >> >> > > > >> > > > -- >> > > > ------------------------ >> > > > Guillaume Nodet >> > > > >> > > > >> > > >> > > -- >> > > ------------------------ >> > > Guillaume Nodet >> > > >> > >> >> >> -- >> ------------------------ >> Guillaume Nodet >> >
