Hi Guillaume, [3] is definitely different from your original problem, but it should explain (together with [2]) why [1] did not remove the second "compile" in your problem case. I wanted to understand why MNG-6566 didn't apply to it. I agree that if it would have applied (removing 2nd compile in "compile -> clean -> compile"), it would have failed the build even harder.
Anyhow, having said that, I believe that your idea of reordering parts of the build would definitely be an improvement. I just think it is ambitious and might be complex to build. I'm still not fully experienced in those parts of Maven core though, so maybe I am incorrectly estimating this complexity. Looking forward to your locking mechanism! Martin Op di 1 jun. 2021 om 18:20 schreef Guillaume Nodet <gno...@apache.org>: > > Le sam. 29 mai 2021 à 10:13, Martin Kanters <martinkant...@apache.org> a > écrit : > >> So we took a look at this. First we wanted to know why the forked >> goals were executed multiple times, while MNG-6566 [1] should have taken >> care of that. But it seems that that only optimizes duplicate goals in the >> context of one project. I attempted to explain that [2]. We created a >> ticket for optimizing this for multi module projects [3], but we think that >> it's not straightforward to fix this. >> > > I still think [3] is slightly different from my original problem. You > seem to indicate that the forked lifecycle is executed after, while in my > case, they were executed at the very beginning of the build, so that the > forked goals were executed before the usual execution from their > respective project. The effect is that for a given project, the following > goals were executed: > compile -> clean -> compile > > The second problem is that the concurrent execution of the clean and > compile goals lead to unpredictable results. > > It seems the issue only stresses a re-execution of the goal as a possible > optimization... That may be just me not reading correctly though... > > >> Back to the problem at hand. IIUC your suggestion is to pause >> project builds halfway, resume them after the required child projects goals >> have been executed (instead of forking), and finally finish the child >> project builds afterwards. I think it's elegant that in this solution, no >> duplicate goals will be executed, since no real forking has to be done. I >> do have the feeling that it would become quite a drastic change. I am also >> not sure whether we can assume that switching halfway through project >> builds works fine in every scenario. For the user it would also become hard >> to follow the build, I guess... >> >> Perhaps it would be easier to ensure projects can only be built once at >> a time (e.g. using locks or something). It wouldn't be the most >> efficient, but I guess it does not involve big changes to both the inner >> workings and user perspective. > > > I could try the second solution, though this could still leave an > execution order: > compile -> clean -> compile > which I think is really wrong. > > I think the first solution brings a few benefits : during parallel builds, > more things could be executed in parallel. For example the ordering could > be improved at the mojo execution level (or at least per group of mojos > execution in the same phase) instead of a global per-project ordering. In > theory, we could even make the project resolution happen in parallel with > some goals execution: compile time dependencies could be resolved during > the clean / resource generation phases, and test dependencies during > compilation, etc... Also, dependencies between projects only really happen > for the compilation (or test) phases, so the clean / resource generation > phases can happen concurrently. > For the output, I'm not so worried. In the case of parallel builds, the > output has to be reordered anyway to be understandable, as done in mvnd. > > I'll have a look at a simple per-project locking mechanism first though... > > >> Martin >> >> [1] https://issues.apache.org/jira/browse/MNG-6566 >> [2] >> >> https://issues.apache.org/jira/browse/MNG-6566?focusedCommentId=17353384&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17353384 >> [3] https://issues.apache.org/jira/browse/MNG-7163 >> >> Op di 18 mei 2021 om 21:45 schreef Martin Kanters < >> martinkant...@apache.org >> >: >> >> > Ah, too bad. I thought it was relevant. Perhaps I can take a look at >> this >> > with Maarten Mulders, but that will not be in the coming week at least. >> > >> > Op di 18 mei 2021 om 21:34 schreef Guillaume Nodet <gno...@apache.org>: >> > >> >> I just tried with master and the problem is the same. >> >> The reason is that in my use case, the forked goals are executed before >> >> the >> >> standard executions, and not even in the same project. >> >> >> >> Le mar. 18 mai 2021 à 20:41, Martin Kanters <martinkant...@apache.org> >> a >> >> écrit : >> >> >> >> > Not sure how I've missed this post. Have you tried this build with >> the >> >> > master build of Maven? >> >> > MNG-6566 [1] should prevent any unnecessary double executions, thus >> >> > optimizing the buildplan. >> >> > I'm interested if this will solve the problem at hand. >> >> > >> >> > Martin >> >> > >> >> > [1] https://issues.apache.org/jira/browse/MNG-6566 >> >> > >> >> > Op di 18 mei 2021 om 14:57 schreef Guillaume Nodet < >> gno...@apache.org>: >> >> > >> >> > > I'm looking a bit at aggregator goals. >> >> > > At first glance, it seems most of the problem comes from the fact >> that >> >> > the >> >> > > build ordering is done mostly per-project. This causes obvious >> >> problems >> >> > > for aggregators when a mojo needs a given phase to be executed for >> all >> >> > > children before it can run (for example the compilation phase >> before >> >> > > creating the javadoc). Another problem comes from the clean task >> >> which >> >> > has >> >> > > obvious side effects. Calling clean with an aggregator goal >> involved >> >> in >> >> > > the build seems like a recipe for problems... >> >> > > >> >> > > Would you see bad side effects to split the build in smaller chunks >> >> > > (basically down to a mojo execution) so that ordering can be >> changed >> >> in a >> >> > > more meaningful way ? For example when running an aggregate >> javadoc, >> >> the >> >> > > top level project would be built first until before the javadoc >> >> aggregate >> >> > > goal. The build would then go to all other projects (like the >> forked >> >> > > lifecycle) until all required goals have been run, then resume from >> >> the >> >> > > javadoc goal. >> >> > > Note that with the current behavior, when you run "clean verify", >> the >> >> > > execution is the following >> >> > > >> >> > > [INFO] Scanning for projects... >> >> > > [INFO] >> >> > > >> >> >> ------------------------------------------------------------------------ >> >> > > [INFO] Reactor Build Order: >> >> > > [INFO] >> >> > > [INFO] forked >> >> > > [pom] >> >> > > [INFO] forked-mod1 >> >> > > [jar] >> >> > > [INFO] forked-mod2 >> >> > > [jar] >> >> > > [INFO] >> >> > > [INFO] ---------------< org.mvndaemon.mvnd.test.forked:forked >> >> > > >---------------- >> >> > > [INFO] Building forked 0.0.1-SNAPSHOT >> >> > > [1/3] >> >> > > [INFO] --------------------------------[ pom >> >> > > ]--------------------------------- >> >> > > [INFO] >> >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked >> --- >> >> > > [INFO] >>> maven-javadoc-plugin:3.2.0:aggregate-jar >> (aggregate-jar) > >> >> > > compile @ forked >>> >> >> > > [INFO] >> >> > > >> >> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >> > > [INFO] Forking forked-mod1 0.0.1-SNAPSHOT >> >> > > [INFO] >> >> > > >> >> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >> > > [INFO] --- maven-resources-plugin:2.6:resources >> (default-resources) @ >> >> > > forked-mod1 --- >> >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> >> > > forked-mod1 --- >> >> > > [INFO] >> >> > > >> >> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >> > > [INFO] Forking forked-mod2 0.0.1-SNAPSHOT >> >> > > [INFO] >> >> > > >> >> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >> > > [INFO] --- maven-resources-plugin:2.6:resources >> (default-resources) @ >> >> > > forked-mod2 --- >> >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> >> > > forked-mod2 --- >> >> > > [INFO] <<< maven-javadoc-plugin:3.2.0:aggregate-jar >> (aggregate-jar) < >> >> > > compile @ forked <<< >> >> > > [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar >> (aggregate-jar) @ >> >> > > forked --- >> >> > > [INFO] >> >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod1 >> >> > > >------------- >> >> > > [INFO] Building forked-mod1 0.0.1-SNAPSHOT >> >> > > [2/3] >> >> > > [INFO] --------------------------------[ jar >> >> > > ]--------------------------------- >> >> > > [INFO] >> >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ >> forked-mod1 >> >> --- >> >> > > [INFO] --- maven-resources-plugin:2.6:resources >> (default-resources) @ >> >> > > forked-mod1 --- >> >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> >> > > forked-mod1 --- >> >> > > [INFO] --- maven-resources-plugin:2.6:testResources >> >> > (default-testResources) >> >> > > @ forked-mod1 --- >> >> > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile >> >> (default-testCompile) >> >> > @ >> >> > > forked-mod1 --- >> >> > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ >> >> forked-mod1 >> >> > > --- >> >> > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod1 --- >> >> > > [INFO] >> >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod2 >> >> > > >------------- >> >> > > [INFO] Building forked-mod2 0.0.1-SNAPSHOT >> >> > > [3/3] >> >> > > [INFO] --------------------------------[ jar >> >> > > ]--------------------------------- >> >> > > [INFO] >> >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ >> forked-mod2 >> >> --- >> >> > > [INFO] --- maven-resources-plugin:2.6:resources >> (default-resources) @ >> >> > > forked-mod2 --- >> >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> >> > > forked-mod2 --- >> >> > > [INFO] --- maven-resources-plugin:2.6:testResources >> >> > (default-testResources) >> >> > > @ forked-mod2 --- >> >> > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile >> >> (default-testCompile) >> >> > @ >> >> > > forked-mod2 --- >> >> > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ >> >> forked-mod2 >> >> > > --- >> >> > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod2 --- >> >> > > [INFO] >> >> > > >> >> >> ------------------------------------------------------------------------ >> >> > > [INFO] Reactor Summary for forked 0.0.1-SNAPSHOT: >> >> > > [INFO] >> >> > > [INFO] forked ............................................. >> SUCCESS [ >> >> > > 1.921 s] >> >> > > [INFO] forked-mod1 ........................................ >> SUCCESS [ >> >> > > 0.498 s] >> >> > > [INFO] forked-mod2 ........................................ >> SUCCESS [ >> >> > > 0.046 s] >> >> > > [INFO] >> >> > > >> >> >> ------------------------------------------------------------------------ >> >> > > [INFO] BUILD SUCCESS >> >> > > [INFO] >> >> > > >> >> >> ------------------------------------------------------------------------ >> >> > > [INFO] Total time: 2.535 s >> >> > > [INFO] Finished at: 2021-05-18T14:32:09+02:00 >> >> > > [INFO] >> >> > > >> >> >> ------------------------------------------------------------------------ >> >> > > >> >> > > As you can see, the resources and compile goals are run twice for >> >> each >> >> > > project. Furthermore, given the clean goal is in the loop, the >> >> compiler >> >> > > actually compiles things twice. >> >> > > >> >> > > The build plan could be rewritten as >> >> > > >> >> > > [INFO] Scanning for projects... >> >> > > [INFO] >> >> > > >> >> >> ------------------------------------------------------------------------ >> >> > > [INFO] Reactor Build Order: >> >> > > [INFO] >> >> > > [INFO] forked >> >> > > [pom] >> >> > > [INFO] forked-mod1 >> >> > > [jar] >> >> > > [INFO] forked-mod2 >> >> > > [jar] >> >> > > [INFO] >> >> > > [INFO] ---------------< org.mvndaemon.mvnd.test.forked:forked >> >> > > >---------------- >> >> > > [INFO] Building forked 0.0.1-SNAPSHOT >> >> > > [1/3] >> >> > > [INFO] --------------------------------[ pom >> >> > > ]--------------------------------- >> >> > > [INFO] >> >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked >> --- >> >> > > ... normal lifecycle interrupted because the next goal is the >> >> aggregator >> >> > > goal ... >> >> > > [INFO] >> >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod1 >> >> > > >------------- >> >> > > [INFO] Building forked-mod1 0.0.1-SNAPSHOT >> >> > > [2/3] >> >> > > [INFO] --------------------------------[ jar >> >> > > ]--------------------------------- >> >> > > [INFO] >> >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ >> forked-mod1 >> >> --- >> >> > > [INFO] --- maven-resources-plugin:2.6:resources >> (default-resources) @ >> >> > > forked-mod1 --- >> >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> >> > > forked-mod1 --- >> >> > > ... interrupting forked-mod1 build as it reached the required point >> >> > > execution ... >> >> > > [INFO] >> >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod2 >> >> > > >------------- >> >> > > [INFO] Building forked-mod2 0.0.1-SNAPSHOT >> >> > > [3/3] >> >> > > [INFO] --------------------------------[ jar >> >> > > ]--------------------------------- >> >> > > [INFO] >> >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ >> forked-mod2 >> >> --- >> >> > > [INFO] --- maven-resources-plugin:2.6:resources >> (default-resources) @ >> >> > > forked-mod2 --- >> >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> >> > > forked-mod2 --- >> >> > > ... interrupting forked-mod2 build as it reached the required point >> >> > > execution ... >> >> > > ... resuming the top level project... >> >> > > [INFO] >> >> > > [INFO] ---------------< org.mvndaemon.mvnd.test.forked:forked >> >> > > >---------------- >> >> > > [INFO] Building forked 0.0.1-SNAPSHOT >> >> > > [1/3] >> >> > > [INFO] --------------------------------[ pom >> >> > > ]--------------------------------- >> >> > > [INFO] >> >> > > [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar >> (aggregate-jar) @ >> >> > > forked --- >> >> > > ... then resuming the forked-mod1 project... >> >> > > [INFO] >> >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod1 >> >> > > >------------- >> >> > > [INFO] Building forked-mod1 0.0.1-SNAPSHOT >> >> > > [2/3] >> >> > > [INFO] --------------------------------[ jar >> >> > > ]--------------------------------- >> >> > > [INFO] >> >> > > [INFO] --- maven-resources-plugin:2.6:testResources >> >> > (default-testResources) >> >> > > @ forked-mod1 --- >> >> > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile >> >> (default-testCompile) >> >> > @ >> >> > > forked-mod1 --- >> >> > > [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar >> (aggregate-jar) @ >> >> > > forked --- >> >> > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ >> >> forked-mod1 >> >> > > --- >> >> > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod1 --- >> >> > > ... and resuming the forked-mod2 project... >> >> > > [INFO] >> >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod2 >> >> > > >------------- >> >> > > [INFO] Building forked-mod2 0.0.1-SNAPSHOT >> >> > > [3/3] >> >> > > [INFO] --------------------------------[ jar >> >> > > ]--------------------------------- >> >> > > [INFO] >> >> > > [INFO] --- maven-resources-plugin:2.6:testResources >> >> > (default-testResources) >> >> > > @ forked-mod2 --- >> >> > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile >> >> (default-testCompile) >> >> > @ >> >> > > forked-mod2 --- >> >> > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ >> >> forked-mod2 >> >> > > --- >> >> > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod2 --- >> >> > > [INFO] >> >> > > >> >> >> ------------------------------------------------------------------------ >> >> > > [INFO] Reactor Summary for forked 0.0.1-SNAPSHOT: >> >> > > [INFO] >> >> > > [INFO] forked ............................................. >> SUCCESS [ >> >> > > 1.921 s] >> >> > > [INFO] forked-mod1 ........................................ >> SUCCESS [ >> >> > > 0.498 s] >> >> > > [INFO] forked-mod2 ........................................ >> SUCCESS [ >> >> > > 0.046 s] >> >> > > [INFO] >> >> > > >> >> >> ------------------------------------------------------------------------ >> >> > > [INFO] BUILD SUCCESS >> >> > > [INFO] >> >> > > >> >> >> ------------------------------------------------------------------------ >> >> > > [INFO] Total time: 2.535 s >> >> > > [INFO] Finished at: 2021-05-18T14:32:09+02:00 >> >> > > [INFO] >> >> > > >> >> >> ------------------------------------------------------------------------ >> >> > > >> >> > > This would also solve the parallel build problem because each >> project >> >> > would >> >> > > not be able to be both "cleaned" and "compiled" concurrently, >> which is >> >> > > currently possible in the current state. >> >> > > >> >> > > >> >> > > Le lun. 17 mai 2021 à 14:06, Guillaume Nodet <gno...@apache.org> a >> >> > écrit : >> >> > > >> >> > > > I've raised https://issues.apache.org/jira/browse/MNG-7156 >> >> > > > >> >> > > > Le mer. 12 mai 2021 à 17:57, Falko Modler <f.mod...@gmx.net> a >> >> écrit : >> >> > > > >> >> > > >> Hi Guillaume, >> >> > > >> >> >> > > >> aggregation goals and parallel builds in combination are a bit >> of a >> >> > > >> mess, e.g.: >> >> > > >> >> >> > > >> - https://issues.apache.org/jira/browse/MNG-6843 >> >> > > >> - https://github.com/apache/maven/pull/413 >> >> > > >> - >> https://www.mail-archive.com/dev@maven.apache.org/msg123439.html >> >> > > >> >> >> > > >> Cheers, >> >> > > >> >> >> > > >> Falko >> >> > > >> >> >> > > >> Am 12.05.2021 um 17:25 schrieb Guillaume Nodet: >> >> > > >> > Hi >> >> > > >> > >> >> > > >> > I've analyzed a bug reported on mvnd this afternoon ( >> >> > > >> > https://github.com/mvndaemon/mvnd/issues/408). It appears >> that >> >> the >> >> > > >> parent >> >> > > >> > pom executes the javadoc aggregate goal, which forks the >> >> lifecycle >> >> > of >> >> > > >> the >> >> > > >> > children modules in order to compile the sources. In a >> >> traditional >> >> > > >> build, >> >> > > >> > this does not cause any real problem, but in a parallel >> build, a >> >> > > clean >> >> > > >> > verify can definitely cause issues if the forked lifecycle and >> >> the >> >> > > >> normal >> >> > > >> > project build (and especially the clean) are run concurrently. >> >> > > >> > This definitely looks like an issue to me. Any idea where I >> >> should >> >> > > >> look at >> >> > > >> > how to solve the problem ? I wonder if the MojoExecutor >> should >> >> > > somehow >> >> > > >> > delegate to the Builder which is responsible for synchronizing >> >> the >> >> > > >> > executions in the case of a multithreaded build... >> >> > > >> > Thoughts ? >> >> > > >> > >> >> > > >> >> >> > > >> >> >> > > >> >> >> --------------------------------------------------------------------- >> >> > > >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> >> > > >> For additional commands, e-mail: dev-h...@maven.apache.org >> >> > > >> >> >> > > >> >> >> > > > >> >> > > > -- >> >> > > > ------------------------ >> >> > > > Guillaume Nodet >> >> > > > >> >> > > > >> >> > > >> >> > > -- >> >> > > ------------------------ >> >> > > Guillaume Nodet >> >> > > >> >> > >> >> >> >> >> >> -- >> >> ------------------------ >> >> Guillaume Nodet >> >> >> > >> > > > -- > ------------------------ > Guillaume Nodet > >
