Hey, We don't need the module-b jar built in the reactor to be included in the war if the changes made in module-b won't affect the war. Instead, in this scenario, we're fine with using a jar resolved from the local repository. The requisites for this scenario are:
* Any change made to module-b is only in its test tree * The only changes made to the direct and transitive dependencies of module-b are to its test scoped dependencies So basically module-b is only in the reactor for verification purposes rather than ‘src packaging’ purposes. Thanks, Joe On 2024/02/08 12:12:03 Tamás Cservenák wrote: > Hej, > > yes, basically the "hack" is well described by you: > - it retains "full' reactor project > - introduces "needs build" on project > - when it comes to building, it skips the project if flag set > > This results that module-b -> app link is not lost. > And as for your parallelization request: the truth is that there IS and > edge between app and module-b (via module-a), so despite module-a is > "done", its transitive module-b is not done, hence you cannot build these > two in parallel, as you do have an edge between app and module-b. > > Or to invert the question: what module-b JAR you want to have included in > app WAR, when building? The one built, or the one from the local > repository? If the latter, why are you building it in the first place? > > On Thu, Feb 8, 2024 at 12:06 PM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > > > Hi Tamás, > > Thank you for hacking around this so rapidly. I am not familiar with the > > maven code base, so let me know if I have misinterpreted the change: Is > > your hack to: > > > > 1. Retain the full multi-module build dependency graph such that the > > build graph always remains consistent regardless of whether a “-pl” arg has > > been provided > > 2. Introduce a skip build option to satisfy the functionality of the > > “-pl” arg: The reactor will report the module has been built but the build > > is basically a no-op > > If so, this does resolve the issue but at the cost of losing the > > opportunity to gain extra parallelism in the build. > > What I was wondering may be possible (without any idea of how feasible it > > would be to implement) would be the opportunity to recognise in the > > multi-build distinct build graphs and then keep isolation in the reactor > > between these graphs. To elaborate, in the reproducer scenario there would > > be two distinct graphs: > > > > * "testsupport-module-1" followed by "app" > > * "testsupport-module-2" followed by "module-b" > > Ideally these two graphs build in parallel to each other and even if > > ultimately there is a dependency between them they NEVER resolve that > > dependency from within reactor they instead always resolve from the > > external Maven cache. The key statement here is that if a user presents a > > list of modules to be built then they really need to be sure that the > > DIRECT dependencies between these modules truly represent the build > > graph(s) they want. For tools like gitflow-incremental-builder which are > > designed to carefully work out change sets and what needs to build this > > offers a really powerful opportunity to optimise builds. I appreciate this > > functionality could also be seen as a gotcha (although tbf this already > > exists). > > Thanks, > > Joe > > > > > > On 2024/02/08 10:09:31 Tamás Cservenák wrote: > > > Seems we are on track with this. To prove my last-night theory I created > > a > > > "hack" (is really just that) and guess what? > > > It makes reproducer behave "as expected": > > > https://github.com/apache/maven/pull/1406 > > > > > > T > > > > > > On Wed, Feb 7, 2024 at 10:05 PM Tamás Cservenák <ta...@cservenak.net> > > wrote: > > > > > > > Howdy, > > > > > > > > Thank you very much, the reproducer works. Did not dig thru it fully, > > but > > > > here are some related issues: > > > > > > > > https://issues.apache.org/jira/browse/MNG-8028 (funny thing, I created > > > > this few weeks ago) > > > > https://issues.apache.org/jira/browse/MNG-6300 > > > > > > > > Will report back tomorrow (EU TZ) > > > > T > > > > > > > > > > > > On Wed, Feb 7, 2024 at 7:48 PM Joseph Leonard < > > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > >> Hi Tamás, > > > >> I have created a simple example here: > > > >> https://github.com/josple/mvn-multibuild-issue > > > >> Hopefully the README is clear enough – let me know if I can clarify > > > >> anything. > > > >> Thanks, > > > >> Joe > > > >> > > > >> On 2024/02/07 17:33:08 Tamás Cservenák wrote: > > > >> > Howdy, > > > >> > > > > >> > In that case, there is something fishy with the project, my blind > > guess > > > >> > would be some "hidden" inter-module dependency maybe? > > > >> > > > > >> > Can you provide access to source, or, if not feasible, could you > > provide > > > >> > some reproducer and publish it on Github/Gitlab/whatever (maybe even > > > >> just > > > >> > send it privately as ML strips off attachments and images) for us > > to see > > > >> > this in action? > > > >> > > > > >> > Thanks > > > >> > T > > > >> > > > > >> > On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < > > > >> > joseph.leon...@alfasystems.com> wrote: > > > >> > > > > >> > > Hi Tamás, > > > >> > > We have previously played around a bit with mvnd but not takari > > > >> directly – > > > >> > > I will have a play with it. With regards to this issue, using the > > > >> takari > > > >> > > smart builder unfortunately doesn’t resolve the issue. > > > >> > > Joe > > > >> > > > > > >> > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: > > > >> > > > Can you please try smart builder instead? > > > >> > > > https://github.com/takari/takari-smart-builder > > > >> > > > > > > >> > > > (note: smart builder is used by mvnd as well) > > > >> > > > > > > >> > > > The difference between the two can be seen here: > > > >> > > > http://takari.io/book/30-team-maven.html#takari-smart-builder > > > >> > > > > > > >> > > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < > > > >> > > > joseph.leon...@alfasystems.com> wrote: > > > >> > > > > > > >> > > > > Hi Tamás, > > > >> > > > > Yeah, this was unexpected to me initially as well. From what I > > > >> can tell > > > >> > > > > the Maven reactor only considers direct dependencies (i.e. not > > > >> > > transitive > > > >> > > > > dependencies) between the modules in the reactor when working > > out > > > >> the > > > >> > > build > > > >> > > > > graph. For example if you have a simple linear dependency > > chain > > > >> of: > > > >> > > > > One --> Two --> Three --> Four --> Five > > > >> > > > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 > > will > > > >> result > > > >> > > in > > > >> > > > > two ‘graphs’ being built in parallel ([One,Two] and > > [Four,Five]). > > > >> I > > > >> > > assume > > > >> > > > > this is as designed because it actually offers quite powerful > > > >> > > functionality > > > >> > > > > to improve the parallelism in your build. An example of where > > > >> this is > > > >> > > legit > > > >> > > > > is when: > > > >> > > > > > > > >> > > > > * “Four” has a test scope dependency on “Five” > > > >> > > > > * “One” has a test scoped dependency on “Two” > > > >> > > > > If you made a src code change to “Five” and “Two” then it > > would be > > > >> > > safe to > > > >> > > > > build [One,Two] and [Four,Five] in parallel because you know > > the > > > >> > > changes > > > >> > > > > within these graphs cannot impact each other. > > > >> > > > > Joe > > > >> > > > > > > > >> > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > > >> > > > > > Howdy, > > > >> > > > > > > > > >> > > > > > To me this looks like Maven is not aware that the App > > depends on > > > >> > > > > ModuleB... > > > >> > > > > > Are they "plain dependency" linked? Or what kind of > > dependency > > > >> we > > > >> > > talk > > > >> > > > > > about here? > > > >> > > > > > In short: why would App start while ModuleB (upstream dep) > > is > > > >> not > > > >> > > done? > > > >> > > > > > Something is fishy here. > > > >> > > > > > > > > >> > > > > > T > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > > >> > > > > > joseph.leon...@alfasystems.com> wrote: > > > >> > > > > > > > > >> > > > > > > Hi all, > > > >> > > > > > > > > > >> > > > > > > It would be great to get any thoughts on whether the > > > >> following is a > > > >> > > > > defect: > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > Issue details: > > > >> > > > > > > tl;dr > > > >> > > > > > > > > > >> > > > > > > Maven can resolve dependencies either from: > > > >> > > > > > > > > > >> > > > > > > * an external repo > > > >> > > > > > > * a class directory of a module being built within the > > > >> reactor > > > >> > > > > > > * a packaged jar of a module being built within the > > > >> reactor > > > >> > > > > > > > > > >> > > > > > > If you run a concurrent multi-module build it is possible > > to > > > >> get a > > > >> > > race > > > >> > > > > > > condition whereby the build of module Foo may resolve > > module > > > >> Bar > > > >> > > from > > > >> > > > > > > either of the three resolution channels. This > > inconsistency > > > >> can > > > >> > > result > > > >> > > > > in > > > >> > > > > > > the Maven war plugin sometimes failing to build a > > functional > > > >> war > > > >> > > file. > > > >> > > > > I > > > >> > > > > > > would expect a consistent resolution would always take > > place. > > > >> > > > > > > > > > >> > > > > > > Full details > > > >> > > > > > > Scenario > > > >> > > > > > > > > > >> > > > > > > Consider you have a repo with the following structure: > > > >> > > > > > > > > > >> > > > > > > App > > > >> > > > > > > > > > >> > > > > > > / \ > > > >> > > > > > > > > > >> > > > > > > / \ > > > >> > > > > > > > > > >> > > > > > > (compile scope) (test scope) > > > >> > > > > > > > > > >> > > > > > > / \ > > > >> > > > > > > > > > >> > > > > > > \/_ _\/ > > > >> > > > > > > > > > >> > > > > > > ModuleA TestSupportModule1 > > > >> > > > > > > > > > >> > > > > > > / > > > >> > > > > > > > > > >> > > > > > > / > > > >> > > > > > > > > > >> > > > > > > (compile scope) > > > >> > > > > > > > > > >> > > > > > > / > > > >> > > > > > > > > > >> > > > > > > \/_ > > > >> > > > > > > > > > >> > > > > > > ModuleB > > > >> > > > > > > > > > >> > > > > > > / > > > >> > > > > > > > > > >> > > > > > > / > > > >> > > > > > > > > > >> > > > > > > (test scope) > > > >> > > > > > > > > > >> > > > > > > / > > > >> > > > > > > > > > >> > > > > > > \/_ > > > >> > > > > > > > > > >> > > > > > > TestSupportModule2 > > > >> > > > > > > > > > >> > > > > > > If you were to make a src code change to the following > > test > > > >> support > > > >> > > > > > > modules: > > > >> > > > > > > > > > >> > > > > > > * TestSupportModule1 > > > >> > > > > > > * TestSupportModule2 > > > >> > > > > > > > > > >> > > > > > > Then the minimum number of modules we need to build to > > verify > > > >> the > > > >> > > > > change > > > >> > > > > > > set is OK is: > > > >> > > > > > > > > > >> > > > > > > * TestSupportModule1 > > > >> > > > > > > * TestSupportModule2 > > > >> > > > > > > * ModuleB > > > >> > > > > > > * App > > > >> > > > > > > > > > >> > > > > > > i.e. there is no requirement to build ModuleA because we > > know > > > >> that > > > >> > > > > none of > > > >> > > > > > > the src code changes could impact the classpaths used in > > its > > > >> maven > > > >> > > > > build. > > > >> > > > > > > > > > >> > > > > > > We know that despite 'App' depending (transitively) on > > ModuleB > > > >> > > there > > > >> > > > > is no > > > >> > > > > > > need for the 'App' build to wait for ModuleB to complete > > its > > > >> build > > > >> > > > > because > > > >> > > > > > > the src code change to TestSupportModule2 will not impact > > any > > > >> of > > > >> > > the > > > >> > > > > > > classpaths used in the App maven build. Therefore to get > > the > > > >> most > > > >> > > > > efficient > > > >> > > > > > > build possible we ideally would invoke Maven to run with 2 > > > >> threads > > > >> > > and > > > >> > > > > with > > > >> > > > > > > instruction to build two distinct 'dependency graphs': > > > >> > > > > > > > > > >> > > > > > > * TestSupportModule1 followed by ModuleB > > > >> > > > > > > * TestSupportModule1 followed by App > > > >> > > > > > > > > > >> > > > > > > The following Maven command achieves exactly what we want > > > >> because > > > >> > > the > > > >> > > > > > > reactor build order is based only on the direct (i.e. > > > >> > > non-transitive) > > > >> > > > > > > dependencies of the modules provided to the reactor in the > > > >> build > > > >> > > > > command. > > > >> > > > > > > Therefore the absence of ModuleA results in two distinct > > > >> > > 'dependency > > > >> > > > > > > graphs': > > > >> > > > > > > > > > >> > > > > > > mvn clean verify -pl > > > >> > > TestSupportModule1,TestSupportModule2,ModuleB,App > > > >> > > > > -T 2 > > > >> > > > > > > > > > >> > > > > > > Note: In reality the code base I maintain has a very large > > > >> > > monobuild > > > >> > > > > with > > > >> > > > > > > 100s of modules and this type of build optimisation makes > > a > > > >> > > significant > > > >> > > > > > > difference to the speed of our monobuild (we use > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > >> > > https://github.com/gitflow-incremental-builder/gitflow-incremental-builder > > > >> > > > > > > to automate the logic of determining which modules to > > include > > > >> in > > > >> > > the > > > >> > > > > > > reactor based on our change set). > > > >> > > > > > > > > > >> > > > > > > Issue > > > >> > > > > > > > > > >> > > > > > > We have encountered an issue in the above scenario > > because the > > > >> > > 'App' > > > >> > > > > build > > > >> > > > > > > has a race condition with the ModuleB build which will > > result > > > >> in > > > >> > > one > > > >> > > > > of the > > > >> > > > > > > following three outcomes: > > > >> > > > > > > > > > >> > > > > > > * If the 'App' build starts before the ModuleB build > > has > > > >> > > compiled > > > >> > > > > its > > > >> > > > > > > src classes then the 'App' build will resolve ModuleB > > from the > > > >> > > external > > > >> > > > > > > repo (i.e. equivalent to ModuleB not being in the reactor > > at > > > >> all) > > > >> > > > > > > * If the 'App' build starts after ModuleB has compiled > > > >> its src > > > >> > > > > classes > > > >> > > > > > > but before it has packaged these classes into a jar then > > the > > > >> 'App' > > > >> > > > > build > > > >> > > > > > > will resolve ModuleB's target/classes directory > > > >> > > > > > > * If the 'App' build starts after ModuleB has packaged > > > >> its jar > > > >> > > file > > > >> > > > > > > then the 'App' build will resolve ModuleB's > > target/ModuleB.jar > > > >> > > file. > > > >> > > > > > > > > > >> > > > > > > In many scenarios this dependency resolution inconsistency > > > >> doesn't > > > >> > > > > > > represent a challenge. However, it does cause an issue in > > our > > > >> case > > > >> > > > > because > > > >> > > > > > > the 'App' POM has its Maven packaging stanza configured to > > > >> war and > > > >> > > in > > > >> > > > > the > > > >> > > > > > > scenario where ModuleB's target/classes directory is > > resolved > > > >> by > > > >> > > the > > > >> > > > > 'App' > > > >> > > > > > > then this results in the resultant 'App' war file being > > > >> packaged > > > >> > > with a > > > >> > > > > > > completely empty ModuleB.jar file. > > > >> > > > > > > > > > >> > > > > > > Proposed solution > > > >> > > > > > > > > > >> > > > > > > Ideally we would like the Maven reactor to retain > > isolation > > > >> > > between the > > > >> > > > > > > two distinct 'dependency graphs' it constructs at > > > >> instantiation > > > >> > > > > throughout > > > >> > > > > > > the entire Maven build. This would mean, in the simple > > example > > > >> > > above, > > > >> > > > > that > > > >> > > > > > > the 'App' would always resolves ModuleB from the external > > repo > > > >> > > > > (regardless > [message truncated...]
