Seems we are on track with this. To prove my last-night theory I created a "hack" (is really just that) and guess what? It makes reproducer behave "as expected": https://github.com/apache/maven/pull/1406
T On Wed, Feb 7, 2024 at 10:05 PM Tamás Cservenák <ta...@cservenak.net> wrote: > Howdy, > > Thank you very much, the reproducer works. Did not dig thru it fully, but > here are some related issues: > > https://issues.apache.org/jira/browse/MNG-8028 (funny thing, I created > this few weeks ago) > https://issues.apache.org/jira/browse/MNG-6300 > > Will report back tomorrow (EU TZ) > T > > > On Wed, Feb 7, 2024 at 7:48 PM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > >> Hi Tamás, >> I have created a simple example here: >> https://github.com/josple/mvn-multibuild-issue >> Hopefully the README is clear enough – let me know if I can clarify >> anything. >> Thanks, >> Joe >> >> On 2024/02/07 17:33:08 Tamás Cservenák wrote: >> > Howdy, >> > >> > In that case, there is something fishy with the project, my blind guess >> > would be some "hidden" inter-module dependency maybe? >> > >> > Can you provide access to source, or, if not feasible, could you provide >> > some reproducer and publish it on Github/Gitlab/whatever (maybe even >> just >> > send it privately as ML strips off attachments and images) for us to see >> > this in action? >> > >> > Thanks >> > T >> > >> > On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < >> > joseph.leon...@alfasystems.com> wrote: >> > >> > > Hi Tamás, >> > > We have previously played around a bit with mvnd but not takari >> directly – >> > > I will have a play with it. With regards to this issue, using the >> takari >> > > smart builder unfortunately doesn’t resolve the issue. >> > > Joe >> > > >> > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: >> > > > Can you please try smart builder instead? >> > > > https://github.com/takari/takari-smart-builder >> > > > >> > > > (note: smart builder is used by mvnd as well) >> > > > >> > > > The difference between the two can be seen here: >> > > > http://takari.io/book/30-team-maven.html#takari-smart-builder >> > > > >> > > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < >> > > > joseph.leon...@alfasystems.com> wrote: >> > > > >> > > > > Hi Tamás, >> > > > > Yeah, this was unexpected to me initially as well. From what I >> can tell >> > > > > the Maven reactor only considers direct dependencies (i.e. not >> > > transitive >> > > > > dependencies) between the modules in the reactor when working out >> the >> > > build >> > > > > graph. For example if you have a simple linear dependency chain >> of: >> > > > > One --> Two --> Three --> Four --> Five >> > > > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will >> result >> > > in >> > > > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). >> I >> > > assume >> > > > > this is as designed because it actually offers quite powerful >> > > functionality >> > > > > to improve the parallelism in your build. An example of where >> this is >> > > legit >> > > > > is when: >> > > > > >> > > > > * “Four” has a test scope dependency on “Five” >> > > > > * “One” has a test scoped dependency on “Two” >> > > > > If you made a src code change to “Five” and “Two” then it would be >> > > safe to >> > > > > build [One,Two] and [Four,Five] in parallel because you know the >> > > changes >> > > > > within these graphs cannot impact each other. >> > > > > Joe >> > > > > >> > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: >> > > > > > Howdy, >> > > > > > >> > > > > > To me this looks like Maven is not aware that the App depends on >> > > > > ModuleB... >> > > > > > Are they "plain dependency" linked? Or what kind of dependency >> we >> > > talk >> > > > > > about here? >> > > > > > In short: why would App start while ModuleB (upstream dep) is >> not >> > > done? >> > > > > > Something is fishy here. >> > > > > > >> > > > > > T >> > > > > > >> > > > > > >> > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < >> > > > > > joseph.leon...@alfasystems.com> wrote: >> > > > > > >> > > > > > > Hi all, >> > > > > > > >> > > > > > > It would be great to get any thoughts on whether the >> following is a >> > > > > defect: >> > > > > > > >> > > > > > > >> > > > > > > Issue details: >> > > > > > > tl;dr >> > > > > > > >> > > > > > > Maven can resolve dependencies either from: >> > > > > > > >> > > > > > > * an external repo >> > > > > > > * a class directory of a module being built within the >> reactor >> > > > > > > * a packaged jar of a module being built within the >> reactor >> > > > > > > >> > > > > > > If you run a concurrent multi-module build it is possible to >> get a >> > > race >> > > > > > > condition whereby the build of module Foo may resolve module >> Bar >> > > from >> > > > > > > either of the three resolution channels. This inconsistency >> can >> > > result >> > > > > in >> > > > > > > the Maven war plugin sometimes failing to build a functional >> war >> > > file. >> > > > > I >> > > > > > > would expect a consistent resolution would always take place. >> > > > > > > >> > > > > > > Full details >> > > > > > > Scenario >> > > > > > > >> > > > > > > Consider you have a repo with the following structure: >> > > > > > > >> > > > > > > App >> > > > > > > >> > > > > > > / \ >> > > > > > > >> > > > > > > / \ >> > > > > > > >> > > > > > > (compile scope) (test scope) >> > > > > > > >> > > > > > > / \ >> > > > > > > >> > > > > > > \/_ _\/ >> > > > > > > >> > > > > > > ModuleA TestSupportModule1 >> > > > > > > >> > > > > > > / >> > > > > > > >> > > > > > > / >> > > > > > > >> > > > > > > (compile scope) >> > > > > > > >> > > > > > > / >> > > > > > > >> > > > > > > \/_ >> > > > > > > >> > > > > > > ModuleB >> > > > > > > >> > > > > > > / >> > > > > > > >> > > > > > > / >> > > > > > > >> > > > > > > (test scope) >> > > > > > > >> > > > > > > / >> > > > > > > >> > > > > > > \/_ >> > > > > > > >> > > > > > > TestSupportModule2 >> > > > > > > >> > > > > > > If you were to make a src code change to the following test >> support >> > > > > > > modules: >> > > > > > > >> > > > > > > * TestSupportModule1 >> > > > > > > * TestSupportModule2 >> > > > > > > >> > > > > > > Then the minimum number of modules we need to build to verify >> the >> > > > > change >> > > > > > > set is OK is: >> > > > > > > >> > > > > > > * TestSupportModule1 >> > > > > > > * TestSupportModule2 >> > > > > > > * ModuleB >> > > > > > > * App >> > > > > > > >> > > > > > > i.e. there is no requirement to build ModuleA because we know >> that >> > > > > none of >> > > > > > > the src code changes could impact the classpaths used in its >> maven >> > > > > build. >> > > > > > > >> > > > > > > We know that despite 'App' depending (transitively) on ModuleB >> > > there >> > > > > is no >> > > > > > > need for the 'App' build to wait for ModuleB to complete its >> build >> > > > > because >> > > > > > > the src code change to TestSupportModule2 will not impact any >> of >> > > the >> > > > > > > classpaths used in the App maven build. Therefore to get the >> most >> > > > > efficient >> > > > > > > build possible we ideally would invoke Maven to run with 2 >> threads >> > > and >> > > > > with >> > > > > > > instruction to build two distinct 'dependency graphs': >> > > > > > > >> > > > > > > * TestSupportModule1 followed by ModuleB >> > > > > > > * TestSupportModule1 followed by App >> > > > > > > >> > > > > > > The following Maven command achieves exactly what we want >> because >> > > the >> > > > > > > reactor build order is based only on the direct (i.e. >> > > non-transitive) >> > > > > > > dependencies of the modules provided to the reactor in the >> build >> > > > > command. >> > > > > > > Therefore the absence of ModuleA results in two distinct >> > > 'dependency >> > > > > > > graphs': >> > > > > > > >> > > > > > > mvn clean verify -pl >> > > TestSupportModule1,TestSupportModule2,ModuleB,App >> > > > > -T 2 >> > > > > > > >> > > > > > > Note: In reality the code base I maintain has a very large >> > > monobuild >> > > > > with >> > > > > > > 100s of modules and this type of build optimisation makes a >> > > significant >> > > > > > > difference to the speed of our monobuild (we use >> > > > > > > >> > > > > >> > > >> https://github.com/gitflow-incremental-builder/gitflow-incremental-builder >> > > > > > > to automate the logic of determining which modules to include >> in >> > > the >> > > > > > > reactor based on our change set). >> > > > > > > >> > > > > > > Issue >> > > > > > > >> > > > > > > We have encountered an issue in the above scenario because the >> > > 'App' >> > > > > build >> > > > > > > has a race condition with the ModuleB build which will result >> in >> > > one >> > > > > of the >> > > > > > > following three outcomes: >> > > > > > > >> > > > > > > * If the 'App' build starts before the ModuleB build has >> > > compiled >> > > > > its >> > > > > > > src classes then the 'App' build will resolve ModuleB from the >> > > external >> > > > > > > repo (i.e. equivalent to ModuleB not being in the reactor at >> all) >> > > > > > > * If the 'App' build starts after ModuleB has compiled >> its src >> > > > > classes >> > > > > > > but before it has packaged these classes into a jar then the >> 'App' >> > > > > build >> > > > > > > will resolve ModuleB's target/classes directory >> > > > > > > * If the 'App' build starts after ModuleB has packaged >> its jar >> > > file >> > > > > > > then the 'App' build will resolve ModuleB's target/ModuleB.jar >> > > file. >> > > > > > > >> > > > > > > In many scenarios this dependency resolution inconsistency >> doesn't >> > > > > > > represent a challenge. However, it does cause an issue in our >> case >> > > > > because >> > > > > > > the 'App' POM has its Maven packaging stanza configured to >> war and >> > > in >> > > > > the >> > > > > > > scenario where ModuleB's target/classes directory is resolved >> by >> > > the >> > > > > 'App' >> > > > > > > then this results in the resultant 'App' war file being >> packaged >> > > with a >> > > > > > > completely empty ModuleB.jar file. >> > > > > > > >> > > > > > > Proposed solution >> > > > > > > >> > > > > > > Ideally we would like the Maven reactor to retain isolation >> > > between the >> > > > > > > two distinct 'dependency graphs' it constructs at >> instantiation >> > > > > throughout >> > > > > > > the entire Maven build. This would mean, in the simple example >> > > above, >> > > > > that >> > > > > > > the 'App' would always resolves ModuleB from the external repo >> > > > > (regardless >> > > > > > > of whether the reactor has built ModuleB or not in a separate >> > > > > 'dependency >> > > > > > > graph' in the reactor). >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > Joseph Leonard >> > > > > > > Manager >> > > > > > > >> > > > > > > Alfa >> > > > > > > ________________________________ >> > > > > > > e: joseph.leon...@alfasystems.com | w: alfasystems.com< >> > > > > > > https://www.alfasystems.com> >> > > > > > > t: +44 (0)20 7588 1800 | Moor Place, 1 Fore Street Avenue, >> London, >> > > EC2Y >> > > > > > > 9DT, GB >> > > > > > > ________________________________ >> > > > > > > >> > > > > > > The contents of this communication are not intended to be >> binding >> > > or >> > > > > > > constitute any form of offer or acceptance or give rise to any >> > > legal >> > > > > > > obligations on behalf of the sender or Alfa. The views or >> opinions >> > > > > > > expressed represent those of the author and not necessarily >> those >> > > of >> > > > > Alfa. >> > > > > > > This email and any attachments are strictly confidential and >> are >> > > > > intended >> > > > > > > solely for use by the individual or entity to whom it is >> > > addressed. If >> > > > > you >> > > > > > > are not the addressee (or responsible for delivery of the >> message >> > > to >> > > > > the >> > > > > > > addressee) you may not copy, forward, disclose or use any >> part of >> > > the >> > > > > > > message or its attachments. At present the integrity of email >> > > across >> > > > > the >> > > > > > > internet cannot be guaranteed and messages sent via this >> medium are >> > > > > > > potentially at risk. All liability is excluded to the extent >> > > permitted >> > > > > by >> > > > > > > law for any claims arising as a result of the use of this >> medium to >> > > > > > > transmit information by or to Alfa or its affiliates. >> > > > > > > >> > > > > > > Alfa Financial Software Ltd >> > > > > > > Reg. in England No: 0248 2325 >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
