Hey, Yeah, exactly: ideally we want module-b to be resolved in different ways within the same reactor. I noticed that this kind of already happens (albeit in a simpler way): In my reproducer testing if you play with the test timeouts to dictate the module build order then with the same modules in the reactor you can get different resolution behaviour: * If "app" builds before "module-b" has started compiling then "app" will resolve "module-b" from the local repo * If "app" builds after "module-b" has compiled but before "module-b" has packaged then "app" will resolve the "module-b" class directory (this is the out-of-the-box scenario the reproducer produces) * If "app" builds after "module-b" has packaged then "app" will resolve the "module-b" jar from the "module-b" target directory
So within this logic to perform different types of resolves, it would be neat if the reactor could perform an initial check first: * if the dependency being resolved is within the same 'sub-build graph' of the module requesting the resolve then look within the reactor (current behaviour) * otherwise, only resolve from the local repository I don't know whether the code architecture means this is quite easy to do or very difficult. But to be fair, I think this extra parallelism opportunity is a bit of niche edge case and it probably isn't worth it if this is very complex to achieve (or considered a smell generally by the community!) On 2024/02/08 Tamás Cservenák wrote > Howdy, > > Well, sadly, Maven currently has no means to make one module "this and that" > at the same time. > It is either in the reactor or is out of it. > To me, it sounds like you want one reactor, but where module-b is "out" and > 'in", at the same time (within same session) > > > T > > On Thu, Feb 8, 2024 at 1:27 PM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > > > Hey, > > We don't need the module-b jar built in the reactor to be included in > > the war if the changes made in module-b won't affect the war. Instead, > > in this scenario, we're fine with using a jar resolved from the local > > repository. > > The requisites for this scenario are: > > > > * Any change made to module-b is only in its test tree > > * The only changes made to the direct and transitive dependencies of > > module-b are to its test scoped dependencies So basically module-b is > > only in the reactor for verification purposes rather than ‘src > > packaging’ purposes. > > Thanks, > > Joe > > > > On 2024/02/08 12:12:03 Tamás Cservenák wrote: > > > Hej, > > > > > > yes, basically the "hack" is well described by you: > > > - it retains "full' reactor project > > > - introduces "needs build" on project > > > - when it comes to building, it skips the project if flag set > > > > > > This results that module-b -> app link is not lost. > > > And as for your parallelization request: the truth is that there IS > > > and edge between app and module-b (via module-a), so despite > > > module-a is "done", its transitive module-b is not done, hence you > > > cannot build these two in parallel, as you do have an edge between app > > > and module-b. > > > > > > Or to invert the question: what module-b JAR you want to have > > > included in app WAR, when building? The one built, or the one from > > > the local repository? If the latter, why are you building it in the first > > > place? > > > > > > On Thu, Feb 8, 2024 at 12:06 PM Joseph Leonard < > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > Hi Tamás, > > > > Thank you for hacking around this so rapidly. I am not familiar > > > > with > > the > > > > maven code base, so let me know if I have misinterpreted the > > > > change: Is your hack to: > > > > > > > > 1. Retain the full multi-module build dependency graph such > > > > that the build graph always remains consistent regardless of whether a > > > > “-pl” > > arg has > > > > been provided > > > > 2. Introduce a skip build option to satisfy the functionality > > > > of the “-pl” arg: The reactor will report the module has been > > > > built but the > > build > > > > is basically a no-op > > > > If so, this does resolve the issue but at the cost of losing the > > > > opportunity to gain extra parallelism in the build. > > > > What I was wondering may be possible (without any idea of how > > > > feasible > > it > > > > would be to implement) would be the opportunity to recognise in > > > > the multi-build distinct build graphs and then keep isolation in > > > > the > > reactor > > > > between these graphs. To elaborate, in the reproducer scenario > > > > there > > would > > > > be two distinct graphs: > > > > > > > > * "testsupport-module-1" followed by "app" > > > > * "testsupport-module-2" followed by "module-b" > > > > Ideally these two graphs build in parallel to each other and even > > > > if ultimately there is a dependency between them they NEVER > > > > resolve that dependency from within reactor they instead always > > > > resolve from the external Maven cache. The key statement here is > > > > that if a user > > presents a > > > > list of modules to be built then they really need to be sure that > > > > the DIRECT dependencies between these modules truly represent the > > > > build > > > > graph(s) they want. For tools like gitflow-incremental-builder > > > > which > > are > > > > designed to carefully work out change sets and what needs to build > > > > this offers a really powerful opportunity to optimise builds. I > > > > appreciate > > this > > > > functionality could also be seen as a gotcha (although tbf this > > > > already exists). > > > > Thanks, > > > > Joe > > > > > > > > > > > > On 2024/02/08 10:09:31 Tamás Cservenák wrote: > > > > > Seems we are on track with this. To prove my last-night theory I > > created > > > > a > > > > > "hack" (is really just that) and guess what? > > > > > It makes reproducer behave "as expected": > > > > > https://github.com/apache/maven/pull/1406 > > > > > > > > > > T > > > > > > > > > > On Wed, Feb 7, 2024 at 10:05 PM Tamás Cservenák > > > > > <ta...@cservenak.net > > > > > > > wrote: > > > > > > > > > > > Howdy, > > > > > > > > > > > > Thank you very much, the reproducer works. Did not dig thru it > > fully, > > > > but > > > > > > here are some related issues: > > > > > > > > > > > > https://issues.apache.org/jira/browse/MNG-8028 (funny thing, I > > created > > > > > > this few weeks ago) > > > > > > https://issues.apache.org/jira/browse/MNG-6300 > > > > > > > > > > > > Will report back tomorrow (EU TZ) T > > > > > > > > > > > > > > > > > > On Wed, Feb 7, 2024 at 7:48 PM Joseph Leonard < > > > > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > > > > > >> Hi Tamás, > > > > > >> I have created a simple example here: > > > > > >> https://github.com/josple/mvn-multibuild-issue > > > > > >> Hopefully the README is clear enough – let me know if I can > > clarify > > > > > >> anything. > > > > > >> Thanks, > > > > > >> Joe > > > > > >> > > > > > >> On 2024/02/07 17:33:08 Tamás Cservenák wrote: > > > > > >> > Howdy, > > > > > >> > > > > > > >> > In that case, there is something fishy with the project, my > > blind > > > > guess > > > > > >> > would be some "hidden" inter-module dependency maybe? > > > > > >> > > > > > > >> > Can you provide access to source, or, if not feasible, could you > > > > provide > > > > > >> > some reproducer and publish it on Github/Gitlab/whatever (maybe > > even > > > > > >> just > > > > > >> > send it privately as ML strips off attachments and images) for > > us > > > > to see > > > > > >> > this in action? > > > > > >> > > > > > > >> > Thanks > > > > > >> > T > > > > > >> > > > > > > >> > On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < > > > > > >> > joseph.leon...@alfasystems.com> wrote: > > > > > >> > > > > > > >> > > Hi Tamás, > > > > > >> > > We have previously played around a bit with mvnd but not > > takari > > > > > >> directly – > > > > > >> > > I will have a play with it. With regards to this issue, using > > the > > > > > >> takari > > > > > >> > > smart builder unfortunately doesn’t resolve the issue. > > > > > >> > > Joe > > > > > >> > > > > > > > >> > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: > > > > > >> > > > Can you please try smart builder instead? > > > > > >> > > > https://github.com/takari/takari-smart-builder > > > > > >> > > > > > > > > >> > > > (note: smart builder is used by mvnd as well) > > > > > >> > > > > > > > > >> > > > The difference between the two can be seen here: > > > > > >> > > > > > http://takari.io/book/30-team-maven.html#takari-smart-builder > > > > > >> > > > > > > > > >> > > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < > > > > > >> > > > joseph.leon...@alfasystems.com> wrote: > > > > > >> > > > > > > > > >> > > > > Hi Tamás, > > > > > >> > > > > Yeah, this was unexpected to me initially as well. From > > what I > > > > > >> can tell > > > > > >> > > > > the Maven reactor only considers direct dependencies > > (i.e. not > > > > > >> > > transitive > > > > > >> > > > > dependencies) between the modules in the reactor when > > working > > > > out > > > > > >> the > > > > > >> > > build > > > > > >> > > > > graph. For example if you have a simple linear dependency > > > > chain > > > > > >> of: > > > > > >> > > > > One --> Two --> Three --> Four --> Five > > > > > >> > > > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 > > > > will > > > > > >> result > > > > > >> > > in > > > > > >> > > > > two ‘graphs’ being built in parallel ([One,Two] and > > > > [Four,Five]). > > > > > >> I > > > > > >> > > assume > > > > > >> > > > > this is as designed because it actually offers quite > > powerful > > > > > >> > > functionality > > > > > >> > > > > to improve the parallelism in your build. An example of > > where > > > > > >> this is > > > > > >> > > legit > > > > > >> > > > > is when: > > > > > >> > > > > > > > > > >> > > > > * “Four” has a test scope dependency on “Five” > > > > > >> > > > > * “One” has a test scoped dependency on “Two” > > > > > >> > > > > If you made a src code change to “Five” and “Two” then it > > > > would be > > > > > >> > > safe to > > > > > >> > > > > build [One,Two] and [Four,Five] in parallel because you > > know > > > > the > > > > > >> > > changes > > > > > >> > > > > within these graphs cannot impact each other. > > > > > >> > > > > Joe > > > > > >> > > > > > > > > > >> > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > > > > >> > > > > > Howdy, > > > > > >> > > > > > > > > > > >> > > > > > To me this looks like Maven is not aware that the App > > > > depends on > > > > > >> > > > > ModuleB... > > > > > >> > > > > > Are they "plain dependency" linked? Or what kind of > > > > dependency > > > > > >> we > > > > > >> > > talk > > > > > >> > > > > > about here? > > > > > >> > > > > > In short: why would App start while ModuleB (upstream > > dep) > > > > is > > > > > >> not > > > > > >> > > done? > > > > > >> > > > > > Something is fishy here. > > > > > >> > > > > > > > > > > >> > > > > > T > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > > > > >> > > > > > joseph.leon...@alfasystems.com> wrote: > > > > > >> > > > > > > > > > > >> > > > > > > Hi all, > > > > > >> > > > > > > > > > > > >> > > > > > > It would be great to get any thoughts on whether the > > > > > >> following is a > > > > > >> > > > > defect: > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > Issue details: > > > > > >> > > > > > > tl;dr > > > > > >> > > > > > > > > > > > >> > > > > > > Maven can resolve dependencies either from: > > > > > >> > > > > > > > > > > > >> > > > > > > * an external repo > > > > > >> > > > > > > * a class directory of a module being built > > within the > > > > > >> reactor > > > > > >> > > > > > > * a packaged jar of a module being built within > > the > > > > > >> reactor > > > > > >> > > > > > > > > > > > >> > > > > > > If you run a concurrent multi-module build it is > > possible > > > > to > > > > > >> get a > > > > > >> > > race > > > > > >> > > > > > > condition whereby the build of module Foo may resolve > > > > module > > > > > >> Bar > > > > > >> > > from > > > > > >> > > > > > > either of the three resolution channels. This > > > > inconsistency > > > > > >> can > > > > > >> > > result > > > > > >> > > > > in > > > > > >> > > > > > > the Maven war plugin sometimes failing to build a > > > > functional > > > > > >> war > > > > > >> > > file. > > > > > >> > > > > I > > > > > >> > > > > > > would expect a consistent resolution would always take > > > > place. > > > > > >> > > > > > > > > > > > >> > > > > > > Full details > > > > > >> > > > > > > Scenario > > > > > >> > > > > > > > > > > > >> > > > > > > Consider you have a repo with the following structure: > > > > > >> > > > > > > > > > > > >> > > > > > > App > > > > > >> > > > > > > > > > > > >> > > > > > > / \ > > > > > >> > > > > > > > > > > > >> > > > > > > / \ > > > > > >> > > > > > > > > > > > >> > > > > > > (compile scope) (test scope) > > > > > >> > > > > > > > > > > > >> > > > > > > / \ > > > > > >> > > > > > > > > > > > >> > > > > > > \/_ _\/ > > > > > >> > > > > > > > > > > > >> > > > > > > ModuleA TestSupportModule1 > > > > > >> > > > > > > > > > > > >> > > > > > > / > > > > > >> > > > > > > > > > > > >> > > > > > > / > > > > > >> > > > > > > > > > > > >> > > > > > > (compile scope) > > > > > >> > > > > > > > > > > > >> > > > > > > / > > > > > >> > > > > > > > > > > > >> > > > > > > \/_ > > > > > >> > > > > > > > > > > > >> > > > > > > ModuleB > > > > > >> > > > > > > > > > > > >> > > > > > > / > > > > > >> > > > > > > > > > > > >> > > > > > > / > > > > > >> > > > > > > > > > > > >> > > > > > > (test scope) > > > > > >> > > > > > > > > > > > >> > > > > > > / > > > > > >> > > > > > > > > > > > >> > > > > > > \/_ > > > > > >> > > > > > > > > > > > >> > > > > > > TestSupportModule2 > > > > > >> > > > > > > > > > > > >> > > > > > > If you were to make a src code change to the following > > > > test > > > > > >> support > > > > > >> > > > > > > modules: > > > > > >> > > > > > > > > > > > >> > > > > > > * TestSupportModule1 > > > > > >> > > > > > > * TestSupportModule2 > > > > > >> > > > > > > > > > > > >> > > > > > > Then the minimum number of modules we need to build to > > > > verify > > > > > >> the > > > > > >> > > > > change > > > > > >> > > > > > > set is OK is: > > > > > >> > > > > > > > > > > > >> > > > > > > * TestSupportModule1 > > > > > >> > > > > > > * TestSupportModule2 > > > > > >> > > > > > > * ModuleB > > > > > >> > > > > > > * App > > > > > >> > > > > > > > > > > > >> > > > > > > i.e. there is no requirement to build ModuleA because > > we > > > > know > > > > > >> that > > > > > >> > > > > none of > > > > > >> > > > > > > the src code changes could impact the classpaths used > > in > > > > its > > > > > >> maven > > > > > >> > > > > build. > > > > > >> > > > > > > > > > > > >> > > > > > > We know that despite 'App' depending (transitively) on > > > > ModuleB > > > > > >> > > there > > > > > >> > > > > is no > > > > > >> > > > > > > need for the 'App' build to wait for ModuleB to > > complete > > > > its > > > > > >> build > > > > > >> > > > > because > > > > > >> > > > > > > the src code change to TestSupportModule2 will not > > impact > > > > any > > > > > >> of > > > > > >> > > the > > > > > >> > > > > > > classpaths used in the App maven build. Therefore to > > get > > > > the > > > > > >> most > > > > > >> > > > > efficient > > > > > >> > > > > > > build possible we ideally would invoke Maven to run > > with 2 > > > > > >> threads > > > > > >> > > and > > > > > >> > > > > with > > > > > >> > > > > > > instruction to build two distinct 'dependency graphs': > > > > > >> > > > > > > > > > > > >> > > > > > > * TestSupportModule1 followed by ModuleB > > > > > >> > > > > > > * TestSupportModule1 followed by App > > > > > >> > > > > > > > > > > > >> > > > > > > The following Maven command achieves exactly what we > > want > > > > > >> because > > > > > >> > > the > > > > > >> > > > > > > reactor build order is based only on the direct (i.e. > > > > > >> > > non-transitive) > > > > > >> > > > > > > dependencies of the modules provided to the reactor > > in the > > > > > >> build > > > > > >> > > > > command. > > > > > >> > > > > > > Therefore the absence of ModuleA results in two > > distinct > > > > > >> > > 'dependency > > > > > >> > > > > > > graphs': > > > > > >> > > > > > > > > > > > >> > > > > > > mvn clean verify -pl > > > > > >> > > TestSupportModule1,TestSupportModule2,ModuleB,App > > > > > >> > > > > -T 2 > > > > > >> > > > > > > > > > > > >> > > > > > > Note: In reality the code base I maintain has a very > > large > > > > > >> > > monobuild > > > > > >> > > > > with > > > > > >> > > > > > > 100s of modules and this type of build optimisation > > makes > > > > a > > > > > >> > > significant > > > > > >> > > > > > > difference to the speed of our monobuild (we use > > > > > >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > https://github.com/gitflow-incremental-builder/gitflow-incremental-builder > > > > > >> > > > > > > to automate the logic of determining which modules to > > > > include > > > > > >> in > > > > > >> > > the > > > > > >> > > > > > > reactor based on our change set). > > > > > >> > > > > > > > > > > > >> > > > > > > Issue > > > > > >> > > > > > > > > > > > >> > > > > > > We have encountered an issue in the above scenario > > > > because the > > > > > >> > > 'App' > > > > > >> > > > > build > > > > > >> > > > > > > has a race condition with the ModuleB build which will > > > > result > > > > > >> in > > > > > >> > > one > > > > > >> > > > > of the > > > > > >> > > > > > > following three outcomes: > > > > > >> > > > > > > > > > > > >> > > > > > > * If the 'App' build starts before the ModuleB > > build > > > > has > > > > > >> > > compiled > > > > > >> > > > > its > > > > > >> > > > > > > src classes then the 'App' build will resolve ModuleB > > > > from the > > > > > >> > > external > > > > > >> > > > > > > repo (i.e. equivalent to ModuleB not being in the > > reactor > > > > at > > > > > >> all) > > > > > >> > > > > > > * If the 'App' build starts after ModuleB has > > compiled > > > > > >> its src > > > > > >> > > > > classes > > > > > >> > > > > > > but before it has packaged these classes into a jar > > then > > > > the > > > > > >> 'App' > > > > > >> > > > > build > > > > > >> > > > > > > will resolve ModuleB's target/classes directory > > > > > >> > > > > > > * If the 'App' build starts after ModuleB has > > packaged > > > > > >> its jar > > > > > >> > > file > > > > > >> > > > > > > then the 'App' build will resolve ModuleB's > > > > target/ModuleB.jar > > > > > >> > > file. > > > > > >> > > > > > > > > > > > >> > > > > > > In many scenarios this dependency resolution > > inconsistency > > > > > >> doesn't > > > > > >> > > > > > > represent a challenge. However, it does cause an > > issue in > > > > our > > > > > >> case > > > > > >> > > > > because > > > > > >> > > > > > > the 'App' POM has its Maven packaging stanza > > configured to > > > > > >> war and > > > > > >> > > in > > > > > >> > > > > the > > > > > >> > > > > > > scenario where ModuleB's target/classes directory is > > > > resolved > > > > > >> by > > > > > >> > > the > > > > > >> > > > > 'App' > > > > > >> > > > > > > then this results in the resultant 'App' war file > > being > > > > > >> packaged > > > > > >> > > with a > > > > > >> > > > > > > completely empty ModuleB.jar file. > > > > > >> > > > > > > > > > > > >> > > > > > > Proposed solution > > > > > >> > > > > > > > > > > > >> > > > > > > Ideally we would like the Maven reactor to retain > > > > isolation > > > > > >> > > between the > > > > > >> > > > > > > two distinct 'dependency graphs' it constructs at > > > > > >> instantiation > > > > > >> > > > > throughout > > > > > >> > > > > > > the entire Maven build. This would mean, in the simple > > > > example > > > > > >> > > above, > > > > > >> > > > > that > > > > > >> > > > > > > the 'App' would always resolves ModuleB from the > > external > > > > repo > > > > > >> > > > > (regardless > > > > > [message truncated...] > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
