Hi Nils, Ah I see. We actually already use ‘-pl …’ arguments in conjunction with the Develocity Build Cache implementation because we have found this to still generate very significant time savings. Primarily because even with hits on the build cache there is still fairly substantial time spent resolving cached artifacts (and the reactor doing all its goal steps and determining cache hits). The other advantage of using ‘-pl …’ is that it gives opportunity for even greater parallelism (per my sub-thread example with Tamás). Joe
On 2024/02/07 10:59:28 Nils Breunese wrote: > Hi Jospeh, > > I didn’t necessarily expect that enabling the extension would solve/avoid the > issue you described, but I mentioned it, because it would allow you to not > have to specify an argument like '-pl > TestSupportModule1,TestSupportModule2,ModuleB,App’ in the first place, > because the Build Cache Extension should automatically determine which > modules need to be built and for which ones previously cached artifacts can > be used. > > Nils. > > > Op 6 feb 2024, om 15:11 heeft Joseph Leonard <jo...@alfasystems.com> het > > volgende geschreven: > > > > Thanks Nils, > > maven-build-cache-extension looks very interesting generally – I will have > > a play with it. > > With regards to this issue, the maven-build-cache-extension overview > > references “Subtree support for multimodule projects builds part of the > > codebase in isolation” which sounded similar to the solution I am proposing > > for this issue. Unfortunately, after enabling the > > maven-build-cache-extension I still hit the same issue. > > Joe > > > > On 2024/02/06 12:54:59 Nils Breunese wrote: > >> I can’t comment on your question directly, but I just wanted to say that > >> your use case sounds like it could benefit from the Maven Build Cache > >> Extension > >> (https://maven.apache.org/extensions/maven-build-cache-extension/). > > > >> > >> Just my 2 cents. > >> > >> Nils. > >> > >>> Op 6 feb 2024 om 11:40 heeft Joseph Leonard <jo...@alfasystems.com> het > >>> volgende geschreven: > >>> > >>> Hi all, > >>> > >>> It would be great to get any thoughts on whether the following is a > >>> defect: > >>> > >>> > >>> Issue details: > >>> tl;dr > >>> > >>> Maven can resolve dependencies either from: > >>> > >>> * an external repo > >>> * a class directory of a module being built within the reactor > >>> * a packaged jar of a module being built within the reactor > >>> > >>> If you run a concurrent multi-module build it is possible to get a race > >>> condition whereby the build of module Foo may resolve module Bar from > >>> either of the three resolution channels. This inconsistency can result in > >>> the Maven war plugin sometimes failing to build a functional war file. I > >>> would expect a consistent resolution would always take place. > > > >>> > >>> Full details > >>> Scenario > >>> > >>> Consider you have a repo with the following structure: > >>> > >>> App > >>> > >>> / \ > >>> > >>> / \ > >>> > >>> (compile scope) (test scope) > >>> > >>> / \ > >>> > >>> \/_ _\/ > >>> > >>> ModuleA TestSupportModule1 > >>> > >>> / > >>> > >>> / > >>> > >>> (compile scope) > >>> > >>> / > >>> > >>> \/_ > >>> > >>> ModuleB > >>> > >>> / > >>> > >>> / > >>> > >>> (test scope) > >>> > >>> / > >>> > >>> \/_ > >>> > >>> TestSupportModule2 > >>> > >>> If you were to make a src code change to the following test support > >>> modules: > >>> > >>> * TestSupportModule1 > >>> * TestSupportModule2 > >>> > >>> Then the minimum number of modules we need to build to verify the change > >>> set is OK is: > >>> > >>> * TestSupportModule1 > >>> * TestSupportModule2 > >>> * ModuleB > >>> * App > >>> > >>> i.e. there is no requirement to build ModuleA because we know that none > >>> of the src code changes could impact the classpaths used in its maven > >>> build. > > > >>> > >>> We know that despite 'App' depending (transitively) on ModuleB there is > >>> no need for the 'App' build to wait for ModuleB to complete its build > >>> because the src code change to TestSupportModule2 will not impact any of > >>> the classpaths used in the App maven build. Therefore to get the most > >>> efficient build possible we ideally would invoke Maven to run with 2 > >>> threads and with instruction to build two distinct 'dependency graphs': > > > >>> > >>> * TestSupportModule1 followed by ModuleB > >>> * TestSupportModule1 followed by App > >>> > >>> The following Maven command achieves exactly what we want because the > >>> reactor build order is based only on the direct (i.e. non-transitive) > >>> dependencies of the modules provided to the reactor in the build command. > >>> Therefore the absence of ModuleA results in two distinct 'dependency > >>> graphs': > > > >>> > >>> mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App -T > >>> 2 > >>> > >>> Note: In reality the code base I maintain has a very large monobuild with > >>> 100s of modules and this type of build optimisation makes a significant > >>> difference to the speed of our monobuild (we use > >>> https://github.com/gitflow-incremental-builder/gitflow-incremental-builder > >>> to automate the logic of determining which modules to include in the > >>> reactor based on our change set). > > > >>> > >>> Issue > >>> > >>> We have encountered an issue in the above scenario because the 'App' > >>> build has a race condition with the ModuleB build which will result in > >>> one of the following three outcomes: > > > >>> > >>> * If the 'App' build starts before the ModuleB build has compiled its > >>> src classes then the 'App' build will resolve ModuleB from the external > >>> repo (i.e. equivalent to ModuleB not being in the reactor at all) > > > >>> * If the 'App' build starts after ModuleB has compiled its src classes > >>> but before it has packaged these classes into a jar then the 'App' build > >>> will resolve ModuleB's target/classes directory > > > >>> * If the 'App' build starts after ModuleB has packaged its jar file > >>> then the 'App' build will resolve ModuleB's target/ModuleB.jar file. > > > >>> > >>> In many scenarios this dependency resolution inconsistency doesn't > >>> represent a challenge. However, it does cause an issue in our case > >>> because the 'App' POM has its Maven packaging stanza configured to war > >>> and in the scenario where ModuleB's target/classes directory is resolved > >>> by the 'App' then this results in the resultant 'App' war file being > >>> packaged with a completely empty ModuleB.jar file. > > > >>> > >>> Proposed solution > >>> > >>> Ideally we would like the Maven reactor to retain isolation between the > >>> two distinct 'dependency graphs' it constructs at instantiation > >>> throughout the entire Maven build. This would mean, in the simple example > >>> above, that the 'App' would always resolves ModuleB from the external > >>> repo (regardless of whether the reactor has built ModuleB or not in a > >>> separate 'dependency graph' in the reactor). > > > >>> > >>> > >>> > >>> Joseph Leonard > >>> Manager > >>> > >>> Alfa > >>> ________________________________ > >>> e: joseph.leon...@alfasystems.com | w: > >>> alfasystems.com<https://www.alfasystems.com> > >>> t: +44 (0)20 7588 1800 | Moor Place, 1 Fore Street Avenue, London, EC2Y > >>> 9DT, GB > >>> ________________________________ > >>> > >>> The contents of this communication are not intended to be binding or > >>> constitute any form of offer or acceptance or give rise to any legal > >>> obligations on behalf of the sender or Alfa. The views or opinions > >>> expressed represent those of the author and not necessarily those of > >>> Alfa. This email and any attachments are strictly confidential and are > >>> intended solely for use by the individual or entity to whom it is > >>> addressed. If you are not the addressee (or responsible for delivery of > >>> the message to the addressee) you may not copy, forward, disclose or use > >>> any part of the message or its attachments. At present the integrity of > >>> email across the internet cannot be guaranteed and messages sent via this > >>> medium are potentially at risk. All liability is excluded to the extent > >>> permitted by law for any claims arising as a result of the use of this > >>> medium to transmit information by or to Alfa or its affiliates. > > > >>> > >>> Alfa Financial Software Ltd > >>> Reg. in England No: 0248 2325 > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > For additional commands, e-mail: users-h...@maven.apache.org > >
