I can confirm the buildplan plugin indicates the correct order with v0.3. Thanks again! Delany
On Thu, 29 May 2025 at 11:01, Sergey Chernov <serega.mo...@gmail.com> wrote: > Hey, Delany. > > > I'm running with this extension going forward. I don't use the build > cache (yet) so that's not an issue. > > I recommend you to try > https://github.com/seregamorph/maven-surefire-cached > (free) which is compatible with this extension (and supports jacoco reports > as well). > > > We have one huge module though (project we're calling it now) with > extensive tests, and its a bottleneck for all that follow. > > Exactly. This is the type of the bottleneck that this extension solves. > > > I did notice jacoco coverage was disabled as I bound the > jacoco-maven-plugin report goal to the prepare-package phase. > > The jacoco should work. Maybe indeed the phase of the plugin should be > adjusted. Feel free to contact me directly or submit an issue request on > github. > > > But can you make it work with the buildplan-maven-plugin:list-phases goal > so I can actually see which phases runs when, or is that up to this plugin > to figure out? > > Thank you very much for this question! It's addressed now (and also > mentioned in the readme), please try extension version 0.3 with the fix. > > On Tue, May 27, 2025 at 10:17 PM Sergey Chernov <serega.mo...@gmail.com> > wrote: > > > Hey, Guillaume. > > > > Thanks for your feedback, I appreciate it. > > > > > Regarding your suggestion about upfront downloads > > > > It was my reflection (assumption) about takari-smart-builder, the turbo > > builder does not load upfront. > > > > > The Maven 4 concurrent builder decomposes projects at the phase level > > > > This is nice. This looks somehow similar to how Gradle is scheduling > tasks > > graph (instead of modules) having granular build graph. I was also > thinking > > this direction as long-term goal. > > > > > I’d love to hear more about your PoC and explore how it could inform > > Maven’s evolution. Have you considered testing it against Maven 4’s > > concurrent builder to compare performance and compatibility? > > > > To be honest, I wasn't aware about this refactoring in Maven 4 (all > > discussions I've seen were focused on the separate deployed POM > feature). I > > will take a look for sure. Unfortunately, first experiments with building > > project locally on Maven 4 are not promising regarding performance (for > > clarity: I removed all extensions to avoid side effects of non-standard > > solutions). If you can give some guidance how can I inspect bottlenecks > of > > the Maven 4 builds, it would be great. I tried to analyze it via > VisualVM - > > there is no obvious heap or thread leakage. But it looks like the maven > > build is gradually loosing speed - the logs are printed slower and > slower. > > I think I can try takari-timeline (BTW I 😍 it), but not sure it'll give > > answers. Theoretically this can be related to kotlin plugin, I don't > know. > > > > On Tue, May 27, 2025 at 9:51 PM Guillaume Nodet <gno...@apache.org> > wrote: > > > >> Hi Sergey, > >> > >> Your maven-turbo-builder sounds very similar to the concurrent builder > >> in Maven 4, so I’d like to share some insights for comparison. > >> > >> The Maven 4 concurrent builder decomposes projects at the phase level, > >> with phases ordered based on fine-grained constraints. Unlike the > >> traditional builder, these constraints allow more flexibility. For > >> instance, if project A depends on project B at test scope, both can > >> start building concurrently, but A’s test phase will wait until B > >> completes its necessary phases (e.g., resource preparation, > >> compilation, and class post-processing) to ensure the required > >> artifacts are available. Downloads are deferred to avoid pausing the > >> build unnecessarily, which aligns with optimizing build performance. > >> > >> To maintain compatibility with existing Maven behavior, the standard > >> phase order is preserved. However, with the concurrent builder, phases > >> form a dependency tree rather than a linear list, as detailed in the > >> Maven 4 lifecycle documentation [1]. This structure allows phases like > >> package to run independently of test, and intra-project dependencies > >> can rely on a ready phase, which precedes package. This enables > >> earlier scheduling of downstream tasks. > >> > >> Regarding your suggestion about upfront downloads, I agree it could be > >> beneficial to initiate downloads early with a strategy to prioritize > >> artifacts as needed. However, since dependency downloads typically > >> occur only once per project, their impact on overall build time may be > >> less significant compared to phase scheduling optimizations. > >> > >> The maven-turbo-builder’s approach—scheduling downstream builds after > >> the upstream package phase and reordering test before package—is an > >> interesting optimization. It seems to align with the concurrent > >> builder’s philosophy but takes a more aggressive stance by breaking > >> from traditional phase ordering. Your reported 20m to 14m reduction in > >> build time for a large project like Miro’s monolith is impressive and > >> highlights the potential of such optimizations on multi-core systems. > >> > >> I’d love to hear more about your PoC and explore how it could inform > >> Maven’s evolution. Have you considered testing it against Maven 4’s > >> concurrent builder to compare performance and compatibility? I’ll take > >> a look at your code in the maven-turbo-builder subdirectory and try it > >> out on a sample project. > >> > >> Thanks for sharing, and I’m excited to continue this discussion! > >> > >> Cheers, > >> Guillaume Nodet > >> > >> > >> [1] > >> > https://maven.apache.org/ref/4.0.0-rc-3/maven-impl-modules/maven-core/lifecycles.html > >> > >> Le mar. 27 mai 2025 à 21:26, Sergey Chernov <serega.mo...@gmail.com> a > >> écrit : > >> > > >> > Thanks for asking, I planned to raise this topic soon in the mailing > >> list > >> > > >> > Here you can find a presentation board > >> > https://miro.com/app/board/uXjVLYUPRas=/ which also has a 5 minute > >> video > >> > with explanation (click Play on the first slide, no auth is required). > >> > > >> > Long story short: standard Maven builder (either single or > >> multi-threaded) > >> > has a pretty strong contract: it schedules the module to be built not > >> > earlier than all upstream dependencies are resolved (if it's a jar > >> library > >> > - it's downloaded from remote to .m2, if it's another module - ALL > build > >> > phases are executed). And it's a bottleneck in multi-module projects > >> built > >> > on multi-core CPUs. > >> > > >> > It looks like takari-smart-builder respects this contract, but > schedules > >> > modules in different order, also resolving libraries eagerly for > >> > optimization purposes. > >> > > >> > The maven-turbo-builder is a more radical change: > >> > > >> > 1. it schedules downstream dependency build once the upstream > >> finished > >> > package phase (instead of all requested phases), because package > >> creates > >> > jar artifact which is needed by downstream projects > >> > 2. it changes order of test and package phases (test phases are > >> before > >> > package by default), because it allows to schedule downstream > >> > dependencies earlier. In other words, it suggests optimistic > >> strategy there > >> > instead of original pessimistic > >> > > >> > As a result, this solution leverages multi-core CPUs way more > >> efficiently. > >> > E.g., for our huge Miro monolith application the test execution time > >> (with > >> > compiling and running only unit tests) is reduced from 20m to 14m. The > >> > parallelization is so good that I even have to use less cores than > T1C, > >> > like T0.6C because of Amdahl's law effects. The takari-smart-builder, > >> for > >> > comparison, gives a very small boost (few percent if all library > >> > dependencies are already downloaded to .m2). > >> > > >> > In the project readme you can find extension adoption for sample > >> > maven-surefire project or validate it on your own. > >> > > >> > The extension is also considered as PoC to change the standard > behavior > >> of > >> > Maven, which was quite good when it was created 20 years ago. At this > >> > moment we did not have multi-core machines, but today this can be > >> revisited > >> > to maximize build performance. Also *this is a breaking change* for > >> plugins > >> > that rely on the order or phases, e.g. when "test-jar" classified > >> artifact > >> > is used as a dependency (which should be avoided IMO 😊). There can be > >> > different strategies like opt-in this behavior or introduce a new > phase > >> > before test. While these disadvantages may look severe, the advantage > >> is a > >> > huge potential performance boost. > >> > > >> > I'm curious about your opinion about it and it would be great if you > can > >> > try it (you can inspect the code, it's very small - see > >> maven-turbo-builder > >> > subdir; other modules are side playground projects which are not > >> directly > >> > related to turbo builder extension). > >> > > >> > > >> > On Tue, May 27, 2025 at 5:26 PM Tamás Cservenák <ta...@cservenak.net> > >> wrote: > >> > > >> > > Hey Sergey, > >> > > > >> > > Can you tell us more about this "pet project" of yours? > >> > > > >> > > How is it different from, for example, the takari-smart-builder? > >> > > > >> > > Thanks > >> > > T > >> > > > >> > > > --------------------------------------------------------------------- > >> > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > >> > > For additional commands, e-mail: dev-h...@maven.apache.org > >> > > > >> > > > >> > >> > >> > >> -- > >> ------------------------ > >> Guillaume Nodet > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > >> For additional commands, e-mail: dev-h...@maven.apache.org > >> > >> >
