Hey, Delany. > I'm running with this extension going forward. I don't use the build cache (yet) so that's not an issue.
I recommend you to try https://github.com/seregamorph/maven-surefire-cached (free) which is compatible with this extension (and supports jacoco reports as well). > We have one huge module though (project we're calling it now) with extensive tests, and its a bottleneck for all that follow. Exactly. This is the type of the bottleneck that this extension solves. > I did notice jacoco coverage was disabled as I bound the jacoco-maven-plugin report goal to the prepare-package phase. The jacoco should work. Maybe indeed the phase of the plugin should be adjusted. Feel free to contact me directly or submit an issue request on github. > But can you make it work with the buildplan-maven-plugin:list-phases goal so I can actually see which phases runs when, or is that up to this plugin to figure out? Thank you very much for this question! It's addressed now (and also mentioned in the readme), please try extension version 0.3 with the fix. On Tue, May 27, 2025 at 10:17 PM Sergey Chernov <serega.mo...@gmail.com> wrote: > Hey, Guillaume. > > Thanks for your feedback, I appreciate it. > > > Regarding your suggestion about upfront downloads > > It was my reflection (assumption) about takari-smart-builder, the turbo > builder does not load upfront. > > > The Maven 4 concurrent builder decomposes projects at the phase level > > This is nice. This looks somehow similar to how Gradle is scheduling tasks > graph (instead of modules) having granular build graph. I was also thinking > this direction as long-term goal. > > > I’d love to hear more about your PoC and explore how it could inform > Maven’s evolution. Have you considered testing it against Maven 4’s > concurrent builder to compare performance and compatibility? > > To be honest, I wasn't aware about this refactoring in Maven 4 (all > discussions I've seen were focused on the separate deployed POM feature). I > will take a look for sure. Unfortunately, first experiments with building > project locally on Maven 4 are not promising regarding performance (for > clarity: I removed all extensions to avoid side effects of non-standard > solutions). If you can give some guidance how can I inspect bottlenecks of > the Maven 4 builds, it would be great. I tried to analyze it via VisualVM - > there is no obvious heap or thread leakage. But it looks like the maven > build is gradually loosing speed - the logs are printed slower and slower. > I think I can try takari-timeline (BTW I 😍 it), but not sure it'll give > answers. Theoretically this can be related to kotlin plugin, I don't know. > > On Tue, May 27, 2025 at 9:51 PM Guillaume Nodet <gno...@apache.org> wrote: > >> Hi Sergey, >> >> Your maven-turbo-builder sounds very similar to the concurrent builder >> in Maven 4, so I’d like to share some insights for comparison. >> >> The Maven 4 concurrent builder decomposes projects at the phase level, >> with phases ordered based on fine-grained constraints. Unlike the >> traditional builder, these constraints allow more flexibility. For >> instance, if project A depends on project B at test scope, both can >> start building concurrently, but A’s test phase will wait until B >> completes its necessary phases (e.g., resource preparation, >> compilation, and class post-processing) to ensure the required >> artifacts are available. Downloads are deferred to avoid pausing the >> build unnecessarily, which aligns with optimizing build performance. >> >> To maintain compatibility with existing Maven behavior, the standard >> phase order is preserved. However, with the concurrent builder, phases >> form a dependency tree rather than a linear list, as detailed in the >> Maven 4 lifecycle documentation [1]. This structure allows phases like >> package to run independently of test, and intra-project dependencies >> can rely on a ready phase, which precedes package. This enables >> earlier scheduling of downstream tasks. >> >> Regarding your suggestion about upfront downloads, I agree it could be >> beneficial to initiate downloads early with a strategy to prioritize >> artifacts as needed. However, since dependency downloads typically >> occur only once per project, their impact on overall build time may be >> less significant compared to phase scheduling optimizations. >> >> The maven-turbo-builder’s approach—scheduling downstream builds after >> the upstream package phase and reordering test before package—is an >> interesting optimization. It seems to align with the concurrent >> builder’s philosophy but takes a more aggressive stance by breaking >> from traditional phase ordering. Your reported 20m to 14m reduction in >> build time for a large project like Miro’s monolith is impressive and >> highlights the potential of such optimizations on multi-core systems. >> >> I’d love to hear more about your PoC and explore how it could inform >> Maven’s evolution. Have you considered testing it against Maven 4’s >> concurrent builder to compare performance and compatibility? I’ll take >> a look at your code in the maven-turbo-builder subdirectory and try it >> out on a sample project. >> >> Thanks for sharing, and I’m excited to continue this discussion! >> >> Cheers, >> Guillaume Nodet >> >> >> [1] >> https://maven.apache.org/ref/4.0.0-rc-3/maven-impl-modules/maven-core/lifecycles.html >> >> Le mar. 27 mai 2025 à 21:26, Sergey Chernov <serega.mo...@gmail.com> a >> écrit : >> > >> > Thanks for asking, I planned to raise this topic soon in the mailing >> list >> > >> > Here you can find a presentation board >> > https://miro.com/app/board/uXjVLYUPRas=/ which also has a 5 minute >> video >> > with explanation (click Play on the first slide, no auth is required). >> > >> > Long story short: standard Maven builder (either single or >> multi-threaded) >> > has a pretty strong contract: it schedules the module to be built not >> > earlier than all upstream dependencies are resolved (if it's a jar >> library >> > - it's downloaded from remote to .m2, if it's another module - ALL build >> > phases are executed). And it's a bottleneck in multi-module projects >> built >> > on multi-core CPUs. >> > >> > It looks like takari-smart-builder respects this contract, but schedules >> > modules in different order, also resolving libraries eagerly for >> > optimization purposes. >> > >> > The maven-turbo-builder is a more radical change: >> > >> > 1. it schedules downstream dependency build once the upstream >> finished >> > package phase (instead of all requested phases), because package >> creates >> > jar artifact which is needed by downstream projects >> > 2. it changes order of test and package phases (test phases are >> before >> > package by default), because it allows to schedule downstream >> > dependencies earlier. In other words, it suggests optimistic >> strategy there >> > instead of original pessimistic >> > >> > As a result, this solution leverages multi-core CPUs way more >> efficiently. >> > E.g., for our huge Miro monolith application the test execution time >> (with >> > compiling and running only unit tests) is reduced from 20m to 14m. The >> > parallelization is so good that I even have to use less cores than T1C, >> > like T0.6C because of Amdahl's law effects. The takari-smart-builder, >> for >> > comparison, gives a very small boost (few percent if all library >> > dependencies are already downloaded to .m2). >> > >> > In the project readme you can find extension adoption for sample >> > maven-surefire project or validate it on your own. >> > >> > The extension is also considered as PoC to change the standard behavior >> of >> > Maven, which was quite good when it was created 20 years ago. At this >> > moment we did not have multi-core machines, but today this can be >> revisited >> > to maximize build performance. Also *this is a breaking change* for >> plugins >> > that rely on the order or phases, e.g. when "test-jar" classified >> artifact >> > is used as a dependency (which should be avoided IMO 😊). There can be >> > different strategies like opt-in this behavior or introduce a new phase >> > before test. While these disadvantages may look severe, the advantage >> is a >> > huge potential performance boost. >> > >> > I'm curious about your opinion about it and it would be great if you can >> > try it (you can inspect the code, it's very small - see >> maven-turbo-builder >> > subdir; other modules are side playground projects which are not >> directly >> > related to turbo builder extension). >> > >> > >> > On Tue, May 27, 2025 at 5:26 PM Tamás Cservenák <ta...@cservenak.net> >> wrote: >> > >> > > Hey Sergey, >> > > >> > > Can you tell us more about this "pet project" of yours? >> > > >> > > How is it different from, for example, the takari-smart-builder? >> > > >> > > Thanks >> > > T >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> > > For additional commands, e-mail: dev-h...@maven.apache.org >> > > >> > > >> >> >> >> -- >> ------------------------ >> Guillaume Nodet >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> For additional commands, e-mail: dev-h...@maven.apache.org >> >>
