Hi, 3/4 stack traces you posted contained the constructor frame with the Objects.hash() call.
Hash code over so many collections can scale badly once the collections grow: https://github.com/apache/maven-resolver/blob/145588be6d5b2c0111b5f367787d4c9347c326da/maven-resolver-util/src/main/java/org/eclipse/aether/util/graph/manager/AbstractDependencyManager.java#L118-L126 It will grab every key/value pair and call hashCode() on it to build the final value. If the dependency manager isn't put into Sets/Maps the hashcode could be potentially calculated lazily (which would ideally not compute it at all and avoid the problem). But I haven't looked at the code any deeper than that - maybe this isn't possible. (as sidenote: managedLocalPaths is missing in equals) I do like using async-profiler* in situations like this since it is easy to use and the flame graphs often provide a good overview showing relative CPU usage. But in this case you likely already found the bottleneck in the stacktraces you took. usage would be: sudo sysctl kernel.kptr_restrict=0 && sudo sysctl kernel.perf_event_paranoid=1 export MAVEN_OPTS=-agentpath:/path/to/async-profiler-4.0-linux-x64/lib/libasyncProfiler.so=start,event=cpu,file=maven.html run mvn command. Check maven.html. best regards, michael * https://github.com/async-profiler/async-profiler On 6/10/25 18:10, Sergey Chernov wrote: > I was able to run the Maven build (~900 modules) with the profiler. > Used `Apache Maven 4.0.0-rc-3` and the simplest possible command `mvn > initialize` which demonstrates reproducible degradation, the build is using > 6 worker threads. > Maven 3 runs this command in 5 sec, Maven 4 - 6 minutes (no mistake). > YourKit highlighted as the most hot > `org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(RepositorySystemSession, > CollectRequest)` method. > I collected several stacktraces during the build and found this common part > of the stacktrace > ``` > at > org.eclipse.aether.util.graph.manager.AbstractDependencyManager.deriveChildManager > at > org.eclipse.aether.internal.impl.collect.bf.BfDependencyCollector.doRecurse(BfDependencyCollector.java:361) > at > org.eclipse.aether.internal.impl.collect.bf.BfDependencyCollector.processDependency(BfDependencyCollector.java:320) > at > org.eclipse.aether.internal.impl.collect.bf.BfDependencyCollector.doCollectDependencies(BfDependencyCollector.java:202) > at > org.eclipse.aether.internal.impl.collect.DependencyCollectorDelegate.collectDependencies(DependencyCollectorDelegate.java:222) > at > org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:79) > at > org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:241) > at > org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:157) > at > org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:260) > at > org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectArtifacts(LifecycleDependencyResolver.java:208) > - locked <xxxxxxxxxxxx> (a > org.apache.maven.project.artifact.DefaultProjectArtifactsCache$CacheKey) > at > org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:128) > at > org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:368) > at > org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:307) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:214) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:179) > at > org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:168) > at > org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:165) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:110) > at > org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder.lambda$createBuildCallable$1(MultiThreadedBuilder.java:191) > at > org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$$Lambda$1189/0x0000000800526240.call(Unknown > Source) > at java.util.concurrent.FutureTask.run(java.base@17.0.14 > /FutureTask.java:264) > at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.14 > /Executors.java:539) > at java.util.concurrent.FutureTask.run(java.base@17.0.14 > /FutureTask.java:264) > at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.14 > /ThreadPoolExecutor.java:1136) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.14 > /ThreadPoolExecutor.java:635) > at java.lang.Thread.run(java.base@17.0.14/Thread.java:840) > ``` > As you see, the top method in the snippet is `deriveChildManager` and in > the captured stack traces there are different combinations of stacktraces > above it. But the below part is always the same. > In this gist you can find 4 real stacktraces of the same Maven process > captured during the build when it was already slow > https://gist.github.com/seregamorph/bace2d5087ba195279583704bb41a5c4 > > Things I'd like to focus attention is: it looks like it's degrading > smoothly, maybe the degradation depends on the depth of dependency tree, > maybe on the number of transitive dependencies (correlating values). But in > the very beginning (first 400 modules) it goes quite fast. It does not fail > with OOM, so probably the problem is around some nested iterations. > > Let me know what else I can do (including patching maven to have more > sensors) as I'm not sure how to provide a reproducible project model (it is > private). > I tried the gradle performance testing tool > https://gist.github.com/melix/c09f6d8d27b0005302cc3317c9c9be05 , but the > project that it generates does not reproduce the problem (probably because > the dependency tree of the huge multi-module project is almost flat). > > > On Fri, May 30, 2025 at 1:59 PM Xeno Amess <xenoam...@gmail.com> wrote: > >> @Guillaume Nodet >> [MNG-8510] Avoid duplicate Utils classes (#2173) >> >> https://github.com/JetBrains/intellij-community/blob/4054fd36584fb2bbb1ed983d7d36cefca040c2cf/plugins/maven/maven40-server-impl/src/com/intellij/maven/server/m40/Maven40ServerEmbedderImpl.java#L74 >> >> yes we can release the version 4.0.0, but after the releasing things like >> this class/file-names would not be changed easily (of course and other >> bigger things), I don't know if people be prepared for a...stablization >> >> Xeno Amess <xenoam...@gmail.com> 于2025年5月29日周四 17:21写道： >> >>> all running with skipTests >>> >>> log4j both success, mvn3 2:00 mvn4 1:50 >>> dubbo both success, mvn3 3:08 mvn4 3:19 >>> >>> hadoop failed due to network reason(for yarn frontend, well considering >>> about where I'm living it is normal...) >>> >>> for structs 2, 90%+ time is wasted on antrun for docs generation in >>> struts2-assembly. both mvn3 and mvn4 be very very slow there. >>> >>> >>> jetty seems has profile-plugin leak when using mvn4. >>> maven-assembly-plugin:3.7.1 wrongly invoking a bom sub-project, which is >>> only defined in a profile (named `config`) which shall not be on. >>> >>> quarkus cannot build with mvn4, maven-invoker-plugin failed >>> >>> Xeno Amess <xenoam...@gmail.com> 于2025年5月29日周四 14:52写道： >>> >>>> nope. neither can the github master latest. >>>> >>>> Xeno Amess <xenoam...@gmail.com> 于2025年5月29日周四 14:40写道： >>>> >>>>> I used apache-maven-4.0.0-rc-3 btw. I would have a try at the latest >>>>> github code version... >>>>> >>>>> Xeno Amess <xenoam...@gmail.com> 于2025年5月29日周四 14:39写道： >>>>> >>>>>> jetty is a good example. >>>>>> mvn3 can build but mvn4 cannot. >>>>>> >>>>>> maven-assembly-plugin:3.7.1 wrongly invoking a bom sub-project >>>>>> >>>>>> [INFO] ---------------< org.eclipse.jetty.ee10:jetty-ee10-bom >>>>>>> ---------------- >>>>>> [INFO] Building EE10 :: BOM 12.0.22-SNAPSHOT >>>>>> [90/354] >>>>>> [INFO] from jetty-ee10/jetty-ee10-bom/pom.xml >>>>>> [INFO] --------------------------------[ pom >>>>>> ]--------------------------------- >>>>>> [INFO] Going to calculate checksum for project >>>>>> [groupId=org.eclipse.jetty.ee10, artifactId=jetty-ee10-bom, >>>>>> version=12.0.22-SNAPSHOT] >>>>>> [INFO] Project inputs calculated in 0 ms. XXMM checksum >>>>>> [7b7b6120251dfd1d] calculated in 0 ms. >>>>>> [INFO] Attempting to restore project >>>>>> org.eclipse.jetty.ee10:jetty-ee10-bom from build cache >>>>>> [INFO] Local build found by checksum 7b7b6120251dfd1d >>>>>> [INFO] Found cached build, restoring >>>>>> org.eclipse.jetty.ee10:jetty-ee10-bom from cache by checksum >>>>>> 7b7b6120251dfd1d >>>>>> [WARNING] Cached build doesn't contains all requested plugin >> executions >>>>>> (missing: >>>>>> [org.apache.maven.plugins:maven-resources-plugin:3.3.1:copy-resources >>>>>> {execution: copy-resources}, >>>>>> org.apache.maven.plugins:maven-assembly-plugin:3.7.1:single >> {execution: >>>>>> config-assembly}]), cannot restore >>>>>> [INFO] Local build was not found by checksum 7b7b6120251dfd1d for >>>>>> org.eclipse.jetty.ee10:jetty-ee10-bom >>>>>> [INFO] >>>>>> [INFO] --- spotless:2.44.4:check (default) @ jetty-ee10-bom --- >>>>>> [INFO] Index file does not exist. Fallback to an empty index >>>>>> [INFO] Sorting file /tmp/pom11239457262347372795.xml >>>>>> [INFO] Pom file is already sorted, exiting >>>>>> [INFO] Spotless.Pom is keeping 1 files clean - 0 needs changes to be >>>>>> clean, 1 were already clean, 0 were skipped because caching >> determined they >>>>>> were already clean >>>>>> [INFO] >>>>>> [INFO] --- checkstyle:3.6.0:check (checkstyle-check) @ jetty-ee10-bom >>>>>> --- >>>>>> [INFO] 开始检查…… >>>>>> 检查完成。 >>>>>> [INFO] You have 0 Checkstyle violations. >>>>>> [INFO] >>>>>> [INFO] --- enforcer:3.5.0:enforce (ban-javax-servlet-api) @ >>>>>> jetty-ee10-bom --- >>>>>> [INFO] Rule 0: >>>>>> org.apache.maven.enforcer.rules.dependency.BannedDependencies passed >>>>>> [INFO] >>>>>> [INFO] --- enforcer:3.5.0:enforce (enforce-java) @ jetty-ee10-bom --- >>>>>> [INFO] Rule 3: >>>>>> >> org.eclipse.jetty.toolchain.enforcer.rules.RequireOsgiCompatibleVersionRule(versionOsgiRule) >>>>>> passed >>>>>> [INFO] Rule 4: >>>>>> org.apache.maven.enforcer.rules.dependency.RequireUpperBoundDeps >> passed >>>>>> [INFO] >>>>>> [INFO] --- build-helper:3.6.0:parse-version (set-osgi-version) @ >>>>>> jetty-ee10-bom --- >>>>>> [INFO] >>>>>> [INFO] --- jacoco:0.8.13:prepare-agent (jacoco-initialize) @ >>>>>> jetty-ee10-bom --- >>>>>> [INFO] argLine set to >>>>>> >> -javaagent:/home/xenoamess/.m2/repository/org/jacoco/org.jacoco.agent/0.8.13/org.jacoco.agent-0.8.13-runtime.jar=destfile=/home/xenoamess/workspace/jetty.project/jetty-ee10/jetty-ee10-bom/target/jacoco.exec,excludes=**/org/eclipse/jetty/ant/**:*/org/eclipse/jetty/maven/its/**:**/org/eclipse/jetty/embedded/**:**/org/eclipse/jetty/asyncrest/**:**/org/eclipse/jetty/demo/**:**/org/eclipse/jetty/gcloud/**:**/org/eclipse/jetty/infinispan/**:**/org/eclipse/jetty/osgi/**:**/org/eclipse/jetty/spring/**:**/org/eclipse/jetty/http/spi/**:**/org/eclipse/jetty/tests/**:**/org/eclipse/jetty/test/** >>>>>> [INFO] >>>>>> [INFO] --- jacoco:0.8.13:prepare-agent (jacoco-setup-m-invoker-p) @ >>>>>> jetty-ee10-bom --- >>>>>> [INFO] invoker.mavenOpts set to >>>>>> >> -javaagent:/home/xenoamess/.m2/repository/org/jacoco/org.jacoco.agent/0.8.13/org.jacoco.agent-0.8.13-runtime.jar=destfile=/home/xenoamess/workspace/jetty.project/jetty-ee10/jetty-ee10-bom/target/jacoco.exec,excludes=**/org/eclipse/jetty/ant/**:*/org/eclipse/jetty/maven/its/**:**/org/eclipse/jetty/embedded/**:**/org/eclipse/jetty/asyncrest/**:**/org/eclipse/jetty/demo/**:**/org/eclipse/jetty/gcloud/**:**/org/eclipse/jetty/infinispan/**:**/org/eclipse/jetty/osgi/**:**/org/eclipse/jetty/spring/**:**/org/eclipse/jetty/http/spi/**:**/org/eclipse/jetty/tests/**:**/org/eclipse/jetty/test/** >>>>>> [INFO] >>>>>> [INFO] --- remote-resources:3.2.0:process (copy-shared-resources) @ >>>>>> jetty-ee10-bom --- >>>>>> [INFO] Preparing remote bundle >>>>>> org.eclipse.jetty:build-resources:12.0.22-SNAPSHOT >>>>>> [INFO] Copying 0 resource from 1 bundle. >>>>>> [INFO] >>>>>> [INFO] --- jetty-version:2.7:attach-version-text (attach-version) @ >>>>>> jetty-ee10-bom --- >>>>>> [INFO] >>>>>> [INFO] --- resources:3.3.1:copy-resources (copy-resources) @ >>>>>> jetty-ee10-bom --- >>>>>> [INFO] skip non existing resourceDirectory >>>>>> >> /home/xenoamess/workspace/jetty.project/jetty-ee10/jetty-ee10-bom/src/main/config >>>>>> [INFO] >>>>>> [INFO] --- flatten:1.7.0:flatten (flatten) @ jetty-ee10-bom --- >>>>>> [INFO] Generating flattened POM of project >>>>>> org.eclipse.jetty.ee10:jetty-ee10-bom:pom:12.0.22-SNAPSHOT... >>>>>> [INFO] >>>>>> [INFO] --- bundle:5.1.9:manifest (default) @ jetty-ee10-bom --- >>>>>> [WARNING] Ignoring project type pom - supportedProjectTypes = [jar, >>>>>> maven-plugin] >>>>>> [INFO] >>>>>> [INFO] --- source:3.3.1:jar-no-fork (attach-sources) @ jetty-ee10-bom >>>>>> --- >>>>>> [INFO] >>>>>> [INFO] --- jacoco:0.8.13:report (jacoco-site) @ jetty-ee10-bom --- >>>>>> [INFO] Skipping JaCoCo execution due to missing execution data file. >>>>>> [INFO] >>>>>> [INFO] --- assembly:3.7.1:single (config-assembly) @ jetty-ee10-bom >> --- >>>>>> >>>>>> >>>>>> [INFO] Mimir session closed (RETRIEVED=0 CACHED=0) >>>>>> [ERROR] Failed to execute goal >>>>>> org.apache.maven.plugins:maven-assembly-plugin:3.7.1:single >>>>>> (config-assembly) on project jetty-ee10-bom: Failed to create >> assembly: >>>>>> Error creating assembly archive config: archive cannot be empty -> >> [Help 1] >>>>>> [ERROR] >>>>>> >>>>>> >>>>>> checked it did not invoke in mvn3 building log... >>>>>> >>>>>> Sergey Chernov <serega.mo...@gmail.com> 于2025年5月29日周四 00:17写道： >>>>>> >>>>>>> Hey, Delany. I will look into this. >>>>>>> Also to avoid confusion let's continue this discussion in the proper >>>>>>> thread >>>>>>> as it's not related to maven 4 release >>>>>>> >>>>>>> On Wed, May 28, 2025 at 4:22 PM Delany <delany.middle...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Sergei, >>>>>>>> >>>>>>>> I'm running with this extension going forward. I don't use the >> build >>>>>>> cache >>>>>>>> (yet) so that's not an issue. >>>>>>>> When trying to promote unit tests to a company that's resistant its >>>>>>> helpful >>>>>>>> to be able to say the tests won't slow down the build. >>>>>>>> >>>>>>>> We have one huge module though (project we're calling it now) with >>>>>>>> extensive tests, and its a bottleneck for all that follow. >>>>>>>> Typically we had to skip tests, but now we don't have to. Build >> time >>>>>>> was >>>>>>>> cut in half. >>>>>>>> >>>>>>>> I did notice jacoco coverage was disabled as I bound the >>>>>>>> jacoco-maven-plugin report goal to the prepare-package phase. >>>>>>>> As the documentation says, the phases have been switched around. >>>>>>>> But can you make it work with the >> buildplan-maven-plugin:list-phases >>>>>>> goal >>>>>>>> so I can actually see which phases runs when, or is that up to this >>>>>>> plugin >>>>>>>> to figure out? >>>>>>>> >>>>>>>> Kind regards, >>>>>>>> Delany >>>>>>>> >>>>>>>> >>>>>>>> On Wed, 28 May 2025 at 14:55, Sergey Chernov < >> serega.mo...@gmail.com >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Guillaume, that's awesome. It's good that there is no rush with >> it. >>>>>>>>> Once I'll have insights, I'll share them. >>>>>>>>> >>>>>>>>> Xeno, the project I'm checking is private unfortunately. Also >> this >>>>>>> can be >>>>>>>>> quite challenging to find a good Maven multi-module project as a >>>>>>>> reference >>>>>>>>> for performance testing and compatibility validations, as many of >>>>>>> big >>>>>>>>> projects e.g. have flaky tests and other issues. >>>>>>>>> Some of the biggest that I've seen: >>>>>>>>> * https://github.com/quarkusio/quarkus (has lots of >>>>>>> customizations in >>>>>>>>> .mvn/extensions.xml) >>>>>>>>> * https://github.com/apache/dubbo >>>>>>>>> * https://github.com/apache/hadoop >>>>>>>>> * https://github.com/apache/struts >>>>>>>>> * https://github.com/vaadin/flow and >>>>>>> https://github.com/vaadin/framework >>>>>>>>> >>>>>>>>> On Wed, May 28, 2025 at 2:16 PM Guillaume Nodet < >> gno...@apache.org >>>>>>>> wrote: >>>>>>>>>> After considering the concerns about API stability, plugin >>>>>>>>>> compatibility, and the performance issues reported by Sergey >> (4x >>>>>>>>>> slower in rc3), I agree with Sylwester, Benjamin, and Elliotte >>>>>>> that we >>>>>>>>>> should opt for an rc4 to address these issues. My proposal is >> to >>>>>>>>>> release a final release candidate (rc4) soon, aiming for a >>>>>>> General >>>>>>>>>> Availability (GA) release in September. Post rc4, I suggest >>>>>>> creating a >>>>>>>>>> 4.0.x branch to start work on 4.1.0. For 4.1.0, we could >>>>>>> introduce API >>>>>>>>>> changes like records and sequenced collections, as well as new >>>>>>>>>> features such as mixins, cascading profiles, server aliases, >> JDK >>>>>>> 21 >>>>>>>>>> (still under discussion) and more.... >>>>>>>>>> >>>>>>>>>> The recent API changes for JPMS, as Martin noted, only affect >> the >>>>>>>>>> experimental API, so they shouldn’t block the release. On >>>>>>> Matthias’s >>>>>>>>>> suggestion, enabling GitHub Issues across all repositories >>>>>>> before the >>>>>>>>>> GA seems like a good idea to improve user feedback. Let’s focus >>>>>>> on >>>>>>>>>> stabilizing rc4 and addressing performance bottlenecks, >> possibly >>>>>>>>>> exploring Martin’s O(N²) concerns with profiling, as Sergey >>>>>>> offered. >>>>>>>>>> What do you think about this plan? >>>>>>>>>> >>>>>>>>>> On a side note, I created a few months ago a project to check >>>>>>> Maven 4 >>>>>>>>>> compatibility with all Apache projects using Maven. The >> results >>>>>>> for >>>>>>>>>> RC-3 can be found at >>>>>>>>>> https://github.com/gnodet/maven4-testing/issues/2812. Most of >>>>>>> the >>>>>>>>>> errors are caused by the use of incompatible plugins (which >>>>>>> needs to >>>>>>>>>> be updated) or incorrect POMs (as Maven 4 is more strict about >> a >>>>>>> few >>>>>>>>>> things). What would be nice is a tool to integrate in the >> shell >>>>>>> that >>>>>>>>>> would fix those known incompatibilities, so that users would >>>>>>> have an >>>>>>>>>> easier migration path. And we could apply it to this project >> to >>>>>>> make >>>>>>>>>> more green boxes. >>>>>>>>>> >>>>>>>>>> Le mer. 21 mai 2025 à 08:11, Guillaume Nodet < >> gno...@apache.org> >>>>>>> a >>>>>>>>> écrit : >>>>>>>>>>> Hey Maven Devs, >>>>>>>>>>> >>>>>>>>>>> We're gearing up to release a new version from the master >>>>>>> branch. I'm >>>>>>>>>> thinking we should go for 4.0.0 instead of rc-4. What do you >> all >>>>>>> think? >>>>>>>>> Any >>>>>>>>>> feedback or ideas on the versioning or release plan? Let’s hear >>>>>>> it! >>>>>>>>>>> Cheers, >>>>>>>>>>> >>>>>>>>>>> Guillaume >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> ------------------------ >>>>>>>>>> Guillaume Nodet >>>>>>>>>> >>>>>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>>>>>>>>> For additional commands, e-mail: dev-h...@maven.apache.org >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
