ibabiankou opened a new pull request #142: URL: https://github.com/apache/maven-resolver/pull/142
Updates `DefaultDependencyCollector` to process descriptors concurrently. Updated implementation does breadth-first traversal of the dependency graph and processes each dependency in a separate thread. Order of the children nodes then restored to match order in descriptor of parent. By default uses 5 threads, which can be changed using either `maven.descriptor.threads` or `maven.artifact.threads` property. All the unit and integration tests are green. The change was also tested locally on a reasonable codebase (~1100 modules with lots dependencies) with 24 modules processed concurrently (-T1C). **Important note**: a few times out of dosens of tests I observed an exception that seems to be related to the change (stack trace attached), however I couldn't really piece it together how it's possible, so ideas are welcome. <details><summary>Stack Trace</summary> <p> ```java [ERROR] Failed to execute goal on project project-name: Could not resolve dependencies for project group-id:project-name:jar:2.16.1-SNAPSHOT: Failed to collect dependencies at dep-group-id:dep-art-id:jar:v2.9.0: Failed to read artifact descriptor for dep-group-id:dep-art-id:jar:v2.9.0: Could not transfer artifact dep-group-id:dep-art-id:pom:v2.9.0 from/to repo-name (https://private.nexus.url.com/repository/maven-public): /path/to/local_repo/v2.9.0/dep-art-id-v2.9.0.pom.part (No such file or directory) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project project-name: Could not resolve dependencies for project group-id:project-name:jar:2.16.1-SNAPSHOT: Failed to collect dependencies at dep-group-id:dep-art-id:jar:v2.9.0 at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:269) at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147) at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196) at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186) at java.util.concurrent.FutureTask.run (FutureTask.java:264) at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539) at java.util.concurrent.FutureTask.run (FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635) at java.lang.Thread.run (Thread.java:833) Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project group-id:project-name:jar:2.16.1-SNAPSHOT: Failed to collect dependencies at dep-group-id:dep-art-id:jar:v2.9.0 at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:179) at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243) at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147) at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196) at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186) at java.util.concurrent.FutureTask.run (FutureTask.java:264) at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539) at java.util.concurrent.FutureTask.run (FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635) at java.lang.Thread.run (Thread.java:833) Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at dep-group-id:dep-art-id:jar:v2.9.0 at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.collectDependencies (DefaultDependencyCollector.java:299) at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies (DefaultRepositorySystem.java:284) at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:170) at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243) at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147) at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196) at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186) at java.util.concurrent.FutureTask.run (FutureTask.java:264) at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539) at java.util.concurrent.FutureTask.run (FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635) at java.lang.Thread.run (Thread.java:833) Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for dep-group-id:dep-art-id:jar:v2.9.0 at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom (DefaultArtifactDescriptorReader.java:259) at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor (DefaultArtifactDescriptorReader.java:175) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.resolveCachedArtifactDescriptor (DefaultDependencyCollector.java:617) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.getArtifactDescriptorResult (DefaultDependencyCollector.java:602) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency (DefaultDependencyCollector.java:491) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.access$000 (DefaultDependencyCollector.java:87) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector$1.run (DefaultDependencyCollector.java:395) at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539) at java.util.concurrent.FutureTask.run (FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635) at java.lang.Thread.run (Thread.java:833) Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact dep-group-id:dep-art-id:pom:v2.9.0 from/to repo-name (https://private.nexus.url.com/repository/maven-public): /path/to/local_repo/v2.9.0/dep-art-id-v2.9.0.pom.part (No such file or directory) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:425) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:229) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact (DefaultArtifactResolver.java:207) at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom (DefaultArtifactDescriptorReader.java:244) at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor (DefaultArtifactDescriptorReader.java:175) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.resolveCachedArtifactDescriptor (DefaultDependencyCollector.java:617) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.getArtifactDescriptorResult (DefaultDependencyCollector.java:602) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency (DefaultDependencyCollector.java:491) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.access$000 (DefaultDependencyCollector.java:87) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector$1.run (DefaultDependencyCollector.java:395) at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539) at java.util.concurrent.FutureTask.run (FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635) at java.lang.Thread.run (Thread.java:833) Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact dep-group-id:dep-art-id:pom:v2.9.0 from/to repo-name (https://private.nexus.url.com/repository/maven-public): /path/to/local_repo/v2.9.0/dep-art-id-v2.9.0.pom.part (No such file or directory) at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed (ArtifactTransportListener.java:52) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:369) at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run (RunnableErrorForwarder.java:75) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute (BasicRepositoryConnector.java:628) at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get (BasicRepositoryConnector.java:262) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads (DefaultArtifactResolver.java:514) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:402) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:229) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact (DefaultArtifactResolver.java:207) at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom (DefaultArtifactDescriptorReader.java:244) at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor (DefaultArtifactDescriptorReader.java:175) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.resolveCachedArtifactDescriptor (DefaultDependencyCollector.java:617) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.getArtifactDescriptorResult (DefaultDependencyCollector.java:602) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency (DefaultDependencyCollector.java:491) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.access$000 (DefaultDependencyCollector.java:87) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector$1.run (DefaultDependencyCollector.java:395) at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539) at java.util.concurrent.FutureTask.run (FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635) at java.lang.Thread.run (Thread.java:833) Caused by: java.io.FileNotFoundException: /path/to/local_repo/v2.9.0/dep-art-id-v2.9.0.pom.part (No such file or directory) at java.io.FileInputStream.open0 (Native Method) at java.io.FileInputStream.open (FileInputStream.java:216) at java.io.FileInputStream.<init> (FileInputStream.java:157) at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy (DefaultFileProcessor.java:163) at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy (DefaultFileProcessor.java:151) at org.eclipse.aether.internal.impl.DefaultFileProcessor.move (DefaultFileProcessor.java:252) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask (BasicRepositoryConnector.java:482) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:364) at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run (RunnableErrorForwarder.java:75) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute (BasicRepositoryConnector.java:628) at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get (BasicRepositoryConnector.java:262) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads (DefaultArtifactResolver.java:514) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:402) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:229) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact (DefaultArtifactResolver.java:207) at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom (DefaultArtifactDescriptorReader.java:244) at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor (DefaultArtifactDescriptorReader.java:175) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.resolveCachedArtifactDescriptor (DefaultDependencyCollector.java:617) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.getArtifactDescriptorResult (DefaultDependencyCollector.java:602) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency (DefaultDependencyCollector.java:491) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.access$000 (DefaultDependencyCollector.java:87) at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector$1.run (DefaultDependencyCollector.java:395) at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539) at java.util.concurrent.FutureTask.run (FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635) at java.lang.Thread.run (Thread.java:833) ``` </p> </details> - Reviewing individual commits might be easier. - Some unit tests will be added if there are no objections WRT overall approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
