[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693194#comment-17693194 ] wei cai commented on MRESOLVER-7: - [~knut] Thanks for your feedback! This is a closed JIRA, let's discuss about BF + Skipper + Parallel downloading in https://issues.apache.org/jira/browse/MRESOLVER-324 For your reported issue, it is related with https://issues.apache.org/jira/browse/MRESOLVER-325 > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693076#comment-17693076
]
Knut Wannheden commented on MRESOLVER-7:
Yes, the new parallel collector is working very nicely!
In one build I did see an error, which I was thinking might be related to using
the BF collector, but I haven't been able to reproduce it again. In my
settings.xml I have multiple Maven repositories configured which "overlap"
(i.e. some artifacts are contained in multiple of these repos) and I have never
been able to figure out how Maven decides from which repo to download an
artifact (yes, I know this is documented, but it doesn't appear to match my
observations). I don't know if this extra information is relevant or helpful,
but anyway, here is the error I saw. Please let me know if you want me to
create an issue.
{noformat}
[ERROR] Failed to execute goal
org.jacoco:jacoco-maven-plugin:0.8.8:prepare-agent (default-prepare-agent) on
project glu-test-contract-service-parent: Execution default-prepare-agent of
goal org.jacoco:jacoco-maven-plugin:0.8.8:prepare-agent failed: Plugin
org.jacoco:jacoco-maven-plugin:0.8.8 or one of its dependencies could not be
resolved: Failed to collect dependencies at
org.jacoco:jacoco-maven-plugin:jar:0.8.8 ->
org.apache.maven.shared:file-management:jar:1.2.1 ->
org.apache.maven:maven-plugin-api:jar:2.0.6: java.io.UncheckedIOException:
java.nio.file.AccessDeniedException:
c:\dev\.m2-nexus\repository\org\apache\maven\maven-parent\5\maven-parent-5.pom.lastUpdated.7796060141987260961.tmp
->
c:\dev\.m2-nexus\repository\org\apache\maven\maven-parent\5\maven-parent-5.pom.lastUpdated
-> [Help 1]{noformat}
> Download dependency POMs in parallel in BF collector
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamas Cservenak
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17692969#comment-17692969 ] wei cai commented on MRESOLVER-7: - [~jhaber] Thanks for your feedback! Really glad it helps your Company and Team. [~Chealer] The new changes all go to BF by default as the resolver is very critical to the project build & deploy result. And as discussed earlier before with Maven boards, the plan is to making BF as a default option in future versions based on more community feedback. Please try the BF option and share your feedback to: https://issues.apache.org/jira/browse/MRESOLVER-324 > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17692967#comment-17692967 ] Philippe Cloutier commented on MRESOLVER-7: --- Thank you [~wecai] Considering the change in this ticket's scope, I suggest those who voted for this ticket hoping to see a change in default behavior to also vote for MNG-5896. > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17692962#comment-17692962 ] Jonathan Haber commented on MRESOLVER-7: We started rolling out Maven 3.9.0 with parallel downloads to our engineering team, and the positive response has been overwhelming. I included a picture below of some of the Slack reactions the announcement received in the first few hours. Just wanted to give a big thanks to [~wecai] and everyone else who worked on this !https://i.imgur.com/mkGNFmY.png! > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17632341#comment-17632341
]
Björn Michael commented on MRESOLVER-7:
---
Is it possible to set it in {{pom.xml}} as well? I'd like to put it into a
parent POM.
{code:xml}
bf
{code}
> Download dependency POMs in parallel in BF collector
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamas Cservenak
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626936#comment-17626936 ] wei cai commented on MRESOLVER-7: - [~Chealer] You may find more details in https://issues.apache.org/jira/browse/MRESOLVER-247 about the BF + Skipper. > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626603#comment-17626603 ] Philippe Cloutier commented on MRESOLVER-7: --- Thank you very much [~wecai] for these details. I would appreciate a clarification of what "Skipper to skip unnecessary resolution" means. > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626429#comment-17626429 ] wei cai commented on MRESOLVER-7: - BF collector is a combination of [https://github.com/apache/maven-resolver/pull/158] [https://github.com/apache/maven-resolver/pull/178] It includes some performance improvements based on what we've done in our company to speed up maven build. So far the BF and original DF algorithm co-exists and the DF is still the default. When maven 3.9.x ships with maven-resolver 1.9.0, then users can simply put a [.mvn/maven.config|https://maven.apache.org/configure.html] within project root with below content: "-Daether.dependencyCollector.impl=bf" to enable these features: * Skipper to skip unnecessary resolution * Parallel descriptor files (pom, metadata.xml) downloading Not sure what's the plan to make the BF as default one. All related Jiras about parallel pom downloading can be closed accordingly until BF becomes the default option. > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17625917#comment-17625917 ] Philippe Cloutier commented on MRESOLVER-7: --- Thank you very much for your work as well as for this valuable explanation [~cstamas] I know nothing about collectors and the model builder nuance, but if I understand the situation correctly, I recommend to: * Restore this ticket's original title and IN PROGRESS status * If a ticket specifically about the BF collector is needed or useful: ** create one or clone this one ** set it as a dependency of this one ** mark it as resolved * If a ticket specifically about the DF collector is needed or useful: ** create one or clone this one ** set it as a dependency of this one * Ensure that the assignee of each ticket is representative This may have been overdue, but seeing how much work was already needed, congratulations for what was already accomplished. > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17624230#comment-17624230 ] Hudson commented on MRESOLVER-7: Build succeeded in Jenkins: Maven » Maven TLP » maven-resolver » master #68 See https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven-resolver/job/master/68/ > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17624229#comment-17624229
]
ASF GitHub Bot commented on MRESOLVER-7:
cstamas commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r1005285209
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +466,89 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
+
+/**
+ * Artifact ID -> Future of DescriptorResolutionResult
+ */
+final Map> results = new
ConcurrentHashMap<>( 256 );
+final Logger logger = LoggerFactory.getLogger( getClass() );
+
+ParallelDescriptorResolver( RepositorySystemSession session )
+{
+this.executorService = getExecutorService( session );
+}
+
+void resolveDescriptors( Artifact artifact,
Callable callable )
+{
+results.computeIfAbsent( ArtifactIdUtils.toId( artifact ),
+key -> this.executorService.submit( callable ) );
+}
+
+void cacheVersionRangeDescriptor( Artifact artifact,
DescriptorResolutionResult resolutionResult )
+{
+results.computeIfAbsent( ArtifactIdUtils.toId( artifact ),
+key -> ConcurrentUtils.constantFuture( resolutionResult )
);
+}
+
+Future find( Artifact artifact )
+{
+return results.get( ArtifactIdUtils.toId( artifact ) );
+}
+
+void shutdown()
+{
+executorService.shutdown();
+}
+
+private ExecutorService getExecutorService( RepositorySystemSession
session )
+{
+int nThreads = ConfigUtils.getInteger( session, 5,
"maven.artifact.threads" );
Review Comment:
done in 7f56672be3ca032d145bdd2e8e89231dea68226e
> Download dependency POMs in parallel in BF collector
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamas Cservenak
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17624222#comment-17624222 ] Hudson commented on MRESOLVER-7: Build succeeded in Jenkins: Maven » Maven TLP » maven-resolver » master #67 See https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven-resolver/job/master/67/ > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17624221#comment-17624221 ] ASF GitHub Bot commented on MRESOLVER-7: cstamas merged PR #178: URL: https://github.com/apache/maven-resolver/pull/178 > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17624217#comment-17624217
]
ASF GitHub Bot commented on MRESOLVER-7:
cstamas commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r1005272778
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +466,89 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
+
+/**
+ * Artifact ID -> Future of DescriptorResolutionResult
+ */
+final Map> results = new
ConcurrentHashMap<>( 256 );
+final Logger logger = LoggerFactory.getLogger( getClass() );
+
+ParallelDescriptorResolver( RepositorySystemSession session )
+{
+this.executorService = getExecutorService( session );
+}
+
+void resolveDescriptors( Artifact artifact,
Callable callable )
+{
+results.computeIfAbsent( ArtifactIdUtils.toId( artifact ),
+key -> this.executorService.submit( callable ) );
+}
+
+void cacheVersionRangeDescriptor( Artifact artifact,
DescriptorResolutionResult resolutionResult )
+{
+results.computeIfAbsent( ArtifactIdUtils.toId( artifact ),
+key -> ConcurrentUtils.constantFuture( resolutionResult )
);
+}
+
+Future find( Artifact artifact )
+{
+return results.get( ArtifactIdUtils.toId( artifact ) );
+}
+
+void shutdown()
+{
+executorService.shutdown();
+}
+
+private ExecutorService getExecutorService( RepositorySystemSession
session )
+{
+int nThreads = ConfigUtils.getInteger( session, 5,
"maven.artifact.threads" );
Review Comment:
One more thing, but I will do this post merge: if you look at
BasicRepositoryConnector, it uses CONFIG_PROP_THREADS and just as "secondary"
property uses this "maven.artifact.threads" as a fallback. I will update this
post merge, to make them behave siimilarly; have it's own prop but fall back to
"maven.artifact.threas" as well
> Download dependency POMs in parallel in BF collector
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamas Cservenak
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17620176#comment-17620176 ] ASF GitHub Bot commented on MRESOLVER-7: cstamas commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1283766597 @michael-o can you please specify what changes are requested with your "change request" PR review? It is unclear to me. As for me, this PR is good to merge (and probably improve it later re shared executor, but is def out of scope for this PR) > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17617104#comment-17617104 ] ASF GitHub Bot commented on MRESOLVER-7: caiwei-ebay2 commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1277725713 @michael-o How's everything? Got a chance to give more comments to this PR? And may I know what's the remaining steps to get this PR merged? > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17616505#comment-17616505 ] Tamas Cservenak commented on MRESOLVER-7: - Also, have to remark that in case of collection "POM downloads" happens in two distinct cases (but interleaved during collection): * a) when downloading the dependency POM (for example D depends on D1, D2, D3, then corresponding D1.pom, D2.pom, D3.pom will be downloaded) – these can and are downloaded in parallel with provided PR for BF collector. * b) but, when D1.pom is being built (in model builder), and IF D1.pom has D1-parent.pom, amd parent has D1-parent-parent.pom (and so on, but similar for POM imports, etc), these are still sequentially downloaded, as model bullder processes/builds the D1.pom. Model builder was not altered by PR. The PR implements case a) in BF collector only. DF collector and model builder is unchanged. > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17616484#comment-17616484 ] Tamas Cservenak commented on MRESOLVER-7: - [~Chealer] the requirements to make this happen requires several changes, and one of them was done as part of breadth-first collecting of dependencies (BF) while so far "original" collector and default as well is depth-first (DF). The provided PR enables the new BF collector to download POMs in parallel, while the "original" DF is still unchanged... But that said, I think there may be a slight mixup here: the PR was done as part of this issue (while it modified BF collector only, something that when this issue was created did not even exist yet), so to say, this issue got "hijacked"? > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel in BF collector
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17616469#comment-17616469 ] Philippe Cloutier commented on MRESOLVER-7: --- [~cstamas] : Does this specifically affect the BF collector as opposed to other collectors? Is there a workaround? > Download dependency POMs in parallel in BF collector > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Tamas Cservenak >Priority: Major > Fix For: 1.9.0 > > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613181#comment-17613181
]
ASF GitHub Bot commented on MRESOLVER-7:
caiwei-ebay commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r985689590
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
Totally agreed. Could refactor the executor part in a separate PR so all of
the download related logic could share.
Another issue you've pointed out is fixed. Thanks for the review. Please let
me know if anything else I can help.
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamás Cservenák
>Priority: Major
> Fix For: resolver-next
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611834#comment-17611834
]
ASF GitHub Bot commented on MRESOLVER-7:
cstamas commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r985065751
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
For now I'd follow very same pattern as in connector, it creates executor
for each remote repos (not good, but not bad either). So creating executor per
resolve invocation is also ok. Later, we could do something around this, but it
requires larger scale changes, so leave it for now IMHO
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamás Cservenák
>Priority: Major
> Fix For: resolver-next
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611807#comment-17611807
]
ASF GitHub Bot commented on MRESOLVER-7:
caiwei-ebay commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r985027783
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
@cstamas Any suggestions about whether we should use separate executors for
the Maven parallel build?
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamás Cservenák
>Priority: Major
> Fix For: resolver-next
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611050#comment-17611050
]
ASF GitHub Bot commented on MRESOLVER-7:
caiwei-ebay commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r983545613
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +466,90 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
+
+/**
+ * Artifact ID -> Future of DescriptorResolutionResult
+ */
+final Map> results = new
ConcurrentHashMap<>( 256 );
+final Logger logger = LoggerFactory.getLogger( getClass() );
+
+ParallelDescriptorResolver( RepositorySystemSession session )
+{
+this.executorService = getExecutorService( session );
+}
+
+Future resolveDescriptors( Artifact
artifact,
Review Comment:
Let me fix.
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamás Cservenák
>Priority: Major
> Fix For: resolver-next
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611042#comment-17611042
]
ASF GitHub Bot commented on MRESOLVER-7:
caiwei-ebay commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r983554260
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
For a multiple module project, collect method will be called when building
each module, and when run in parallel, collect method will be called by
multiple threads. If we consider parallel build case, creates multiple
executors seems more reasonable.
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamás Cservenák
>Priority: Major
> Fix For: resolver-next
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611041#comment-17611041
]
ASF GitHub Bot commented on MRESOLVER-7:
caiwei-ebay commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r983545613
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +466,90 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
+
+/**
+ * Artifact ID -> Future of DescriptorResolutionResult
+ */
+final Map> results = new
ConcurrentHashMap<>( 256 );
+final Logger logger = LoggerFactory.getLogger( getClass() );
+
+ParallelDescriptorResolver( RepositorySystemSession session )
+{
+this.executorService = getExecutorService( session );
+}
+
+Future resolveDescriptors( Artifact
artifact,
Review Comment:
It is asnyc, in this way, multiple poms are downloaded in parallel.
cacheVersionRangeDescriptor is actually caching a constantFuture.
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamás Cservenák
>Priority: Major
> Fix For: resolver-next
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611022#comment-17611022
]
ASF GitHub Bot commented on MRESOLVER-7:
cstamas commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r983504722
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
the executor exists in connector, this is impl module, connector does not
depend on impl nor impl depend on connector, see
https://maven.apache.org/resolver/index.html
Now, not only that, but also it is quite possible that execution does not
"even get to" connector (ie. all is present locally), so this would require
major undertake, probably to move executor providing component to SPI (as util
is not quite the place for this), then change both uses (so BF collector and
connector), but in connector module we'd need to provide some "stub"
implementation (as again, it does not depend on impl), and finally that would
become one whole. But even writing this down is long, and seems totally
unrelated to this PR.
Am currently totally not concerned about sharing executor, but more that
executor is created per invocation of collect merhd...
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamás Cservenák
>Priority: Major
> Fix For: resolver-next
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611021#comment-17611021
]
ASF GitHub Bot commented on MRESOLVER-7:
cstamas commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r983504722
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
the executor exists in connector, this is impl module, connector does not
depend on impl nor impl depend on connector, see
https://maven.apache.org/resolver/index.html
Now, not only that, but also it is quite possible that execution does not
"even get to" connector (ie. all is present locally), so this would require
major undertake, probably to move executor providing component to SPI (as util
is not quite the place for this), then change both uses (so BF collector and
connector), but in connector module we'd need to provide some "stub"
implementation (as again, it does not depend on impl), and finally that would
become one whole.
Am currently totally not concerned about sharing executor, but more that
executor is created per invocation of collect merhd...
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamás Cservenák
>Priority: Major
> Fix For: resolver-next
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611007#comment-17611007
]
ASF GitHub Bot commented on MRESOLVER-7:
michael-o commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r983485610
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
I don't understand why we need another executor again. We have an executor
for artifacts, POMs are artifacts.
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamás Cservenák
>Priority: Major
> Fix For: resolver-next
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17610993#comment-17610993
]
ASF GitHub Bot commented on MRESOLVER-7:
cstamas commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r983453759
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
Own executor is just fine, we should not raise complexity but, as I see
executor is created per invocation, and this may be problematic (parallel
build?), so instead, why not cache it during lifespan of session? That would
make one executor being created and shared as long as session exists, no?
But then question comes, when to shut it down? Like, the shutdown method
could perform some "ref counting"? And the last leaving the building turn off
lights?
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamás Cservenák
>Priority: Major
> Fix For: resolver-next
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17610990#comment-17610990
]
ASF GitHub Bot commented on MRESOLVER-7:
cstamas commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r983449326
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +466,90 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
+
+/**
+ * Artifact ID -> Future of DescriptorResolutionResult
+ */
+final Map> results = new
ConcurrentHashMap<>( 256 );
+final Logger logger = LoggerFactory.getLogger( getClass() );
+
+ParallelDescriptorResolver( RepositorySystemSession session )
+{
+this.executorService = getExecutorService( session );
+}
+
+Future resolveDescriptors( Artifact
artifact,
Review Comment:
why not make this method void just like method below? (the
cacheVersionRangeDescriptor)
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Tamás Cservenák
>Priority: Major
> Fix For: resolver-next
>
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17610551#comment-17610551 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1260843029 For me, this won't happen before October. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17610547#comment-17610547 ] ASF GitHub Bot commented on MRESOLVER-7: cstamas commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1260837215 @caiwei-ebay will get to it soon, sorry for delay > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17610374#comment-17610374 ] ASF GitHub Bot commented on MRESOLVER-7: caiwei-ebay commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1260460287 > > > Proper evaluation I'd mandatory > > > > > > I understand and appreciate all the work you guys have done. If I remember clearly, some integration tests and details were discussed in a private ASF slack channel which I had no access... > > Just want to know if there's any ETA (already one month since last update)? Anyway, please let me know if there's anything I can help (if that would help to proceed this). > > @cstamas we need to discuss this sometime this month. @michael-o @cstamas May I know the latest status of discussion? Please never hesitate to let me know if anything I can help or the code I should revise. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600315#comment-17600315 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1236813296 > > Proper evaluation I'd mandatory > > I understand and appreciate all the work you guys have done. If I remember clearly, some integration tests and details were discussed in a private ASF slack channel which I had no access... > > Just want to know if there's any ETA (already one month since last update)? Anyway, please let me know if there's anything I can help (if that would help to proceed this). @cstamas we need to discuss this sometime this month. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600212#comment-17600212 ] ASF GitHub Bot commented on MRESOLVER-7: Eskibear commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1236566854 > Proper evaluation I'd mandatory I understand and appreciate all the work you guys have done. If I remember clearly, some integration tests and details were discussed in a private ASF slack channel which I had no access... Just want to know if there's any ETA (already one month since last update)? Anyway, please let me know if there's anything I can help (if that would help to proceed this). > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577267#comment-17577267
]
ASF GitHub Bot commented on MRESOLVER-7:
michael-o commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r941055530
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
Will check
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Priority: Major
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577265#comment-17577265
]
ASF GitHub Bot commented on MRESOLVER-7:
caiwei-ebay commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r941047977
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
Sorry for the late reply. Busy working these days.
Uploaded a diagram to explain how the dependency resolution works.
In a Maven build:
- DependencyCollector focus on resolve dependency graph by downloading and
parsing metadata.xml and pom.xmls which is running in sequence as of now.
- MojoExecutor calls ensureDependencyResolved which download jars in
parallel based on the dependency graph calculated by DependencyCollector.
Using the former (fully unitize get ) in DependencyCollector
seems not possible, I think we need to use a separate thread pool for
dependency resolution in DependencyCollector which includes 2 steps:
- Resolve version range (which resort to VersionRangeResolver ->
MetadataResolver -> RepositoryConnector)
- Resolve descriptor (download or pick local poms and then parse poms, which
resort to ArtifactDescriptorReader -> ArtifactResolver -> RepositoryConnector)
By running the above 2 steps in a separate thread pool can achieve better
parallelism.
Like the MetadataResolver, it is also using a separate ThreadPool to
download metadata.xml one by one from multiple repositories.
May I know any concerns that we should not use another thread pool for
dependency resolution here?
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Priority: Major
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577266#comment-17577266 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1209077396 > Any updates on this PR? Can we expect it to be merged soon? > > I'm a contributor of [JDT Language Server](https://github.com/eclipse/eclipse.jdt.ls), and one of our core component [m2e](https://github.com/eclipse-m2e/m2e-core) is depending on maven-resolver. We've been observing that downloading poms one by one takes a lot of time when importing Maven projects. To be honest, it's one of our performance bottleneck and we can hardly improve it on our own, in downstream... So it would be great if this improvement can be shipped in coming releases. Proper evaluation I'd mandatory, we already had severe regressions due to the previous approach. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577263#comment-17577263 ] ASF GitHub Bot commented on MRESOLVER-7: caiwei-ebay commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1209075785 > Any updates on this PR? Can we expect it to be merged soon? > > I'm a contributor of [JDT Language Server](https://github.com/eclipse/eclipse.jdt.ls), and one of our core component [m2e](https://github.com/eclipse-m2e/m2e-core) is depending on maven-resolver. We've been observing that downloading poms one by one takes a lot of time when importing Maven projects. To be honest, it's one of our performance bottleneck and we can hardly improve it on our own, in downstream... So it would be great if this improvement can be shipped in coming releases. Friends, the PR is under review. Feel free to participate into the contribution. :) > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolve_deps.png, resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577262#comment-17577262
]
ASF GitHub Bot commented on MRESOLVER-7:
caiwei-ebay commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r941051657
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
@michael-o @cstamas Please share your thoughts about using a separate pool
here. Or please give some design suggestions on how to make the code to meet
the merge criteria.
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Priority: Major
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577258#comment-17577258
]
ASF GitHub Bot commented on MRESOLVER-7:
caiwei-ebay commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r941047977
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
Uploaded a diagram to explain how the dependency resolution works.
In a Maven build:
- DependencyCollector focus on resolve dependency graph by downloading and
parsing metadata.xml and pom.xmls which is running in sequence as of now.
- MojoExecutor calls ensureDependencyResolved which download jars in
parallel based on the dependency graph calculated by DependencyCollector.
Using the former (fully unitize get ) in DependencyCollector
seems not possible, I think we need to use a separate thread pool for
dependency resolution in DependencyCollector which includes 2 steps:
- Resolve version range (which resort to VersionRangeResolver ->
MetadataResolver -> RepositoryConnector)
- Resolve descriptor (download or pick local poms and then parse poms, which
resort to ArtifactDescriptorReader -> ArtifactResolver -> RepositoryConnector)
By running the above 2 steps in a separate thread pool can achieve better
parallelism.
Like the MetadataResolver, it is also using a separate ThreadPool to
download metadata.xml one by one from multiple repositories.
May I know any concerns that we should not use another thread pool for
dependency resolution here?
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Priority: Major
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577100#comment-17577100 ] ASF GitHub Bot commented on MRESOLVER-7: Eskibear commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1208851614 Any updates on this PR? Can we expect it to be merged soon? I'm a contributor of [JDT Language Server](https://github.com/eclipse/eclipse.jdt.ls), and one of our core component [m2e](https://github.com/eclipse-m2e/m2e-core) is depending on maven-resolver. We've been observing that downloading poms one by one takes a lot of time when importing Maven projects. To be honest, it's one of our performance bottleneck and we can hardly improve it on our own, in downstream... So it would be great if this improvement can be shipped in coming releases. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17570927#comment-17570927
]
ASF GitHub Bot commented on MRESOLVER-7:
michael-o commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r928938065
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
I mean the former, fully utilize `get(List, List)` with all the `GetTask`,
`PeekTask`. Downloading POMs in parallel shouldn't be any different than before
sequentially.
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Priority: Major
> Attachments: resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17570922#comment-17570922
]
ASF GitHub Bot commented on MRESOLVER-7:
caiwei-ebay commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r928932146
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
@michael-o
Thanks for the advice. Agreed we should reuse the executor service.
The BfDependencyCollector calls DefaultArtifactsResolver and the latter then
calls (Basic)RepositoryConnector.
Do you mean we should make BfDependencyCollector calls RepositoryConnector's
**get(artifactDownloads, metadataDownloads)** to download poms? or simply make
the **getExecutor** method as public and then make BfDependencyCollector call
the **getExcutor** to reuse the executor service. Please help clarify.
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Priority: Major
> Attachments: resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17570924#comment-17570924
]
ASF GitHub Bot commented on MRESOLVER-7:
caiwei-ebay commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r928932519
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -334,6 +370,82 @@ private void doRecurse( Args args,
DependencyProcessingContext parentContext,
}
}
+private boolean filter( DependencyProcessingContext context )
+{
+if ( context.depSelector != null &&
!context.depSelector.selectDependency( context.dependency ) )
Review Comment:
Sure.
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Priority: Major
> Attachments: resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17570351#comment-17570351
]
ASF GitHub Bot commented on MRESOLVER-7:
michael-o commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r928152490
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -334,6 +370,82 @@ private void doRecurse( Args args,
DependencyProcessingContext parentContext,
}
}
+private boolean filter( DependencyProcessingContext context )
+{
+if ( context.depSelector != null &&
!context.depSelector.selectDependency( context.dependency ) )
Review Comment:
Why not compact to `return ( context.dep...)`?
##
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+static class ParallelDescriptorResolver
+{
+final ExecutorService executorService;
Review Comment:
Can this resolver leverage the same executor service as for artifacts with
`aether.connector.basic.threads`? At the end it does not matter what the
connector transports, no?
See: org.eclipse.aether.connector.basic.BasicRepositoryConnector
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Priority: Major
> Attachments: resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563381#comment-17563381 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1176545713 Will likey happen either this Friday or in 10 days. I think this should not be in Maven 3.9.0 just to avoid any breakings. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563262#comment-17563262 ] ASF GitHub Bot commented on MRESOLVER-7: cstamas commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1176283345 Will do around weekend, busy few days till then. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563247#comment-17563247 ] ASF GitHub Bot commented on MRESOLVER-7: caiwei-ebay commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1176251038 @michael-o @cstamas Folks, please help review the PR and let me know if anything needs from me. This PR can make poms/metadata.xml download much faster. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558026#comment-17558026 ] ASF GitHub Bot commented on MRESOLVER-7: caiwei-ebay commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1164305940 @michael-o @cstamas I have converted the PR from draft to ready. Please help review. The basic idea of this PR is: - Given dependency A's children B, C, D - Iterate B, C, D one by one, take B as example below. - Should B be filtered? If yes, won't add to the queue. Logic [here](https://github.com/apache/maven-resolver/blob/2c805b5a7979b0d53ad22d9eb6947f6613a13357/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java#L354) - If B should not filtered, then figure out the managed version of B, ex B1, resolve B1's descriptor in a thread, then add B1 to the queue. Logic [here](https://github.com/apache/maven-resolver/blob/2c805b5a7979b0d53ad22d9eb6947f6613a13357/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java#L359) In such way, poms and metadata.xml are downloaded in parallel. The issue @cstamas raised has been fixed. The failure was caused incorrect filtering (should filter with original dependency, B instead of B1 as described in above flow) Also runs the ITs with both DF and BF (with above parallel downloading) mode, same test result below: 2 tests not related with this PR failed with both mode. Not sure if this is expected. ` [ERROR] Failures: [ERROR] MavenITmng5576CdFriendlyVersions>AbstractMavenIntegrationTestCase.runTest:260->testContinuousDeliveryFriendlyVersionsAreWarningFreeWithBuildConsumer:98 [WARNING] [ERROR] MavenITmng5576CdFriendlyVersions>AbstractMavenIntegrationTestCase.runTest:260->testContinuousDeliveryFriendlyVersionsAreWarningFreeWithoutBuildConsumer:67 [WARNING] [INFO] [ERROR] Tests run: 912, Failures: 2, Errors: 0, Skipped: 0` Also dry-run 1000+ apps in our company by comparing the maven dependency tree with DF and BF mode (based on this PR), no issues found so far. Test method is like this: - mvn clean install -DskipTests -Daether.collector.impl=bf -Dmaven.repo.local=bf - mvn dependency:tree -DoutputFile=bf-tree.txt -Daether.collector.impl=bf -Dmaven.repo.local=bf - mvn clean install -DskipTests -Dmaven.repo.local=df - mvn dependency:tree -DoutputFile=df-tree.txt -Dmaven.repo.local=df - compare bf-tree.txt df-tree.txt Please help review and feel free to let me know if anything I can help. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17551980#comment-17551980 ] ASF GitHub Bot commented on MRESOLVER-7: caiwei-ebay commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1150721126 @cstamas I can reproduce. Let me do some research. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17550590#comment-17550590 ] ASF GitHub Bot commented on MRESOLVER-7: cstamas commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1147694146 Something is wrong here, unsure what. But, I tried to build maven master using maven-3.9.x + this PR + empty local repo and it fails: ``` [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile (default-testCompile) on project maven-model-builder: Compilation failure: Compilation failure: [ERROR] /home/cstamas/Worx/apache-maven/maven/maven-model-builder/src/test/java/org/apache/maven/model/interpolation/StringSearchModelInterpolatorTest.java:[36,27] package org.hamcrest does not exist [ERROR] /home/cstamas/Worx/apache-maven/maven/maven-model-builder/src/test/java/org/apache/maven/model/interpolation/StringSearchModelInterpolatorTest.java:[36,1] static import only from classes and interfaces [ERROR] /home/cstamas/Worx/apache-maven/maven/maven-model-builder/src/test/java/org/apache/maven/model/interpolation/StringSearchModelInterpolatorTest.java:[37,27] package org.hamcrest does not exist [ERROR] /home/cstamas/Worx/apache-maven/maven/maven-model-builder/src/test/java/org/apache/maven/model/interpolation/StringSearchModelInterpolatorTest.java:[37,1] static import only from classes and interfaces [ERROR] -> [Help 1] ``` The cmd used to test: ``` [cstamas@infinity maven (maven-3.9.x)]$ ~/bin/maven/apache-maven-3.9.0-SNAPSHOT/bin/mvn clean install -Dmaven.repo.local=local-repo -Drat.skip -DskipTests -Daether.collector.impl=bf ``` > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17538348#comment-17538348 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1129124711 Much simpler to read now... > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17538064#comment-17538064 ] ASF GitHub Bot commented on MRESOLVER-7: caiwei-ebay commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1128643994 Separated the PR for PremanagedDependency refactoring and rebased the code. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537846#comment-17537846 ] ASF GitHub Bot commented on MRESOLVER-7: caiwei-ebay commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1128255102 @michael-o Created separate PR for moving the PremanagedDependency class. https://github.com/apache/maven-resolver/pull/179 > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537499#comment-17537499 ] ASF GitHub Bot commented on MRESOLVER-7: caiwei-ebay commented on PR #178: URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1127577923 @michael-o @cstamas @ibabiankou I think the code is already ready for review, I would like to run more tests before converting the PR from Draft to final. Please help review and share your comments with this PR. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466473#comment-17466473 ] Ivan Babiankou commented on MRESOLVER-7: I opened [a PR|https://github.com/apache/maven-resolver/pull/142] to implement this change. Unit and integration tests pass, however there seems to be a race condition (see stack trace [^resolver.log] ) that is hard to reproduce. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement > Components: Resolver >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Attachments: resolver.log > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17274268#comment-17274268 ] Michael Osipov commented on MRESOLVER-7: [~ppanda-beta], if you really need this feature then hire a contractor, pay him €€€ and we will review his PR. You obviously don't understand that we are volunteers don't have to do nothing for you! > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273947#comment-17273947 ] Palash Das commented on MRESOLVER-7: You people keep fighting over decency of a PR or deadlocks of multi threaded downloads, but in reality this forces us to pay free money to the cloud providers for CI time. When we have # sufficient network bandwidth # High speed SSD # And good amount of cpu *we still have to pay 3 hrs of CI time just to download 3.4GBs of jar files.* Great collaboration skills guys. Great appreciations towards the OSS efforts that people show on their PRs. Approve the PR, merge it and revert it within 1 month. Awesome reward received. Time will come when we will switch the build tool instead of a dependency plugin. Cheers to gradle / bazel. Those comment fights will be remembered! > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17188027#comment-17188027 ] Michael commented on MRESOLVER-7: - I regret I can not like the previous comment. :) [~michael-o] cheers. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17171271#comment-17171271 ] Michael Osipov commented on MRESOLVER-7: [~rajjaiswalsaumya], when you provide a decent PR. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17171245#comment-17171245 ] Rohit Gupta commented on MRESOLVER-7: - When this will make way in maven ? > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17152577#comment-17152577 ] Michael Osipov commented on MRESOLVER-7: [~kwin], to be honest, no idea. Maybe, you can try yourself. This is currently out of scope for me. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17152569#comment-17152569 ] Konrad Windszus commented on MRESOLVER-7: - [~michael-o] Does MRESOLVER-123 fix some of the issues which appeared with https://github.com/apache/maven-resolver/pull/30? > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907627#comment-16907627 ] Hudson commented on MRESOLVER-7: Build succeeded in Jenkins: Maven TLP » maven-resolver » master #30 See https://builds.apache.org/job/maven-box/job/maven-resolver/job/master/30/ > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904845#comment-16904845 ] Hudson commented on MRESOLVER-7: Build succeeded in Jenkins: Maven TLP » maven-resolver » pre-MRESOLVER-92 #2 See https://builds.apache.org/job/maven-box/job/maven-resolver/job/pre-MRESOLVER-92/2/ > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885618#comment-16885618
]
Tibor Digana commented on MRESOLVER-7:
--
[~hwellmann]
We have to revert this feature in resolver and resolver:1.4.0 in Maven. It
should be reviewed in a separate branch.
As an explanation, our Maven builds started failing after resolver:1.4.0 has
been applied.
To be unbiased and fully logical, I investigated the issue and the feature on
ASF Slack. It was me who found several problems in the code, root cause of
hanging Maven and deadlock of thread-resources starvation, problems with
thread-unsafe classes, overrided objects across threads, one unclear part in
the algorithm. And I also found that the algorithm crashed 11 unit tests in
{{maven-resolver}} after i removed ThreadPoolExecutor (in a playgrounded
non-master branch) and thus made it single-threaded.
I am sorry and not glad to say these things, but I would like you and ASF Maven
to make code review of this feature, where every question and line must be
clarified. Are you open for attending in such code review?
I may copy paste the text from Slack regarding our findings.
But currently we are responsible to solve our internal problems with failing
builds.
The Jenkins builds have been failing over one month!
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Michael Osipov
>Priority: Major
> Fix For: 1.4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818910#comment-16818910 ] Hudson commented on MRESOLVER-7: Build failed in Jenkins: Maven TLP » maven-resolver » master #12 See https://builds.apache.org/job/maven-box/job/maven-resolver/job/master/12/ > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806227#comment-16806227
]
Harald Wellmann commented on MRESOLVER-7:
-
Rebased Sylwester's changes in a new PR
https://github.com/apache/maven-resolver/pull/30 and fixed the failing tests.
Added a new configuration property {{aether.artifact.descriptor.threads}} for
the number of threads used for parallel download of POMs.
The default is 5. Setting this to 1 reverts to the previous behaviour.
> Download dependency POMs in parallel
>
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
>Affects Versions: Aether 1.0.2
>Reporter: Harald Wellmann
>Assignee: Sylwester Lachiewicz
>Priority: Major
> Fix For: 1.4.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730962#comment-16730962 ] Karl Heinz Marbaise commented on MRESOLVER-7: - Ok Thanks Sylwester for the feedback. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: version-next > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730955#comment-16730955 ] Sylwester Lachiewicz commented on MRESOLVER-7: -- Please go without this fix, I'll try to work on this in next year :D > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: version-next > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730809#comment-16730809 ] Karl Heinz Marbaise commented on MRESOLVER-7: - [~slachiewicz] Can you take a look at it cause I want to start a release within the next few days...? > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: version-next > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645491#comment-16645491 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-428714346 @slachiewicz, do you want to take another look at it. We want to roll 1.3.1 next week. I'd be cool if we could include this. My last questions are still open. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: version-next > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636740#comment-16636740 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-426581962 While I understand MRESOLVER-9 I do not understand this failure. How can this happen if we don't have any changes in dep resolution, do we? Do I need to merge MRESOLVER-9 first? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.2.0 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636306#comment-16636306 ] ASF GitHub Bot commented on MRESOLVER-7: slachiewicz edited a comment on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-426475151 Testcase: org.apache.maven.project.inheritance.t06.ProjectInheritanceTest#testDependencyManagement and poms in maven-compat/src/test/resources/inheritance-repo/t06 Expected: Artifact: maven-test:t06-a:jar 1.0 scope=compile optional=false Artifact: maven-test:t06-b:jar 1.0 scope=compile optional=false Artifact: maven-test:t06-d:jar 1.0 scope=test optional=false Artifact: maven-test:t06-c:jar 1.0 scope=compile optional=false Actual: Artifact: maven-test:t06-a:jar 1.0 scope=compile optional=false Artifact: maven-test:t06-b:jar 1.0 scope=compile optional=false Artifact: maven-test:t06-c:jar 1.0 scope=compile optional=false Can be related to https://issues.apache.org/jira/browse/MRESOLVER-9 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.2.0 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636277#comment-16636277 ] ASF GitHub Bot commented on MRESOLVER-7: slachiewicz commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-426475151 Testcase: org.apache.maven.project.inheritance.t06.ProjectInheritanceTest#testDependencyManagement and poms in maven-compat/src/test/resources/inheritance-repo/t06 Expected: Artifact: maven-test:t06-a:jar 1.0 scope=compile optional=false Artifact: maven-test:t06-b:jar 1.0 scope=compile optional=false Artifact: maven-test:t06-d:jar 1.0 scope=test optional=false Artifact: maven-test:t06-c:jar 1.0 scope=compile optional=false Actual: Artifact: maven-test:t06-a:jar 1.0 scope=compile optional=false Artifact: maven-test:t06-b:jar 1.0 scope=compile optional=false Artifact: maven-test:t06-c:jar 1.0 scope=compile optional=false This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.2.0 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632978#comment-16632978 ] ASF GitHub Bot commented on MRESOLVER-7: slachiewicz commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-425641765 That use of DDCC is hard to remove without extending resolver-api and its jhard to understand how this work. I’m able to reproduce this bug but no idea about reason. I’ll look into this more later. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630829#comment-16630829 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-425187159 @slachiewicz I'd like to publish MRESOLVER in a couple of days. Can you have a look at the PR again please? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627842#comment-16627842 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-424476391 Maven master fails with: [INFO] Running org.apache.maven.repository.MirrorProcessorTest [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in org.apache.maven.repository.MirrorProcessorTest [INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] ProjectInheritanceTest.testDependencyManagement:68 Set size should be 4, is 3 [INFO] [ERROR] Tests run: 142, Failures: 1, Errors: 0, Skipped: 0 Please review. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627835#comment-16627835 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on a change in pull request #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#discussion_r220323887 ## File path: maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/DefaultDependencyCollector.java ## @@ -91,6 +98,13 @@ private VersionRangeResolver versionRangeResolver; +private static final ThreadGroup THREAD_GROUP = new ThreadGroup( "Maven Resolver dependency resolution" ); + +/** + * Setup executor with 5 daemon threads in pool, max to 10 concurrent and keep alive to 10 sec inactivity. Review comment: `10 sec` to > `10 s`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627834#comment-16627834 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on a change in pull request #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#discussion_r220323542 ## File path: maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/DefaultDependencyCollector.java ## @@ -91,6 +98,13 @@ private VersionRangeResolver versionRangeResolver; +private static final ThreadGroup THREAD_GROUP = new ThreadGroup( "Maven Resolver dependency resolution" ); + +/** + * Setup executor with 5 daemon threads in pool, max to 10 concurrent and keep alive to 10 sec inactivity. + */ +private ExecutorService executor = createExecutorService( 5, 10, 10 ); Review comment: Last arg is `10L`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627822#comment-16627822 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-424467210 I will drop cf1a1f7 because this is a different issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618173#comment-16618173 ] ASF GitHub Bot commented on MRESOLVER-7: slachiewicz commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-422178076 yes, it’s final version :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618168#comment-16618168 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-422177767 Is this one now complete and can be reviewed again by me? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616851#comment-16616851 ] Michael Osipov commented on MRESOLVER-7: Will review next week. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616825#comment-16616825 ] Sylwester Lachiewicz commented on MRESOLVER-7: -- [~michael-o] any plan to review? Maybe include this one with new Maven 3.6? > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562583#comment-16562583 ] ASF GitHub Bot commented on MRESOLVER-7: slachiewicz commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-409028201 ok, finaly i reuse in f1954149b7f4eb7ec7ccad6591c78702c959975e existing code used to create threads (org.eclipse.aether.util.concurrency.WorkerThreadFactory) in org.eclipse.aether.internal.impl.DefaultMetadataResolver#getExecutor This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: Maven Artifact Resolver 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16552177#comment-16552177 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-406898751 Can someone tell why the daemon threads along with the cached pool is necessary? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: Maven Artifact Resolver 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534157#comment-16534157 ] ASF GitHub Bot commented on MRESOLVER-7: slachiewicz commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-402853151 Thx @mikehearn for tests and suggestion to use cachedThreadPool. I made small modification to use 5 core threads and max to 10 thread. Could you test new version? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: Maven Artifact Resolver 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532759#comment-16532759 ] ASF GitHub Bot commented on MRESOLVER-7: mikehearn commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-402480226 For what it's worth I've been testing it locally a fair bit and it's been working fine. However I had to fork Maven Resolver and add a few commits on top to get it working as well as I wanted it to: https://github.com/mikehearn/maven-resolver/commits/master Specifically, I had to roll back a change that conflicted with this PR, and then I added this commit: https://github.com/mikehearn/maven-resolver/commit/1e2ed26e5c6b7f439882339deae8ff4effec50ac to avoid creating non-daemon threads. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: Maven Artifact Resolver 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530450#comment-16530450 ] Michael Osipov commented on MRESOLVER-7: Master is currently broken. > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Assignee: Michael Osipov >Priority: Major > Fix For: Maven Artifact Resolver 1.1.2 > > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528933#comment-16528933 ] ASF GitHub Bot commented on MRESOLVER-7: slachiewicz commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-401574365 I did rebase code and I compile few projects with clean repo - it looks good and fast. @steve-todorov could you try to test with Your projects also? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528898#comment-16528898 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o edited a comment on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-401566300 @steve-todorov Did you try the patch yourself? Unfortunately, there is a very little amount of people working on Resolver right now. I am personally too busy. There is so much new code that I have trouble to understand it, especially because I am not fully comfortable with the Resolver code. But I am still looking into it. @slachiewicz Can you rebase again? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528897#comment-16528897 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-401566300 @steve-todorov Did you try the patch yourself? Unfortunately, there is a very little amount of people working on Resolver right now. I am personally too busy. There is so much new code that I have trouble to understand it, especially because I am not fully comfortable with the Resolver code. @slachiewicz Can you rebase again? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528845#comment-16528845 ] ASF GitHub Bot commented on MRESOLVER-7: steve-todorov commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-401558666 Hi guys, are there any plans for this to be merged anytime soon? We have a bit project which might benefit from this. Is there anything more needed to be done? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429737#comment-16429737 ] ASF GitHub Bot commented on MRESOLVER-7: slachiewicz commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-379544562 and I’m just trying to help:) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429736#comment-16429736 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-379544123 @slachiewicz I maybe wrote gibberish, I am working on five projects in parallel and simply lost track. Don't take my request as imperative. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429733#comment-16429733 ] ASF GitHub Bot commented on MRESOLVER-7: slachiewicz commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-379543557 strange, but You requested to move #11 to diffrent PR. So now You need to revert #11 then you can apply again this PR.. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MRESOLVER-7) Download dependency POMs in parallel
[ https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429732#comment-16429732 ] ASF GitHub Bot commented on MRESOLVER-7: michael-o commented on issue #10: [MRESOLVER-7] Download dependency POMs in parallel v2 URL: https://github.com/apache/maven-resolver/pull/10#issuecomment-379543204 @mikehearn What needs to be changed to make classes accesible for this PR? I'll change it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Download dependency POMs in parallel > > > Key: MRESOLVER-7 > URL: https://issues.apache.org/jira/browse/MRESOLVER-7 > Project: Maven Resolver > Issue Type: Improvement >Affects Versions: Aether 1.0.2 >Reporter: Harald Wellmann >Priority: Major > > h3. Background > When building a project with dependencies not yet available in the local > repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the > dependency POMs _sequentially_ and then proceeds downloading the dependency > JARs with up to 5 threads _in parallel_. > Due to this, when first building a project with a large number of > dependencies, downloading a large number of small POMs may take a lot longer > than downloading the much larger JARs, or even longer than building the > project itself, especially when a repository manager is used which increases > the download latency. > h3. Enhancement > Download POMs of (transitive) dependencies in parallel to significantly speed > up initial builds of large projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
