[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558026#comment-17558026
]
ASF GitHub Bot commented on MRESOLVER-7:
----------------------------------------
caiwei-ebay commented on PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#issuecomment-1164305940
@michael-o @cstamas
I have converted the PR from draft to ready. Please help review.
The basic idea of this PR is:
- Given dependency A's children B, C, D
- Iterate B, C, D one by one, take B as example below.
- Should B be filtered? If yes, won't add to the queue. Logic
[here](https://github.com/apache/maven-resolver/blob/2c805b5a7979b0d53ad22d9eb6947f6613a13357/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java#L354)
- If B should not filtered, then figure out the managed version of B, ex
B1, resolve B1's descriptor in a thread, then add B1 to the queue. Logic
[here](https://github.com/apache/maven-resolver/blob/2c805b5a7979b0d53ad22d9eb6947f6613a13357/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java#L359)
In such way, poms and metadata.xml are downloaded in parallel.
The issue @cstamas raised has been fixed. The failure was caused incorrect
filtering (should filter with original dependency, B instead of B1 as described
in above flow)
Also runs the ITs with both DF and BF (with above parallel downloading)
mode, same test result below:
2 tests not related with this PR failed with both mode. Not sure if this is
expected.
`
[ERROR] Failures:
[ERROR]
MavenITmng5576CdFriendlyVersions>AbstractMavenIntegrationTestCase.runTest:260->testContinuousDeliveryFriendlyVersionsAreWarningFreeWithBuildConsumer:98
[WARNING]
[ERROR]
MavenITmng5576CdFriendlyVersions>AbstractMavenIntegrationTestCase.runTest:260->testContinuousDeliveryFriendlyVersionsAreWarningFreeWithoutBuildConsumer:67
[WARNING]
[INFO]
[ERROR] Tests run: 912, Failures: 2, Errors: 0, Skipped: 0`
Also dry-run 1000+ apps in our company by comparing the maven dependency
tree with DF and BF mode (based on this PR), no issues found so far. Test
method is like this:
- mvn clean install -DskipTests -Daether.collector.impl=bf
-Dmaven.repo.local=bf
- mvn dependency:tree -DoutputFile=bf-tree.txt -Daether.collector.impl=bf
-Dmaven.repo.local=bf
- mvn clean install -DskipTests -Dmaven.repo.local=df
- mvn dependency:tree -DoutputFile=df-tree.txt -Dmaven.repo.local=df
- compare bf-tree.txt df-tree.txt
Please help review and feel free to let me know if anything I can help.
> Download dependency POMs in parallel
> ------------------------------------
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
> Affects Versions: Aether 1.0.2
> Reporter: Harald Wellmann
> Priority: Major
> Attachments: resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
