davsclaus opened a new pull request, #29782: URL: https://github.com/apache/camel/pull/29782
JIRA: [CAMEL-26901](https://issues.apache.org/jira/browse/CAMEL-26901) (finding 6). **Stacked on #29781** (repository timeouts), which is stacked on #29778 and #29777. This branch also merges #29774 (fail fast on declared dependencies), because both change `CommandLineDependencyDownloader` and `DownloadThreadPool`. **Review only the last commit**: 8b7855fe4316. ### Resolve the given dependencies together `camel run` resolved every dependency on its own, one after another, waiting for each. The dependencies given with `--dep` or `camel.jbang.dependencies` are known up front, so `CommandLineDependencyDownloader` now passes them to `DependencyDownloader.downloadDependencies(gavs, required)`, which resolves them in one request: - One dependency graph, so Maven Resolver's version mediation applies across them. - Their JARs are downloaded in parallel. - When they cannot be resolved together (one does not exist), they are resolved one at a time. Only the one that cannot be downloaded fails (`camel run`), or is logged (`camel export`). The error still names it. - The interface method has a default that loops, so other implementations keep working. ### The breadth-first collector Measured alone, the batch gained nothing: the time goes into collecting the graph, and the depth-first collector (Maven Resolver's default) downloads the POMs one at a time. `camel-tooling-maven` now uses the breadth-first collector (`aether.dependencyCollector.impl=bf`) with 10 threads, which downloads the POMs in parallel. It resolves the same dependencies: the classpath was identical in every comparison below, and `camel export` writes the same project. The system properties still win, so `-Daether.dependencyCollector.impl=df` goes back. ### Measured Cold `camel run` (empty local repository) of a timer route with `--dep=camel:groovy,camel:jsonpath,camel:kafka,camel:http,camel:aws2-s3`, launcher distribution patched with the JARs of this stack: | | wall time | |---|---| | before (one at a time, depth-first) | 12.8s - 13.8s (4 runs) | | together, depth-first | 12.8s, 13.1s | | one at a time, breadth-first (5 threads) | 11.0s, 11.8s | | together, breadth-first (5 threads) | 9.0s, 9.6s | | **together, breadth-first (10 threads), this PR** | **8.4s, 9.2s** | All of them: the same 65 JARs, 117 POMs. Warm (eight components, everything in `~/.m2`): the batch resolves in about 280 ms instead of 450-670 ms. ### Tests - `MavenDependencyDownloaderBatchTest` (a real `MavenDependencyDownloader` against a local file repository, local repository in a temp dir): - Dependencies resolved together get one version of a shared library. - With one that does not exist: the others are still downloaded when not required, and when required it fails naming that one. - `MavenDownloaderImplTest`: the Camel defaults of the session. - camel-tooling-maven (45 tests) and camel-kamelet-main (85 tests): all pass. Earlier selector and managed-version tests now run with the breadth-first collector too. - By hand: the runs above, a route using groovy, jsonpath and mustache, a missing `--dep` (exit 1, names it), and `camel export` (same project). Docs: the running page ("Downloading JARs over the internet") and a 4.24 upgrade guide entry, placed in the middle of the file next to the #29781 entry to avoid another end-of-file conflict. _Claude Code on behalf of Claus Ibsen (davsclaus)_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
