[
https://issues.apache.org/jira/browse/CAMEL-26901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144064#comment-18144064
]
Claus Ibsen commented on CAMEL-26901:
-------------------------------------
The PRs for this issue, in merge order:
# https://github.com/apache/camel/pull/29774 - camel run stops when a declared
dependency cannot be downloaded (exit 1, one line); a remote SNAPSHOT has its
base version (findings 2, 3). Targets main.
# https://github.com/apache/camel/pull/29777 - leave out what is already on the
classpath while resolving, also with the launcher's manifest Class-Path;
artifact ids with dots (finding 4 + the two from the comment above). Targets
main. Cold run with the launcher: 6.5-7s instead of 13.7-16s.
# https://github.com/apache/camel/pull/29778 - one version of each library
(finding 1 step 1): the libraries already loaded are managed versions for the
next resolutions, per groupId:artifactId whatever the classifier. 21 libraries
in several versions -> 0. Stacked on #29777.
# https://github.com/apache/camel/pull/29781 - request timeout 60s instead of
30 min; a repository that cannot be connected to is not asked again (finding
5). Stacked on #29778.
# https://github.com/apache/camel/pull/29782 - the given dependencies are
resolved together, with the breadth-first collector of Maven Resolver (finding
6). Stacked on #29781, also merges #29774. Cold run with 5 --dep: 8.4-9.2s
instead of 12.8-13.8s, same classpath.
#29774 and #29778 both append to the end of the 4.24 upgrade guide, so the
second one needs a rebase. Not done: finding 7 (smaller items), and the
cosmetic "Downloaded ... (took: 0ms)" log lines.
_Claude Code on behalf of Claus Ibsen (davsclaus)_
> camel-jbang - camel run: dependency resolution mixes library versions on the
> classpath, ignores failed downloads, and downloads jars already on the
> classpath
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26901
> URL: https://issues.apache.org/jira/browse/CAMEL-26901
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> An analysis of how camel run (the jbang runtime: KameletMain +
> MavenDependencyDownloader + camel-tooling-maven MavenDownloaderImpl) resolves
> and downloads dependencies. A warm run is already fast; the problems are on
> cold runs and on failures, and the most important one is a correctness issue:
> dependencies resolved one by one end up on the classpath in several versions
> at once.
> h3. Measurements (4.24.0-SNAPSHOT CLI, macOS, small YAML route with groovy,
> jsonpath and mustache)
> ||Scenario||Camel boot||Wall time||Notes||
> |Warm ~/.m2, via jbang|0.5s|2.6s|resolution 87 + 33 + 5 ms|
> |Warm, camel-launcher distribution|0.6s|2.5s|jbang adds almost nothing|
> |Warm, plus 10 --dep components|1.7s| |resolution about 40 ms per component|
> |Warm, unreachable repo in --repos|0.5s|2.6s|a warm run never contacts the
> remote repo|
> |Cold (empty local repo)|8-9.5s|10-11.5s|90 downloads, 19 MB of jars|
> |Cold, one missing dep with an unreachable repo|20.8s| |two 10s connect
> timeouts|
> h3. 1. The same library on the classpath in several versions (verified, high)
> Each dependency is resolved as its own graph, and {{alreadyOnClasspath}}
> matches on the exact version ({{MavenDependencyDownloader}} around line 350),
> so a different version of a library that is already loaded is still added to
> the {{DependencyDownloaderClassLoader}}.
> A run with
> {{--dep=camel:kafka,camel:aws2-s3,camel:sql,camel:http,camel:jms,camel:mongodb,camel:google-pubsub,camel:azure-storage-blob,camel:elasticsearch,camel:infinispan}}
> added:
> * netty in 3 versions (4.1.127, 4.1.138, 4.1.139)
> * httpclient5 in 3 versions (5.6.3, 5.6.4, 5.6.5)
> * jackson-core in 3 versions (2.18.2, 2.22.2, 2.22.3)
> * opentelemetry-api/context/common in 2 versions (1.62.0, 1.67.0)
> URLClassLoader finds each class in the first jar that has it, so a class
> missing from one version is loaded from another: NoSuchMethodError /
> IncompatibleClassChangeError depending on the order components are
> discovered. camel export lets Maven pick one version per library, so camel
> run and the exported project can have different classpaths.
> Step 1 (cheap): when a library is already loaded in another version, do not
> add the other version; log it (verbose). This does not change which version
> wins today, it only stops the mixing.
> Step 2: see 6 (resolve the known dependencies as one graph).
> h3. 2. A failed download is logged and the run continues (verified, high)
> {{DownloadThreadPool.awaitCompletion}} (line 84) logs an ERROR with the full
> stack trace and returns. {{--dep=com.example.nope:doesnotexist:1.0}} printed
> about 40 lines of stack trace, then started the routes and exited 0.
> For dependencies the user named ({{--dep}}, {{//DEPS}},
> {{camel.jbang.dependencies}}) the run should stop with a one-line message
> (the artifact and the repositories tried) and a non-zero exit code. Export
> mode may keep the lenient behaviour on purpose. For automatically detected
> components, a short message instead of the stack trace.
> h3. 3. Remote SNAPSHOTs put core jars on the classpath again (verified,
> nightly CLI users)
> {{MavenDownloaderImpl.toMavenArtifact}} uses {{Artifact.getVersion()}}, which
> for a remote snapshot is the timestamped version
> ({{4.24.0-20261112.143700-31}}). It never matches
> {{camel-api-4.24.0-SNAPSHOT.jar}} on the boot classpath (nor the URL check,
> as the file lives in the {{4.24.0-SNAPSHOT}} directory), so the cold run
> added camel-api, camel-support, camel-util, camel-core-model, ... to the
> download classloader, some of them 3 times. Fix: use {{getBaseVersion()}}.
> h3. 4. Cold runs download jars that are already on the boot classpath
> (verified, high cold-start cost)
> 9.0 of the 18.8 MB of jars (48%) in the cold run were camel-api,
> camel-support, camel-core-model, camel-core-processor, camel-base-engine,
> jackson 3, slf4j-api, jakarta.xml.bind-api, ... They are downloaded (with
> their POMs and snapshot metadata round trips) and only filtered out after the
> fact by {{alreadyOnClasspath}}.
> Fix: leave out what is already on the boot classpath during resolution,
> matching on groupId:artifactId only (a DependencySelector / filter in the
> collect request). Classes load parent-first, so the boot version wins today
> anyway.
> h3. 5. Timeouts and unreachable repositories (verified, medium)
> Maven resolver 1.9.27 defaults: connect timeout 10s, request timeout *30
> minutes* ({{DEFAULT_REQUEST_TIMEOUT = 1800000}}). Camel only sets them in
> {{MavenDownloaderImpl.customize()}}, not for camel run.
> * A mirror that accepts the connection and then stalls hangs camel run for up
> to 30 minutes, printing "Downloading: ... (elapsed: ...)" every 5s.
> * An unreachable repository costs 10s per request (POM, then jar), and it is
> not remembered for the rest of the run.
> Fix: a sane request timeout (60-120s), and skip a repository for the rest of
> the run after its first connect failure.
> h3. 6. One dependency at a time, synchronously (verified, medium on cold runs)
> {{DownloadThreadPool.download}} submits the task and waits for it; the thread
> pool only serves the progress log. Cold run: groovy 5.6s, then jsonpath 2.3s,
> then mustache 0.4s, one after another.
> Dependencies known up front ({{--dep}}, {{//DEPS}}, properties) could be
> resolved in a single request: one graph with one version per library (fixes 1
> for them) and parallel downloads. {{downloadDependencyWithParent}}
> (CAMEL-22100) is a natural place for this.
> h3. 7. Smaller items (from reading the code)
> * *Cross-process locking*: the resolver uses {{rwlock-local}} named locks,
> which only coordinate within one JVM. Several cold camel run processes (CI,
> several integrations at once) are not coordinated through them; Maven
> recommends {{file-lock}} for a shared local repository. Not reproduced.
> * *Package scan per jar*:
> {{BasePackageScanDownloadListener.onDownloadedFile}} re-runs the full base
> package scan for every downloaded jar when basePackageScan is set; once per
> resolution is enough.
> * *Misleading log lines*: "Downloaded: ... (took: 0ms)" as the elapsed time
> is hard-coded to {{0L}} in {{MavenDownloaderImpl}}; the POM and the jar are
> both reported as "Downloaded" without the extension; the repository listener
> does not check whether the event carries an error.
> * *Thread safety*: {{downloadRecords}} is a HashMap written from the download
> threads; on interrupt the future is not cancelled.
> h3. Not worth doing
> * Caching the resolved classpath for warm runs: warm resolution is about 0.5s
> for 13 components.
> * Switching to the breadth-first dependency collector:
> {{-Daether.dependencyCollector.impl=bf}} measured 9.8-10.4s against
> 10.7-11.6s for the default, within network noise.
> * jbang vs the launcher: about 0.1s apart.
> h3. Suggested order
> # 3 (one-liner) and 2 (fail fast for explicit dependencies)
> # 1 step 1 (one version per library) together with 4 (leave out boot
> classpath jars during resolution)
> # 5 (timeouts, unreachable repositories)
> # 6 (batch the known dependencies), then measure the cold start again
> _Claude Code on behalf of Claus Ibsen (davsclaus)_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
