brianf opened a new pull request, #58508:
URL: https://github.com/apache/spark/pull/58508
### What changes were proposed in this pull request?
Set the `http.agent` system property in `MavenUtils` so that Ivy identifies
itself as Spark when resolving `--packages` / `spark.jars.packages` coordinates
from remote repositories.
Requests now carry:
```
Apache-Spark/4.0.0 (Apache-Ivy/2.5.3)
```
instead of Ivy's default `Apache Ivy/2.5.3`.
The format follows RFC 9110 Section 10.1.5 (`product *( RWS ( product /
comment ) )`) — a product token carrying Spark's identity and version, plus a
comment carrying the underlying resolution engine, so operator tooling that
already matches on the Ivy version keeps working.
Details:
- `http.agent` is Ivy's own designed override point —
`AbstractURLHandler.getUserAgent()` returns `System.getProperty("http.agent",
"Apache Ivy/" + Ivy.getIvyVersion())`, and both `BasicURLHandler` and
`HttpClientHandler` inherit it. No new Ivy API or custom handler is required.
- The property is set **once, if absent**, rather than set-and-restored
around the download. `URLHandlerRegistry.getHttp()` returns
`HttpClientHandler.DELETE_ON_EXIT_INSTANCE` when httpclient is on the classpath
(it is — httpclient 4.5.14 ships in the distribution), and that singleton bakes
the User-Agent in at class-initialization time via
`HttpClients.custom().setUserAgent(...)`. A save/set/restore wrapper would
therefore have no effect on requests and would race between concurrent
resolutions.
- `buildIvySettings()` / `loadIvySettings()` are the insertion points: all
three call paths (`Artifact.scala`, `IsolatedClientLoader.scala`,
`DependencyUtils.scala`) go through one of them before
`resolveMavenCoordinates()`, and neither touches HTTP itself. In
`loadIvySettings()` it is the first statement, ahead of
`ivySettings.load(file)`, because `XmlSettingsParser` is the one reachable path
that can trigger `HttpClientHandler`'s static init (when the settings XML
carries `httpRequestMethod`).
- `SparkBuildInfo.spark_version` is used rather than
`org.apache.spark.SPARK_VERSION`, which lives in `core` and is not available
from `common/utils`.
### Why are the changes needed?
Repository operators rely on the User-Agent to attribute traffic and to
reach the responsible party when a client misbehaves. A bare `Apache Ivy/2.5.3`
is un-attributable: it could be Spark, Ant, sbt, or a hand-rolled Ivy client.
Operators cannot tell which tool to talk to, cannot distinguish an old Spark
with poor caching behavior from a current one, and have no signal to route a
conversation about excessive consumption.
This matters in practice because Spark clusters without a repository-manager
proxy generate a burst of resolution traffic at every job launch, so Spark is a
meaningful share of this traffic on public repositories.
### Does this PR introduce _any_ user-facing change?
Yes, in two respects, both additive:
1. Requests Spark makes to remote artifact repositories during dependency
resolution now carry `User-Agent: Apache-Spark/<version>
(Apache-Ivy/<version>)` instead of `Apache Ivy/<version>`. No resolution
behavior changes.
2. Setting `-Dhttp.agent=...` explicitly continues to win — the property is
only set when unset — and this is now documented under `spark.jars.packages` in
`docs/configuration.md`.
Note that `http.agent` is a JVM-global property affecting any
`HttpURLConnection`, not only Ivy. In practice the Spark-adjacent HTTP clients
(Hadoop, AWS SDK, Kubernetes client) use their own stacks and set their own
User-Agent, so the blast radius is small, and labelling stray JDK-level
requests from a Spark driver as Spark is accurate anyway. The
set-only-if-absent guard is the mitigation.
### How was this patch tested?
New tests in `MavenUtilsSuite`:
- the User-Agent matches the RFC 9110 `product RWS comment` form, and is
composed from the real `SparkBuildInfo.spark_version` and `Ivy.getIvyVersion`
rather than a hardcoded literal;
- `buildIvySettings` sets the property when it is unset;
- `loadIvySettings` sets it too;
- an explicitly configured value is not overwritten.
The suite's `beforeEach`/`afterEach` were extended to save, clear, and
restore `http.agent`, so the new global side effect cannot leak between tests
or into other suites in the same JVM.
```
./build/mvn -pl common/utils -am test
```
→ 101 tests succeeded, 0 failed (`MavenUtilsSuite`: `tests="17" failures="0"
errors="0"`).
`./dev/scalastyle` → passed.
Additionally, verified on the wire rather than only by inspection, since the
`HttpClientHandler` class-init timing is the part worth confirming empirically.
A local HTTP server was pointed at as the repository and the request headers
logged:
```
CONFIGURED_AGENT=Apache-Spark/5.0.0-SNAPSHOT (Apache-Ivy/2.5.3)
OBSERVED_USER_AGENT=Apache-Spark/5.0.0-SNAPSHOT (Apache-Ivy/2.5.3)
```
and with the override in place:
```
CONFIGURED_AGENT=AcmeCorp/1.0
OBSERVED_USER_AGENT=AcmeCorp/1.0
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
