sarutak opened a new pull request, #58051:
URL: https://github.com/apache/spark/pull/58051
### What changes were proposed in this pull request?
Add a `credential-aws` profile to `assembly/pom.xml`, following the same
pattern as the existing `hadoop-cloud` profile:
```xml
<profile>
<id>credential-aws</id>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-credential-aws_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</profile>
```
This is a follow-up to SPARK-57897, which created the
`connector/credential-aws` module and wired the `-Pcredential-aws` profile into
the root POM, `SparkBuild.scala`, and the CI/release scripts, but did not add
the module to `assembly/pom.xml`.
Because the assembly now pulls in `spark-credential-aws` (and its transitive
AWS SDK v2 dependencies) when `-Pcredential-aws` is active, the dependency
manifest checked by `dev/test-dependencies.sh` is regenerated. This adds the
AWS SDK v2 artifacts (`sts` and its transitive deps, version 2.35.4) to
`dev/deps/spark-deps-hadoop-3-hive-2.3`, consistent with how `hadoop-cloud`
transitive deps (`hadoop-aws`, `hadoop-cloud-storage`, `wildfly-openssl`,
etc.) are already listed there. `dev/test-dependencies.sh` includes
`-Pcredential-aws` in its profile list, so it fails until the manifest is
updated.
### Why are the changes needed?
Without an entry in `assembly/pom.xml`, building with `-Pcredential-aws`
does not place `spark-credential-aws_*.jar` into
`assembly/target/scala-*/jars/`. Both `dev/make-distribution.sh` and
`bin/docker-image-tool.sh` source their JARs from that directory, so the module
could not be included in a custom distribution or a custom Docker image even
when the profile was explicitly requested.
`hadoop-cloud` is an analogous optional module and already has an assembly
profile, so a user can opt in with `-Phadoop-cloud`. `credential-aws` should
behave the same way.
Note: this is an opt-in path only. The official binary distributions are
built without `-Pcredential-aws` (`BINARY_PKGS_ARGS["hadoop3"]="-Phadoop-3
$HIVE_PROFILES"` in `dev/create-release/release-build.sh`), so the module is
still excluded from the default distribution and the default Docker image,
consistent with the design intent of keeping the AWS SDK dependencies out of
the core distribution. The module continues to be published to Maven Central
as a standalone artifact (via `PUBLISH_PROFILES`) for consumption with
`--packages`.
### Interaction with the hadoop-cloud profile
The E2E test (Minikube + LocalStack, SPARK-57900) exercises the full flow
`token -> STS -> S3A read/write`. The S3A read/write half requires
`hadoop-aws`, which is only available at runtime via the `hadoop-cloud` profile
(`connector/credential-aws` itself does not depend on `hadoop-aws`; it only
returns `fs.s3a.*` credential properties). So the E2E image is expected to be
built with both `-Pcredential-aws -Phadoop-cloud`.
When both profiles are active, the assembly contains this module's
individual `sts` jar (plus its transitive AWS SDK v2 deps) as well as
hadoop-cloud's `software.amazon.awssdk:bundle` jar. `bundle` is a fat jar that
already contains the same `software.amazon.awssdk.*` classes (e.g.
`StsClient`), so those classes are duplicated on the classpath. This was
verified to be harmless as long as both resolve to the same version:
- The duplicated classes are byte-identical (same version 2.35.4), so
whichever the classloader picks behaves identically. No `LinkageError`.
- The synchronous `SdkHttpService` implementation registered by both
`bundle` and `apache-client` is the single class `ApacheSdkHttpService`, so the
AWS SDK does not raise "Multiple HTTP implementations were found on the
classpath".
- The STS provider uses the synchronous `StsClient` (Apache HTTP client), so
the async `netty-nio-client` on the classpath is never loaded.
To keep this invariant, `connector/credential-aws/pom.xml` carries a comment
requiring the AWS SDK v2 version to stay pinned to `${aws.java.sdk.v2.version}`
(the single source of truth in the root POM). A divergent version would place
duplicate classes at different versions on the classpath and risk runtime
`NoSuchMethodError` / `LinkageError`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Verified with both sbt and Maven that the module JAR is placed into
`assembly/target/scala-2.13/jars/` when the profile is active, and is absent
otherwise.
sbt:
```bash
# With profile: spark-credential-aws_2.13-*.jar is copied
build/sbt -Pcredential-aws "assembly/package" "assembly/copyDeps"
ls assembly/target/scala-2.13/jars/ | grep credential-aws
# => spark-credential-aws_2.13-5.0.0-SNAPSHOT.jar
# Confirmed via dependencyClasspath that the profile adds the module:
build/sbt -Pcredential-aws 'show assembly/Compile/dependencyClasspath'
# =>
.../connector/credential-aws/target/scala-2.13/spark-credential-aws_2.13-5.0.0-SNAPSHOT.jar
# Without profile: JAR is not (re)created in a clean jars dir
build/sbt "assembly/package"
# => absent
```
Maven:
```bash
build/mvn -Pcredential-aws -pl assembly -am -DskipTests package
# BUILD SUCCESS (reactor includes "Spark AWS Credential Provider" and
# "Spark Project Assembly")
ls assembly/target/scala-2.13/jars/ | grep credential-aws
# => spark-credential-aws_2.13-5.0.0-SNAPSHOT.jar
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Claude
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
