This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 592def6a93ea [SPARK-57897][BUILD][FOLLOWUP] Add `credential-aws`
profile to `assembly/pom.xml`
592def6a93ea is described below
commit 592def6a93eaa45c0a157e3ed7227e95d9a4ef2b
Author: Kousuke Saruta <[email protected]>
AuthorDate: Thu Aug 20 06:02:42 2026 +0900
[SPARK-57897][BUILD][FOLLOWUP] Add `credential-aws` profile to
`assembly/pom.xml`
### What changes were proposed in this pull request?
Add a `credential-aws` profile to `assembly/pom.xml`, following the same
pattern as the existing `hadoop-cloud` profile:
```xml
<profile>
<id>credential-aws</id>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-credential-aws_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</profile>
```
This is a follow-up to SPARK-57897, which created the
`connector/credential-aws` module and wired the `-Pcredential-aws` profile into
the root POM, `SparkBuild.scala`, and the CI/release scripts, but did not add
the module to `assembly/pom.xml`.
Because the assembly now pulls in `spark-credential-aws` (and its
transitive AWS SDK v2 dependencies) when `-Pcredential-aws` is active, the
dependency manifest checked by `dev/test-dependencies.sh` is regenerated. This
adds the
AWS SDK v2 artifacts (`sts` and its transitive deps, version 2.35.4) to
`dev/deps/spark-deps-hadoop-3-hive-2.3`, consistent with how `hadoop-cloud`
transitive deps (`hadoop-aws`, `hadoop-cloud-storage`, `wildfly-openssl`,
etc.) are already listed there. `dev/test-dependencies.sh` includes
`-Pcredential-aws` in its profile list, so it fails until the manifest is
updated.
### Why are the changes needed?
Without an entry in `assembly/pom.xml`, building with `-Pcredential-aws`
does not place `spark-credential-aws_*.jar` into
`assembly/target/scala-*/jars/`. Both `dev/make-distribution.sh` and
`bin/docker-image-tool.sh` source their JARs from that directory, so the module
could not be included in a custom distribution or a custom Docker image even
when the profile was explicitly requested.
`hadoop-cloud` is an analogous optional module and already has an assembly
profile, so a user can opt in with `-Phadoop-cloud`. `credential-aws` should
behave the same way.
Note: this is an opt-in path only. The official binary distributions are
built without `-Pcredential-aws` (`BINARY_PKGS_ARGS["hadoop3"]="-Phadoop-3
$HIVE_PROFILES"` in `dev/create-release/release-build.sh`), so the module is
still excluded from the default distribution and the default Docker image,
consistent with the design intent of keeping the AWS SDK dependencies out of
the core distribution. The module continues to be published to Maven Central
as a standalone artifact (via `PUBLISH_PROFILES`) for consumption with
`--packages`.
### Interaction with the hadoop-cloud profile
The E2E test (Minikube + LocalStack, SPARK-57900) exercises the full flow
`token -> STS -> S3A read/write`. The S3A read/write half requires
`hadoop-aws`, which is only available at runtime via the `hadoop-cloud` profile
(`connector/credential-aws` itself does not depend on `hadoop-aws`; it only
returns `fs.s3a.*` credential properties). So the E2E image is expected to be
built with both `-Pcredential-aws -Phadoop-cloud`.
When both profiles are active, the assembly contains this module's
individual `sts` jar (plus its transitive AWS SDK v2 deps) as well as
hadoop-cloud's `software.amazon.awssdk:bundle` jar. `bundle` is a fat jar that
already contains the same `software.amazon.awssdk.*` classes (e.g.
`StsClient`), so those classes are duplicated on the classpath. This was
verified to be harmless as long as both resolve to the same version:
- The duplicated classes are byte-identical (same version 2.35.4), so
whichever the classloader picks behaves identically. No `LinkageError`.
- The synchronous `SdkHttpService` implementation registered by both
`bundle` and `apache-client` is the single class `ApacheSdkHttpService`, so the
AWS SDK does not raise "Multiple HTTP implementations were found on the
classpath".
- The STS provider uses the synchronous `StsClient` (Apache HTTP client),
so the async `netty-nio-client` on the classpath is never loaded.
To keep this invariant, `connector/credential-aws/pom.xml` carries a
comment requiring the AWS SDK v2 version to stay pinned to
`${aws.java.sdk.v2.version}` (the single source of truth in the root POM). A
divergent version would place duplicate classes at different versions on the
classpath and risk runtime `NoSuchMethodError` / `LinkageError`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Verified with both sbt and Maven that the module JAR is placed into
`assembly/target/scala-2.13/jars/` when the profile is active, and is absent
otherwise.
sbt:
```bash
# With profile: spark-credential-aws_2.13-*.jar is copied
build/sbt -Pcredential-aws "assembly/package" "assembly/copyDeps"
ls assembly/target/scala-2.13/jars/ | grep credential-aws
# => spark-credential-aws_2.13-5.0.0-SNAPSHOT.jar
# Confirmed via dependencyClasspath that the profile adds the module:
build/sbt -Pcredential-aws 'show assembly/Compile/dependencyClasspath'
# =>
.../connector/credential-aws/target/scala-2.13/spark-credential-aws_2.13-5.0.0-SNAPSHOT.jar
# Without profile: JAR is not (re)created in a clean jars dir
build/sbt "assembly/package"
# => absent
```
Maven:
```bash
build/mvn -Pcredential-aws -pl assembly -am -DskipTests package
# BUILD SUCCESS (reactor includes "Spark AWS Credential Provider" and
# "Spark Project Assembly")
ls assembly/target/scala-2.13/jars/ | grep credential-aws
# => spark-credential-aws_2.13-5.0.0-SNAPSHOT.jar
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Claude
Closes #58051 from sarutak/oidc-propagation/assembly-credential-aws-profile.
Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Kousuke Saruta <[email protected]>
---
assembly/pom.xml | 14 ++++++++++++++
connector/credential-aws/pom.xml | 9 +++++++++
dev/deps/spark-deps-hadoop-3-hive-2.3 | 30 +++++++++++++++++++++++++++++-
3 files changed, 52 insertions(+), 1 deletion(-)
diff --git a/assembly/pom.xml b/assembly/pom.xml
index ade100e3bcd4..b1d00776bdf6 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -371,5 +371,19 @@
</dependency>
</dependencies>
</profile>
+
+ <!--
+ Pull in spark-credential-aws and its associated JARs,
+ -->
+ <profile>
+ <id>credential-aws</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-credential-aws_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+ </profile>
</profiles>
</project>
diff --git a/connector/credential-aws/pom.xml b/connector/credential-aws/pom.xml
index e32666347659..552a481605c2 100644
--- a/connector/credential-aws/pom.xml
+++ b/connector/credential-aws/pom.xml
@@ -48,6 +48,15 @@
<groupId>org.apache.spark</groupId>
<artifactId>spark-tags_${scala.binary.version}</artifactId>
</dependency>
+ <!--
+ Keep pinned to ${aws.java.sdk.v2.version}. When this module is built
into the
+ assembly alongside the hadoop-cloud profile, this `sts` jar co-exists
with
+ hadoop-cloud's `bundle` jar and they share the same
software.amazon.awssdk.*
+ classes (e.g. StsClient). This is safe only because both resolve to the
same
+ version, making the duplicated classes byte-identical. A divergent
version here
+ would place duplicate classes at DIFFERENT versions on the classpath and
risk
+ NoSuchMethodError / LinkageError at runtime.
+ -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sts</artifactId>
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index f2d960131cd0..0543c84e1168 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -16,9 +16,11 @@ aliyun-java-sdk-kms/2.11.0//aliyun-java-sdk-kms-2.11.0.jar
aliyun-java-sdk-ram/3.1.0//aliyun-java-sdk-ram-3.1.0.jar
aliyun-sdk-oss/3.18.1//aliyun-sdk-oss-3.18.1.jar
analyticsaccelerator-s3/1.3.1//analyticsaccelerator-s3-1.3.1.jar
+annotations/2.35.4//annotations-2.35.4.jar
antlr-runtime/3.5.2//antlr-runtime-3.5.2.jar
antlr4-runtime/4.13.1//antlr4-runtime-4.13.1.jar
aopalliance-repackaged/3.0.6//aopalliance-repackaged-3.0.6.jar
+apache-client/2.35.4//apache-client-2.35.4.jar
arpack/3.2.0//arpack-3.2.0.jar
arpack_combined_all/0.1//arpack_combined_all-0.1.jar
arrow-compression/19.0.0//arrow-compression-19.0.0.jar
@@ -28,9 +30,12 @@
arrow-memory-netty-buffer-patch/19.0.0//arrow-memory-netty-buffer-patch-19.0.0.j
arrow-memory-netty/19.0.0//arrow-memory-netty-19.0.0.jar
arrow-vector/19.0.0//arrow-vector-19.0.0.jar
audience-annotations/0.12.0//audience-annotations-0.12.0.jar
+auth/2.35.4//auth-2.35.4.jar
avro-ipc/1.12.1//avro-ipc-1.12.1.jar
avro-mapred/1.12.1//avro-mapred-1.12.1.jar
avro/1.12.1//avro-1.12.1.jar
+aws-core/2.35.4//aws-core-2.35.4.jar
+aws-query-protocol/2.35.4//aws-query-protocol-2.35.4.jar
azure-data-lake-store-sdk/2.3.9//azure-data-lake-store-sdk-2.3.9.jar
azure-keyvault-core/1.0.0//azure-keyvault-core-1.0.0.jar
azure-storage/7.0.1//azure-storage-7.0.1.jar
@@ -39,6 +44,8 @@ breeze-macros_2.13/2.1.0//breeze-macros_2.13-2.1.0.jar
breeze_2.13/2.1.0//breeze_2.13-2.1.0.jar
bundle/2.35.4//bundle-2.35.4.jar
cats-kernel_2.13/2.8.0//cats-kernel_2.13-2.8.0.jar
+checksums-spi/2.35.4//checksums-spi-2.35.4.jar
+checksums/2.35.4//checksums-2.35.4.jar
chill-java/0.10.0//chill-java-0.10.0.jar
chill_2.13/0.10.0//chill_2.13-0.10.0.jar
commons-cli/1.11.0//commons-cli-1.11.0.jar
@@ -68,6 +75,8 @@ derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar
derbytools/10.16.1.1//derbytools-10.16.1.1.jar
dom4j/2.1.4//dom4j-2.1.4.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+endpoints-spi/2.35.4//endpoints-spi-2.35.4.jar
+eventstream/1.0.1//eventstream-1.0.1.jar
failureaccess/1.0.3//failureaccess-1.0.3.jar
flatbuffers-java/25.2.10//flatbuffers-java-25.2.10.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
@@ -99,9 +108,15 @@ hive-storage-api/2.8.1//hive-storage-api-2.8.1.jar
hk2-api/3.0.6//hk2-api-3.0.6.jar
hk2-locator/3.0.6//hk2-locator-3.0.6.jar
hk2-utils/3.0.6//hk2-utils-3.0.6.jar
+http-auth-aws-eventstream/2.35.4//http-auth-aws-eventstream-2.35.4.jar
+http-auth-aws/2.35.4//http-auth-aws-2.35.4.jar
+http-auth-spi/2.35.4//http-auth-spi-2.35.4.jar
+http-auth/2.35.4//http-auth-2.35.4.jar
+http-client-spi/2.35.4//http-client-spi-2.35.4.jar
httpclient/4.5.14//httpclient-4.5.14.jar
httpcore/4.4.16//httpcore-4.4.16.jar
icu4j/78.3//icu4j-78.3.jar
+identity-spi/2.35.4//identity-spi-2.35.4.jar
ini4j/0.5.4//ini4j-0.5.4.jar
istack-commons-runtime/4.1.2//istack-commons-runtime-4.1.2.jar
ivy/2.5.3//ivy-2.5.3.jar
@@ -145,6 +160,7 @@ jline/2.14.6//jline-2.14.6.jar
jline/3.29.0/jdk8/jline-3.29.0-jdk8.jar
joda-time/2.14.3//joda-time-2.14.3.jar
jpam/1.1//jpam-1.1.jar
+json-utils/2.35.4//json-utils-2.35.4.jar
json/1.8//json-1.8.jar
json4s-ast_2.13/4.0.7//json4s-ast_2.13-4.0.7.jar
json4s-core_2.13/4.0.7//json4s-core_2.13-4.0.7.jar
@@ -196,6 +212,7 @@ metrics-graphite/4.2.37//metrics-graphite-4.2.37.jar
metrics-jmx/4.2.37//metrics-jmx-4.2.37.jar
metrics-json/4.2.37//metrics-json-4.2.37.jar
metrics-jvm/4.2.37//metrics-jvm-4.2.37.jar
+metrics-spi/2.35.4//metrics-spi-2.35.4.jar
minlog/1.3.0//minlog-1.3.0.jar
netty-all/4.2.17.Final//netty-all-4.2.17.Final.jar
netty-buffer/4.2.17.Final//netty-buffer-4.2.17.Final.jar
@@ -209,6 +226,7 @@ netty-codec/4.2.17.Final//netty-codec-4.2.17.Final.jar
netty-common/4.2.17.Final//netty-common-4.2.17.Final.jar
netty-handler-proxy/4.2.17.Final//netty-handler-proxy-4.2.17.Final.jar
netty-handler/4.2.17.Final//netty-handler-4.2.17.Final.jar
+netty-nio-client/2.35.4//netty-nio-client-2.35.4.jar
netty-resolver-dns/4.2.17.Final//netty-resolver-dns-4.2.17.Final.jar
netty-resolver/4.2.17.Final//netty-resolver-4.2.17.Final.jar
netty-tcnative-boringssl-static/2.0.81.Final/linux-aarch_64/netty-tcnative-boringssl-static-2.0.81.Final-linux-aarch_64.jar
@@ -247,9 +265,14 @@
parquet-format-structures/1.17.1//parquet-format-structures-1.17.1.jar
parquet-hadoop/1.17.1//parquet-hadoop-1.17.1.jar
parquet-jackson/1.17.1//parquet-jackson-1.17.1.jar
pickle/1.5//pickle-1.5.jar
+profiles/2.35.4//profiles-2.35.4.jar
+protocol-core/2.35.4//protocol-core-2.35.4.jar
py4j/0.10.9.9//py4j-0.10.9.9.jar
-reactive-streams/1.0.3//reactive-streams-1.0.3.jar
+reactive-streams/1.0.4//reactive-streams-1.0.4.jar
+regions/2.35.4//regions-2.35.4.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
+retries-spi/2.35.4//retries-spi-2.35.4.jar
+retries/2.35.4//retries-2.35.4.jar
rocksdbjni/10.10.1.1//rocksdbjni-10.10.1.1.jar
scala-compiler/2.13.18//scala-compiler-2.13.18.jar
scala-library/2.13.18//scala-library-2.13.18.jar
@@ -257,6 +280,7 @@
scala-parallel-collections_2.13/1.2.0//scala-parallel-collections_2.13-1.2.0.jar
scala-parser-combinators_2.13/2.4.0//scala-parser-combinators_2.13-2.4.0.jar
scala-reflect/2.13.18//scala-reflect-2.13.18.jar
scala-xml_2.13/2.4.0//scala-xml_2.13-2.4.0.jar
+sdk-core/2.35.4//sdk-core-2.35.4.jar
slf4j-api/2.0.17//slf4j-api-2.0.17.jar
snakeyaml-engine/3.0.1//snakeyaml-engine-3.0.1.jar
snakeyaml/2.5//snakeyaml-2.5.jar
@@ -266,11 +290,15 @@ spire-platform_2.13/0.18.0//spire-platform_2.13-0.18.0.jar
spire-util_2.13/0.18.0//spire-util_2.13-0.18.0.jar
spire_2.13/0.18.0//spire_2.13-0.18.0.jar
stream/2.9.8//stream-2.9.8.jar
+sts/2.35.4//sts-2.35.4.jar
super-csv/2.2.0//super-csv-2.2.0.jar
+third-party-jackson-core/2.35.4//third-party-jackson-core-2.35.4.jar
threeten-extra/1.9.0//threeten-extra-1.9.0.jar
tink/1.23.0//tink-1.23.0.jar
transaction-api/1.1//transaction-api-1.1.jar
univocity-parsers/2.9.1//univocity-parsers-2.9.1.jar
+utils-lite/2.35.4//utils-lite-2.35.4.jar
+utils/2.35.4//utils-2.35.4.jar
vertx-auth-common/4.5.28//vertx-auth-common-4.5.28.jar
vertx-core/4.5.28//vertx-core-4.5.28.jar
vertx-uri-template/4.5.28//vertx-uri-template-4.5.28.jar
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
