ruanwenjun opened a new pull request, #59000: URL: https://github.com/apache/spark/pull/59000
### What changes were proposed in this pull request? Use `software.amazon.awssdk:bundle` instead of `sts` in `connector/credential-aws`, sharing the SDK artifact and version already used by `hadoop-cloud`. Update the dependency comment and regenerate the assembly dependency manifest. Removing the modular SDK also lets the existing Aliyun dependency select `reactive-streams` 1.0.3 instead of 1.0.4. This follows up on #58051 under [SPARK-57897](https://issues.apache.org/jira/browse/SPARK-57897). ### Why are the changes needed? With both `-Pcredential-aws` and `-Phadoop-cloud`, the assembly currently contains the AWS SDK bundle alongside modular SDK jars. The bundle relocates third-party dependencies, so duplicate SDK classes are not binary-compatible even when all artifacts use version 2.35.4. Which jar supplies a duplicated class can determine whether S3A uploads succeed. For example, the bundle's `TransferListenerInvoker` calls `Logger.logger()` with the relocated SLF4J return type. Loading `Logger` from the modular `utils` jar instead causes: ```text java.lang.NoSuchMethodError: 'software.amazon.awssdk.thirdparty.org.slf4j.Logger software.amazon.awssdk.utils.Logger.logger()' ``` This is the failure seen in S3 upload tests in `KubernetesSuite` and `VolcanoSuite`, including [this CI job](https://github.com/ruanwenjun/spark/actions/runs/35857041511/job/107174192748). The mixed dependency set was introduced into the assembly by #58051. Using the bundle for both modules removes that duplicate class source without changing the AWS SDK version or relying on classpath order. ### Does this PR introduce _any_ user-facing change? Yes. It fixes S3A linkage failures in builds that include both optional profiles. Builds using only `credential-aws` now pull in the larger SDK bundle instead of the modular STS dependency. ### How was this patch tested? - `build/sbt -Pcredential-aws -Phadoop-cloud 'credential-aws/test'` and `build/sbt -Pcredential-aws 'credential-aws/test'`: all 69 existing tests passed in each configuration. - Reproduced the exact `NoSuchMethodError` with AWS SDK 2.35.4 modular `utils` before `bundle` on the classpath. The same transfer-listener call passes with only `bundle`: ```java new software.amazon.awssdk.transfer.s3.internal.progress.TransferListenerInvoker( java.util.List.of(new software.amazon.awssdk.transfer.s3.progress.TransferListener() {})) .transferInitiated(null); ``` - `dev/test-dependencies.sh --replace-manifest` completed successfully (offline after populating the Maven cache). - `build/sbt -Pcredential-aws -Phadoop-cloud 'show assembly/Compile/externalDependencyClasspath'`: among 259 external jars, `bundle-2.35.4.jar` is the only AWS SDK artifact. `Logger`, `StsClient`, and `TransferListenerInvoker` each have exactly one class source. - The reproducer also passed on that actual assembly classpath in both the original and reversed jar order. - `git diff --check` passed. No new mock-based test was added: the regression concerns the distribution classpath, and existing S3 upload integration tests exercise the failing code path. Full Kubernetes and OIDC-to-STS-to-S3A E2E tests have not been run locally because Docker is unavailable. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
