This is an automated email from the ASF dual-hosted git repository.

sarutak pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new e9b8b1b13016 [SPARK-57897][BUILD][CORE] Create 
`connector/credential-aws` module with CI/release integration
e9b8b1b13016 is described below

commit e9b8b1b13016d6aba5e77227013632876c1f42f7
Author: Kousuke Saruta <[email protected]>
AuthorDate: Sat Aug 1 18:22:54 2026 +0900

    [SPARK-57897][BUILD][CORE] Create `connector/credential-aws` module with 
CI/release integration
    
    ### What changes were proposed in this pull request?
    This PR creates the `connector/credential-aws` module scaffolding as a part 
of the OIDC Credential Propagation SPIP (SPARK-57703).
    
    **Module definition:**
    - `connector/credential-aws/pom.xml` (parent: spark-parent_2.13, deps: 
spark-core provided, spark-tags, junit-jupiter test)
    - Root `pom.xml`: `<profile><id>credential-aws</id>` with 
`<module>connector/credential-aws</module>`
    - `project/SparkBuild.scala`: added `credentialAws` to 
`optionallyEnabledProjects` and MiMa exclusion list
    - `dev/sparktestsupport/modules.py`: `credential_aws` Module definition
    - `dev/sparktestsupport/utils.py`: doctest expected output update
    
    **ServiceLoader registration and stub provider:**
    - `META-INF/services/org.apache.spark.security.CredentialProvider` pointing 
to `AwsStsCredentialProvider`
    - Stub `AwsStsCredentialProvider` — minimal implementation for 
ServiceLoader discovery testing. This class will be replaced by the full 
implementation in SPARK-57898 (Sub-task 9). The stub is included here to verify 
that the ServiceLoader wiring, test framework setup, and CI profile activation 
work end-to-end.
    
    **CI/release/lint script updates (`-Pcredential-aws` added in 22 
locations):**
    - `.github/workflows/build_and_test.yml` (is-changed.py module list, SBT 
build, Maven build)
    - `.github/workflows/maven_test.yml` (all build/test commands)
    - `.github/workflows/python_hosted_runner_test.yml`
    - `.github/workflows/benchmark.yml`
    - `dev/create-release/release-build.sh`
    - `dev/test-dependencies.sh`
    - `dev/mima`
    - `dev/lint-java`
    - `dev/sbt-checkstyle`
    - `dev/scalastyle`
    - `docs/_plugins/build_api_docs.rb`
    - `dev/spark-test-image-util/docs/build-docs`
    
    **Note:** This PR establishes the build infrastructure only (stub 
provider). The actual provider implementation and its tests will follow in 
SPARK-57898, which will exercise the profile end-to-end (compilation, test 
execution, lint checks).
    
    ### Why are the changes needed?
    The AWS reference provider needs a separate module to keep AWS SDK 
dependencies out of Spark core. Splitting the module scaffolding from the 
implementation:
    
    1. Makes the CI/release integration independently reviewable (12 scripts, 
22 locations).
    2. Allows SPARK-57898 and SPARK-57899 to focus purely on the provider 
implementation and tests after rebasing onto this PR.
    
    ### Does this PR introduce _any_ user-facing change?
    No. The module is activated only via `-Pcredential-aws` (Maven) or by 
including `credential-aws` in SBT's `optionallyEnabledProjects`. It is not 
included in the default build. The stub provider throws 
`UnsupportedOperationException` on `init()` and `resolve()`.
    
    ### How was this patch tested?
    - `mvn -Pcredential-aws -pl connector/credential-aws test`: 5 tests pass 
(ServiceLoader discovery, supportedSchemes, init throws, resolve throws, 
suggestedTtl)
    - `build/sbt -Pcredential-aws "credential-aws/test"`: 5 tests pass
    - `python3 -m doctest dev/sparktestsupport/utils.py`: passes with updated 
expected output
    - `./dev/lint-java`: Checkstyle checks passed
    - `./dev/mima`: success
    - `./dev/scalastyle`: Scalastyle checks passed
    - `mvn -Pcredential-aws -pl connector/credential-aws apache-rat:check`: all 
source files have valid license headers
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Kiro CLI / Claude
    
    Closes #57679 from 
sarutak/oidc-propagation/subtask8-credential-aws-scaffolding.
    
    Authored-by: Kousuke Saruta <[email protected]>
    Signed-off-by: Kousuke Saruta <[email protected]>
    (cherry picked from commit cc19d868c3b563b640715b8c36fd528d13541317)
    Signed-off-by: Kousuke Saruta <[email protected]>
---
 .github/workflows/benchmark.yml                    |  2 +-
 .github/workflows/build_and_test.yml               |  8 +--
 .github/workflows/maven_test.yml                   | 20 +++---
 .github/workflows/python_hosted_runner_test.yml    |  2 +-
 connector/credential-aws/pom.xml                   | 74 ++++++++++++++++++++++
 .../security/aws/AwsStsCredentialProvider.java     | 57 +++++++++++++++++
 .../org.apache.spark.security.CredentialProvider   | 14 +---
 .../aws/AwsStsCredentialProviderSuite.java         | 74 ++++++++++++++++++++++
 dev/create-release/release-build.sh                |  2 +-
 dev/lint-java                                      |  2 +-
 dev/mima                                           |  2 +-
 dev/sbt-checkstyle                                 |  2 +-
 dev/scalastyle                                     |  2 +-
 dev/spark-test-image-util/docs/build-docs          |  2 +-
 dev/sparktestsupport/modules.py                    | 15 +++++
 dev/sparktestsupport/utils.py                      |  3 +-
 dev/test-dependencies.sh                           |  2 +-
 docs/_plugins/build_api_docs.rb                    |  4 +-
 pom.xml                                            |  7 ++
 project/SparkBuild.scala                           |  7 +-
 20 files changed, 259 insertions(+), 42 deletions(-)

diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml
index 4e10641cd13a..a61e1f416bb4 100644
--- a/.github/workflows/benchmark.yml
+++ b/.github/workflows/benchmark.yml
@@ -214,7 +214,7 @@ jobs:
         key: tpcds-${{ hashFiles('.github/workflows/benchmark.yml', 
'sql/core/src/test/scala/org/apache/spark/sql/TPCDSSchema.scala') }}
     - name: Run benchmarks
       run: |
-        ./build/sbt -Pscala-${{ inputs.scala }} -Pyarn -Pkubernetes -Phive 
-Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pspark-ganglia-lgpl 
Test/package
+        ./build/sbt -Pscala-${{ inputs.scala }} -Pyarn -Pkubernetes -Phive 
-Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pcredential-aws 
-Pspark-ganglia-lgpl Test/package
         # Make less noisy
         cp conf/log4j2.properties.template conf/log4j2.properties
         sed -i 's/rootLogger.level = info/rootLogger.level = warn/g' 
conf/log4j2.properties
diff --git a/.github/workflows/build_and_test.yml 
b/.github/workflows/build_and_test.yml
index 3c0bffa1a055..87cf3a9c51ff 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -149,7 +149,7 @@ jobs:
             java17=false
             java25=false
           fi
-          build=`./dev/is-changed.py -m 
"core,unsafe,kvstore,avro,utils,utils-java,network-common,network-shuffle,repl,launcher,examples,sketch,variant,api,catalyst,hive-thriftserver,mllib-local,mllib,graphx,streaming,sql-kafka-0-10,streaming-kafka-0-10,streaming-kinesis-asl,kubernetes,hadoop-cloud,spark-ganglia-lgpl,profiler,protobuf,yarn,connect,sql,hive,pipelines"`
+          build=`./dev/is-changed.py -m 
"core,unsafe,kvstore,avro,utils,utils-java,network-common,network-shuffle,repl,launcher,examples,sketch,variant,api,catalyst,hive-thriftserver,mllib-local,mllib,graphx,streaming,sql-kafka-0-10,streaming-kafka-0-10,streaming-kinesis-asl,credential-aws,kubernetes,hadoop-cloud,spark-ganglia-lgpl,profiler,protobuf,yarn,connect,sql,hive,pipelines"`
           build_core_utils=`./dev/is-changed.py -m 
"core,unsafe,kvstore,utils,utils-java,network-common,network-shuffle,sketch,variant,launcher"`
           precondition="
             {
@@ -276,7 +276,7 @@ jobs:
             mllib-local, mllib, graphx, profiler, pipelines, repl, examples
           - >-
             streaming, sql-kafka-0-10, streaming-kafka-0-10, 
streaming-kinesis-asl,
-            kubernetes, hadoop-cloud, spark-ganglia-lgpl, protobuf, connect, 
avro
+            credential-aws, kubernetes, hadoop-cloud, spark-ganglia-lgpl, 
protobuf, connect, avro
           - yarn
         # Here, we split Hive and SQL tests into some of slow ones and the 
rest of them.
         included-tags: [""]
@@ -598,7 +598,7 @@ jobs:
     - name: Build Spark
       run: |
         ./build/sbt -Phadoop-3 -Pyarn -Pspark-ganglia-lgpl -Phadoop-cloud 
-Phive \
-          -Pkubernetes -Pjvm-profiler -Pkinesis-asl -Phive-thriftserver \
+          -Pkubernetes -Pjvm-profiler -Pkinesis-asl -Pcredential-aws 
-Phive-thriftserver \
           -Pdocker-integration-tests -Pkubernetes-integration-tests -Pvolcano \
           Test/package streaming-kinesis-asl-assembly/assembly 
connect/assembly assembly/package
     - name: Package compile output
@@ -1165,7 +1165,7 @@ jobs:
       run: |
         export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g 
-XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
         export MAVEN_CLI_OPTS="--no-transfer-progress"
-        ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes -Pvolcano 
-Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl 
-Pkinesis-asl clean install
+        ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes -Pvolcano 
-Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl 
-Pkinesis-asl -Pcredential-aws clean install
 
   # Documentation build
   docs:
diff --git a/.github/workflows/maven_test.yml b/.github/workflows/maven_test.yml
index 5b0cc81a6510..a857cf048953 100644
--- a/.github/workflows/maven_test.yml
+++ b/.github/workflows/maven_test.yml
@@ -120,7 +120,7 @@ jobs:
           export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g 
-XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
           export MAVEN_CLI_OPTS="--no-transfer-progress"
           export JAVA_VERSION=${{ inputs.java }}
-          ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes 
-Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler 
-Pspark-ganglia-lgpl -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} clean 
install
+          ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes 
-Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler 
-Pspark-ganglia-lgpl -Pkinesis-asl -Pcredential-aws 
-Djava.version=${JAVA_VERSION/-ea} clean install
       - name: Package compile output
         run: |
           # Exclude assembly/ from the artifact: 11 of 12 matrix entries wipe 
it
@@ -171,7 +171,7 @@ jobs:
           - >-
             repl,sql#hive-thriftserver
           - >-
-            
connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro,connector#kinesis-asl
+            
connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro,connector#kinesis-asl,connector#credential-aws
           - >-
             
sql#api,sql#catalyst,resource-managers#yarn,resource-managers#kubernetes#core
           - >-
@@ -325,10 +325,10 @@ jobs:
             # it here.
             if [ "$MODULES_TO_TEST" = "connect" ]; then
               echo "Building assembly module for connect tests."
-              ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes 
-Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler 
-Pspark-ganglia-lgpl -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} -pl 
assembly install
+              ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes 
-Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler 
-Pspark-ganglia-lgpl -Pkinesis-asl -Pcredential-aws 
-Djava.version=${JAVA_VERSION/-ea} -pl assembly install
             fi
           else
-            ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes 
-Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler 
-Pspark-ganglia-lgpl -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} clean 
install
+            ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes 
-Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler 
-Pspark-ganglia-lgpl -Pkinesis-asl -Pcredential-aws 
-Djava.version=${JAVA_VERSION/-ea} clean install
             # SPARK-51628: wipe the assembly module so tests exercise the
             # SPARK-51600 prepend fallback path. Connect tests strongly depend
             # on a built assembly module, so they are excluded.
@@ -339,27 +339,27 @@ jobs:
           fi
 
           if [[ "$INCLUDED_TAGS" != "" ]]; then
-            ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn 
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler 
-Pspark-ganglia-lgpl -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} 
-Dtest.include.tags="$INCLUDED_TAGS" test -fae
+            ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn 
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler 
-Pspark-ganglia-lgpl -Pkinesis-asl -Pcredential-aws 
-Djava.version=${JAVA_VERSION/-ea} -Dtest.include.tags="$INCLUDED_TAGS" test 
-fae
           elif [[ "$MODULES_TO_TEST" == "connect" && "$INPUT_BRANCH" == 
"branch-4.0" ]]; then
             # SPARK-53914: Remove sql/connect/client/jdbc from `-pl` for 
branch-4.0, this branch can be deleted after the EOL of branch-4.0.
             ./build/mvn $MAVEN_CLI_OPTS -Djava.version=${JAVA_VERSION/-ea} -pl 
sql/connect/client/jvm,sql/connect/common,sql/connect/server test -fae
           elif [[ "$MODULES_TO_TEST" == "connect" ]]; then
             ./build/mvn $MAVEN_CLI_OPTS -Djava.version=${JAVA_VERSION/-ea} -pl 
sql/connect/client/jdbc,sql/connect/client/jvm,sql/connect/common,sql/connect/server
 test -fae
           elif [[ "$EXCLUDED_TAGS" != "" ]]; then
-            ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn 
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler 
-Pspark-ganglia-lgpl -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} 
-Dtest.exclude.tags="$EXCLUDED_TAGS" test -fae
+            ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn 
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler 
-Pspark-ganglia-lgpl -Pkinesis-asl -Pcredential-aws 
-Djava.version=${JAVA_VERSION/-ea} -Dtest.exclude.tags="$EXCLUDED_TAGS" test 
-fae
           elif [[ "$MODULES_TO_TEST" == *"sql#hive-thriftserver"* ]]; then
             # To avoid a compilation loop, for the `sql/hive-thriftserver` 
module, run `clean install` instead
-            ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn 
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler 
-Pspark-ganglia-lgpl -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} clean 
install -fae
+            ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn 
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler 
-Pspark-ganglia-lgpl -Pkinesis-asl -Pcredential-aws 
-Djava.version=${JAVA_VERSION/-ea} clean install -fae
           elif [[ "$MODULES_TO_TEST" == *"sql#pipelines"* && "$INPUT_BRANCH" 
== "branch-4.0" ]]; then
             # SPARK-52441: Remove sql/pipelines from TEST_MODULES for 
branch-4.0, this branch can be deleted after the EOL of branch-4.0.
             TEST_MODULES=${TEST_MODULES/,sql\/pipelines/}
-            ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn 
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Pspark-ganglia-lgpl 
-Phadoop-cloud -Pjvm-profiler -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} 
test -fae
+            ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn 
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Pspark-ganglia-lgpl 
-Phadoop-cloud -Pjvm-profiler -Pkinesis-asl -Pcredential-aws 
-Djava.version=${JAVA_VERSION/-ea} test -fae
           elif [[ "$MODULES_TO_TEST" == *"common#utils-java"* && 
"$INPUT_BRANCH" == "branch-4.0" ]]; then
             # SPARK-53138: Remove common/utils-java from TEST_MODULES for 
branch-4.0, this branch can be deleted after the EOL of branch-4.0.
             TEST_MODULES=${TEST_MODULES/,common\/utils-java/}
-            ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn 
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Pspark-ganglia-lgpl 
-Phadoop-cloud -Pjvm-profiler -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} 
test -fae
+            ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn 
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Pspark-ganglia-lgpl 
-Phadoop-cloud -Pjvm-profiler -Pkinesis-asl -Pcredential-aws 
-Djava.version=${JAVA_VERSION/-ea} test -fae
           else
-            ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn 
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Pspark-ganglia-lgpl 
-Phadoop-cloud -Pjvm-profiler -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} 
test -fae
+            ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn 
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Pspark-ganglia-lgpl 
-Phadoop-cloud -Pjvm-profiler -Pkinesis-asl -Pcredential-aws 
-Djava.version=${JAVA_VERSION/-ea} test -fae
           fi
       - name: Clean up local Maven repository
         run: |  
diff --git a/.github/workflows/python_hosted_runner_test.yml 
b/.github/workflows/python_hosted_runner_test.yml
index 737032c5f5e5..f7e7ded71ce0 100644
--- a/.github/workflows/python_hosted_runner_test.yml
+++ b/.github/workflows/python_hosted_runner_test.yml
@@ -119,7 +119,7 @@ jobs:
       - name: Build Spark
         run: |
           ./build/sbt -Phadoop-3 -Pyarn -Pspark-ganglia-lgpl -Phadoop-cloud 
-Phive \
-            -Pkubernetes -Pjvm-profiler -Pkinesis-asl -Phive-thriftserver \
+            -Pkubernetes -Pjvm-profiler -Pkinesis-asl -Pcredential-aws 
-Phive-thriftserver \
             -Pdocker-integration-tests -Pvolcano \
             Test/package streaming-kinesis-asl-assembly/assembly 
connect/assembly assembly/package
       - name: Package compile output
diff --git a/connector/credential-aws/pom.xml b/connector/credential-aws/pom.xml
new file mode 100644
index 000000000000..b1cdf5c58f37
--- /dev/null
+++ b/connector/credential-aws/pom.xml
@@ -0,0 +1,74 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~    http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+  <modelVersion>4.0.0</modelVersion>
+  <parent>
+    <groupId>org.apache.spark</groupId>
+    <artifactId>spark-parent_2.13</artifactId>
+    <version>5.0.0-SNAPSHOT</version>
+    <relativePath>../../pom.xml</relativePath>
+  </parent>
+
+  <artifactId>spark-credential-aws_2.13</artifactId>
+  <properties>
+    <sbt.project.name>credential-aws</sbt.project.name>
+  </properties>
+  <packaging>jar</packaging>
+  <name>Spark AWS Credential Provider</name>
+  <description>
+    OIDC-to-AWS credential provider that exchanges identity tokens for
+    temporary AWS credentials via STS AssumeRoleWithWebIdentity.
+  </description>
+  <url>https://spark.apache.org/</url>
+
+  <dependencies>
+    <dependency>
+      <groupId>org.apache.spark</groupId>
+      <artifactId>spark-core_${scala.binary.version}</artifactId>
+      <version>${project.version}</version>
+      <scope>provided</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.spark</groupId>
+      <artifactId>spark-tags_${scala.binary.version}</artifactId>
+    </dependency>
+
+    <!-- Test dependencies -->
+    <!--
+      This spark-tags test-dep is needed even though it isn't used in this 
module, otherwise testing-cmds that exclude
+      them will yield errors.
+    -->
+    <dependency>
+      <groupId>org.apache.spark</groupId>
+      <artifactId>spark-tags_${scala.binary.version}</artifactId>
+      <type>test-jar</type>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.junit.jupiter</groupId>
+      <artifactId>junit-jupiter</artifactId>
+      <scope>test</scope>
+    </dependency>
+  </dependencies>
+
+  <build>
+    
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
+    
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
+  </build>
+</project>
diff --git 
a/connector/credential-aws/src/main/java/org/apache/spark/security/aws/AwsStsCredentialProvider.java
 
b/connector/credential-aws/src/main/java/org/apache/spark/security/aws/AwsStsCredentialProvider.java
new file mode 100644
index 000000000000..07d8821b6569
--- /dev/null
+++ 
b/connector/credential-aws/src/main/java/org/apache/spark/security/aws/AwsStsCredentialProvider.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.security.aws;
+
+import java.net.URI;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.spark.annotation.DeveloperApi;
+import org.apache.spark.security.CredentialProvider;
+import org.apache.spark.security.CredentialResolutionException;
+import org.apache.spark.security.ServiceCredential;
+import org.apache.spark.security.UserContext;
+
+/**
+ * :: DeveloperApi ::
+ * AWS STS credential provider that exchanges an OIDC identity token for 
temporary
+ * AWS credentials via {@code AssumeRoleWithWebIdentity}.
+ *
+ * @since 4.3.0
+ */
+@DeveloperApi
+public class AwsStsCredentialProvider implements CredentialProvider {
+
+  @Override
+  public void init(Map<String, String> conf) {
+    throw new UnsupportedOperationException(
+        "AwsStsCredentialProvider is a stub. Full implementation in 
SPARK-57898.");
+  }
+
+  @Override
+  public Set<String> supportedSchemes() {
+    return Set.of("s3a");
+  }
+
+  @Override
+  public ServiceCredential resolve(UserContext user, URI target)
+      throws CredentialResolutionException {
+    throw new UnsupportedOperationException(
+        "AwsStsCredentialProvider is a stub. Full implementation in 
SPARK-57898.");
+  }
+}
diff --git a/dev/lint-java 
b/connector/credential-aws/src/main/resources/META-INF/services/org.apache.spark.security.CredentialProvider
old mode 100755
new mode 100644
similarity index 65%
copy from dev/lint-java
copy to 
connector/credential-aws/src/main/resources/META-INF/services/org.apache.spark.security.CredentialProvider
index ff431301773f..84b31f53ec26
--- a/dev/lint-java
+++ 
b/connector/credential-aws/src/main/resources/META-INF/services/org.apache.spark.security.CredentialProvider
@@ -1,5 +1,3 @@
-#!/usr/bin/env bash
-
 #
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
@@ -17,14 +15,4 @@
 # limitations under the License.
 #
 
-SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
-SPARK_ROOT_DIR="$(dirname $SCRIPT_DIR)"
-
-ERRORS=$($SCRIPT_DIR/../build/mvn -Pkinesis-asl -Pspark-ganglia-lgpl 
-Pkubernetes -Pyarn -Phive -Phive-thriftserver checkstyle:check | grep ERROR)
-
-if test ! -z "$ERRORS"; then
-    echo -e "Checkstyle checks failed at following occurrences:\n$ERRORS"
-    exit 1
-else
-    echo -e "Checkstyle checks passed."
-fi
+org.apache.spark.security.aws.AwsStsCredentialProvider
diff --git 
a/connector/credential-aws/src/test/java/org/apache/spark/security/aws/AwsStsCredentialProviderSuite.java
 
b/connector/credential-aws/src/test/java/org/apache/spark/security/aws/AwsStsCredentialProviderSuite.java
new file mode 100644
index 000000000000..cab91fcae636
--- /dev/null
+++ 
b/connector/credential-aws/src/test/java/org/apache/spark/security/aws/AwsStsCredentialProviderSuite.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.security.aws;
+
+import java.net.URI;
+import java.time.Duration;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.*;
+
+import org.apache.spark.security.CredentialProvider;
+
+/**
+ * Tests that {@link AwsStsCredentialProvider} is discoverable via {@link 
ServiceLoader}.
+ */
+public class AwsStsCredentialProviderSuite {
+
+  @Test
+  void testServiceLoaderDiscovery() {
+    ServiceLoader<CredentialProvider> loader = 
ServiceLoader.load(CredentialProvider.class);
+    boolean found = false;
+    for (CredentialProvider provider : loader) {
+      if (provider instanceof AwsStsCredentialProvider) {
+        found = true;
+        break;
+      }
+    }
+    assertTrue(found, "AwsStsCredentialProvider should be discoverable via 
ServiceLoader");
+  }
+
+  @Test
+  void testSupportedSchemes() {
+    AwsStsCredentialProvider provider = new AwsStsCredentialProvider();
+    Set<String> schemes = provider.supportedSchemes();
+    assertEquals(Set.of("s3a"), schemes);
+  }
+
+  @Test
+  void testInitThrowsUnsupported() {
+    AwsStsCredentialProvider provider = new AwsStsCredentialProvider();
+    assertThrows(UnsupportedOperationException.class, () -> 
provider.init(Map.of()));
+  }
+
+  @Test
+  void testResolveThrowsUnsupported() {
+    AwsStsCredentialProvider provider = new AwsStsCredentialProvider();
+    assertThrows(UnsupportedOperationException.class,
+        () -> provider.resolve(null, URI.create("s3a://bucket/key")));
+  }
+
+  @Test
+  void testSuggestedTtl() {
+    AwsStsCredentialProvider provider = new AwsStsCredentialProvider();
+    assertEquals(Duration.ofMinutes(15), provider.suggestedTtl());
+  }
+}
diff --git a/dev/create-release/release-build.sh 
b/dev/create-release/release-build.sh
index 161dca83738b..63524f3ba7c1 100755
--- a/dev/create-release/release-build.sh
+++ b/dev/create-release/release-build.sh
@@ -683,7 +683,7 @@ SCALA_2_12_PROFILES="-Pscala-2.12"
 HIVE_PROFILES="-Phive -Phive-thriftserver"
 # Profiles for publishing snapshots and release to Maven Central
 # We use Apache Hive 2.3 for publishing
-PUBLISH_PROFILES="$BASE_PROFILES $HIVE_PROFILES -Pspark-ganglia-lgpl 
-Pkinesis-asl -Phadoop-cloud -Pjvm-profiler"
+PUBLISH_PROFILES="$BASE_PROFILES $HIVE_PROFILES -Pspark-ganglia-lgpl 
-Pkinesis-asl -Pcredential-aws -Phadoop-cloud -Pjvm-profiler"
 # Profiles for building binary releases
 BASE_RELEASE_PROFILES="$BASE_PROFILES -Psparkr"
 
diff --git a/dev/lint-java b/dev/lint-java
index ff431301773f..8095de68fb2a 100755
--- a/dev/lint-java
+++ b/dev/lint-java
@@ -20,7 +20,7 @@
 SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
 SPARK_ROOT_DIR="$(dirname $SCRIPT_DIR)"
 
-ERRORS=$($SCRIPT_DIR/../build/mvn -Pkinesis-asl -Pspark-ganglia-lgpl 
-Pkubernetes -Pyarn -Phive -Phive-thriftserver checkstyle:check | grep ERROR)
+ERRORS=$($SCRIPT_DIR/../build/mvn -Pkinesis-asl -Pcredential-aws 
-Pspark-ganglia-lgpl -Pkubernetes -Pyarn -Phive -Phive-thriftserver 
checkstyle:check | grep ERROR)
 
 if test ! -z "$ERRORS"; then
     echo -e "Checkstyle checks failed at following occurrences:\n$ERRORS"
diff --git a/dev/mima b/dev/mima
index 39be02a1dd55..eecf1ad1081d 100755
--- a/dev/mima
+++ b/dev/mima
@@ -24,7 +24,7 @@ set -e
 FWDIR="$(cd "`dirname "$0"`"/..; pwd)"
 cd "$FWDIR"
 
-SPARK_PROFILES=${1:-"-Pkubernetes -Pyarn -Pspark-ganglia-lgpl -Pkinesis-asl 
-Phive-thriftserver -Phive"}
+SPARK_PROFILES=${1:-"-Pkubernetes -Pyarn -Pspark-ganglia-lgpl -Pkinesis-asl 
-Pcredential-aws -Phive-thriftserver -Phive"}
 
 # Capture sbt output to show errors if the command fails
 OLD_DEPS_OUTPUT="$(build/sbt -DcopyDependencies=false $SPARK_PROFILES "export 
oldDeps/fullClasspath" 2>&1)" || {
diff --git a/dev/sbt-checkstyle b/dev/sbt-checkstyle
index f2d5a0fa304a..a148af603918 100755
--- a/dev/sbt-checkstyle
+++ b/dev/sbt-checkstyle
@@ -17,7 +17,7 @@
 # limitations under the License.
 #
 
-SPARK_PROFILES=${1:-"-Pkinesis-asl -Pspark-ganglia-lgpl -Pkubernetes -Pyarn 
-Phive -Phive-thriftserver -Pjvm-profiler"}
+SPARK_PROFILES=${1:-"-Pkinesis-asl -Pcredential-aws -Pspark-ganglia-lgpl 
-Pkubernetes -Pyarn -Phive -Phive-thriftserver -Pjvm-profiler"}
 
 # NOTE: echo "q" is needed because SBT prompts the user for input on 
encountering a build file
 # with failure (either resolution or compilation); the "q" makes SBT quit.
diff --git a/dev/scalastyle b/dev/scalastyle
index 09e6c2372614..f77fb3e1ee22 100755
--- a/dev/scalastyle
+++ b/dev/scalastyle
@@ -17,7 +17,7 @@
 # limitations under the License.
 #
 
-SPARK_PROFILES=${1:-"-Pkubernetes -Pyarn -Pspark-ganglia-lgpl -Pkinesis-asl 
-Phive-thriftserver -Phive -Pvolcano -Pjvm-profiler -Phadoop-cloud 
-Pdocker-integration-tests -Pkubernetes-integration-tests"}
+SPARK_PROFILES=${1:-"-Pkubernetes -Pyarn -Pspark-ganglia-lgpl -Pkinesis-asl 
-Pcredential-aws -Phive-thriftserver -Phive -Pvolcano -Pjvm-profiler 
-Phadoop-cloud -Pdocker-integration-tests -Pkubernetes-integration-tests"}
 
 # NOTE: echo "q" is needed because SBT prompts the user for input on 
encountering a build file
 # with failure (either resolution or compilation); the "q" makes SBT quit.
diff --git a/dev/spark-test-image-util/docs/build-docs 
b/dev/spark-test-image-util/docs/build-docs
index ca59769f2423..c2ee826832ef 100755
--- a/dev/spark-test-image-util/docs/build-docs
+++ b/dev/spark-test-image-util/docs/build-docs
@@ -35,7 +35,7 @@ FWDIR="$(cd "`dirname "${BASH_SOURCE[0]}"`"; pwd)"
 SPARK_HOME="$(cd "`dirname "${BASH_SOURCE[0]}"`"/../../..; pwd)"
 
 # 1.Compile spark outside the container to prepare for generating documents 
inside the container.
-build/sbt -Phive -Pkinesis-asl clean unidoc package
+build/sbt -Phive -Pkinesis-asl -Pcredential-aws clean unidoc package
 
 # 2.Build container image.
 docker buildx build \
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 898a3831f484..9763591f588e 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -410,6 +410,21 @@ streaming_kinesis_asl = Module(
 )
 
 
+credential_aws = Module(
+    name="credential-aws",
+    dependencies=[tags, core],
+    source_file_regexes=[
+        "connector/credential-aws/",
+    ],
+    build_profile_flags=[
+        "-Pcredential-aws",
+    ],
+    sbt_test_goals=[
+        "credential-aws/test",
+    ],
+)
+
+
 streaming_kafka_0_10 = Module(
     name="streaming-kafka-0-10",
     dependencies=[streaming, core],
diff --git a/dev/sparktestsupport/utils.py b/dev/sparktestsupport/utils.py
index 029b1627bd0b..1657aa56da67 100755
--- a/dev/sparktestsupport/utils.py
+++ b/dev/sparktestsupport/utils.py
@@ -121,7 +121,8 @@ def determine_modules_to_test(changed_modules, 
deduplicated=True):
     >>> sorted([x.name for x in determine_modules_to_test(
     ...     [modules.sql, modules.core], deduplicated=False)])
     ... # doctest: +NORMALIZE_WHITESPACE
-    ['avro', 'catalyst', 'connect', 'core', 'docker-integration-tests', 
'examples', 'graphx',
+    ['avro', 'catalyst', 'connect', 'core', 'credential-aws', 
'docker-integration-tests',
+     'examples', 'graphx',
      'hive', 'hive-thriftserver', 'mllib', 'mllib-local', 'pipelines', 
'protobuf',
      'pyspark-connect', 'pyspark-core', 'pyspark-errors', 'pyspark-ml', 
'pyspark-ml-connect',
      'pyspark-mllib', 'pyspark-pandas', 'pyspark-pandas-connect', 
'pyspark-pandas-slow',
diff --git a/dev/test-dependencies.sh b/dev/test-dependencies.sh
index 68c61232ea2a..ad93304e7373 100755
--- a/dev/test-dependencies.sh
+++ b/dev/test-dependencies.sh
@@ -31,7 +31,7 @@ export LC_ALL=C
 # NOTE: These should match those in the release publishing script, and be kept 
in sync with
 #   dev/create-release/release-build.sh
 HADOOP_MODULE_PROFILES="-Phive-thriftserver -Pkubernetes -Pyarn -Phive \
-    -Pspark-ganglia-lgpl -Pkinesis-asl -Phadoop-cloud -Pjvm-profiler"
+    -Pspark-ganglia-lgpl -Pkinesis-asl -Pcredential-aws -Phadoop-cloud 
-Pjvm-profiler"
 MVN="build/mvn"
 HADOOP_HIVE_PROFILES=(
     hadoop-3-hive-2.3
diff --git a/docs/_plugins/build_api_docs.rb b/docs/_plugins/build_api_docs.rb
index 429cef5aa026..4b6251b80423 100644
--- a/docs/_plugins/build_api_docs.rb
+++ b/docs/_plugins/build_api_docs.rb
@@ -45,7 +45,7 @@ def build_spark_if_necessary
 
   print_header "Building Spark."
   cd(SPARK_PROJECT_ROOT)
-  command = "NO_PROVIDED_SPARK_JARS=0 build/sbt -Phive -Pkinesis-asl clean 
package"
+  command = "NO_PROVIDED_SPARK_JARS=0 build/sbt -Phive -Pkinesis-asl 
-Pcredential-aws clean package"
   puts "Running '#{command}'; this may take a few minutes..."
   system(command) || raise("Failed to build Spark")
   # SPARK-53327: Use the modified ResourceImpl.class in spark-catalyst which 
is compatible with Java 25
@@ -129,7 +129,7 @@ def build_spark_scala_and_java_docs_if_necessary
     return
   end
 
-  command = "build/sbt -Pkinesis-asl unidoc"
+  command = "build/sbt -Pkinesis-asl -Pcredential-aws unidoc"
   puts "Running '#{command}'..."
 
   # Two filter passes on the unidoc output, plus an additive fatal-error 
summary:
diff --git a/pom.xml b/pom.xml
index 50f8579ecc2c..d84f5771c8e9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -3524,6 +3524,13 @@
       </modules>
     </profile>
 
+    <profile>
+      <id>credential-aws</id>
+      <modules>
+        <module>connector/credential-aws</module>
+      </modules>
+    </profile>
+
     <profile>
       <id>test-java-home</id>
       <activation>
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 9e301a79f547..4421bb334d86 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -72,10 +72,10 @@ object BuildCommons {
     udfWorkerProjects
 
   val optionallyEnabledProjects@Seq(kubernetes, yarn,
-    sparkGangliaLgpl, streamingKinesisAsl, profiler,
+    sparkGangliaLgpl, streamingKinesisAsl, profiler, credentialAws,
     dockerIntegrationTests, hadoopCloud, kubernetesIntegrationTests) =
     Seq("kubernetes", "yarn",
-      "ganglia-lgpl", "streaming-kinesis-asl", "profiler",
+      "ganglia-lgpl", "streaming-kinesis-asl", "profiler", "credential-aws",
       "docker-integration-tests", "hadoop-cloud", 
"kubernetes-integration-tests").map(ProjectRef(buildLocation, _))
 
   val assemblyProjects@Seq(networkYarn, streamingKafka010Assembly, 
streamingKinesisAslAssembly) =
@@ -416,7 +416,8 @@ object SparkBuild extends PomBuild {
     Seq(
       spark, hive, hiveThriftServer, repl, networkCommon, networkShuffle, 
networkYarn,
       unsafe, tags, tokenProviderKafka010, sqlKafka010, pipelines, 
connectCommon, connect,
-      connectJdbc, connectClient, variant, connectShims, profiler, 
commonUtilsJava, sparkConfig,
+      connectJdbc, connectClient, variant, connectShims, profiler, 
credentialAws,
+      commonUtilsJava, sparkConfig,
       udfWorkerProto, udfWorkerCore, udfWorkerGrpc
     ).contains(x)
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to