This is an automated email from the ASF dual-hosted git repository.
kou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new 120b0b760d GH-50899: [CI][C++] Use OIDC instead of ACCESS_KEY and
SECRET_KEY for sccache's s3 creds (#51028)
120b0b760d is described below
commit 120b0b760da48452523a7f2a23aae424626acd5b
Author: paultiq <[email protected]>
AuthorDate: Sun Sep 6 03:04:43 2026 -0400
GH-50899: [CI][C++] Use OIDC instead of ACCESS_KEY and SECRET_KEY for
sccache's s3 creds (#51028)
### Rationale for this change
This PR moves crossbow builds to short-lived S3 credentials. It does this
via GitHub's OIDC provider and an AWS ROLE_ARN with a trust relationship
configured. This configuration is detailed here:
- [Use IAM roles to connect GitHub Actions to actions in
AWS](https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/)
The benefits of short-lived / temporary credentials discussed further here:
[AWS Well-architected Framework Best Practices SEC02-BP02 Use temporary
credentials](https://docs.aws.amazon.com/wellarchitected/latest/framework/sec_identities_unique.html)
sccache supports a variety of S3 creds: [S3
credentials](https://github.com/mozilla/sccache/blob/main/docs/S3.md#credentials)
### What changes are included in this PR?
Use
[AssumeRoleWithWebIdentity](https://github.com/mozilla/sccache/blob/main/docs/S3.md#credentials)
for sccache S3 credentials.
By assuming the AWS_ROLE_ARN, the sccache session relies on credentials
that expire\*, issued only to workflows in the trusted repository, and scoped
to the sccache bucket.
Specific tasks:
- create a role in IAM with the bucket and object level permissions,
- trust the github oidc-provider,
- define AWS_ROLE_ARN and set id-token permission
- grant sts:AssumeRoleWithWebIdentity for the specific repositories or
organization.
- pass AWS_SESSION_TOKEN
- add a `aws-actions/configure-aws-credentials@ v6` step to the workflow.
Separately, someone will need to add the role and trust to AWS:
- Procedure is in
https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/
- Set a "Maximum session duration" to maximum runtime of any job... maybe 4
hours? 8 hours?
- Permissions: Role needs BucketLevel and ObjectLevels permissions to the
bucket.
\* I can share an example if needed
### Are these changes tested?
Yes, locally and on a fork.
### Are there any user-facing changes?
No
* GitHub Issue: #50899
Authored-by: [email protected] <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
---
compose.yaml | 1 +
dev/tasks/docker-tests/github.linux.yml | 6 ++++++
dev/tasks/macros.jinja | 18 ++++++++++++++++--
dev/tasks/matlab/github.yml | 8 ++++++++
dev/tasks/python-wheels/github.linux.yml | 3 +++
dev/tasks/r/github.linux.arrow.version.back.compat.yml | 6 ++++++
dev/tasks/r/github.linux.cran.yml | 6 ++++++
dev/tasks/r/github.linux.offline.build.yml | 6 ++++++
dev/tasks/r/github.linux.sanitizers.yml | 6 ++++++
dev/tasks/r/github.linux.versions.yml | 6 ++++++
dev/tasks/r/github.macos-linux.local.yml | 6 ++++++
dev/tasks/r/github.packages.yml | 9 +++++++++
12 files changed, 79 insertions(+), 2 deletions(-)
diff --git a/compose.yaml b/compose.yaml
index 00e791ccea..48a1bb060a 100644
--- a/compose.yaml
+++ b/compose.yaml
@@ -73,6 +73,7 @@ x-cpp: &cpp
x-sccache: &sccache
AWS_ACCESS_KEY_ID:
AWS_SECRET_ACCESS_KEY:
+ AWS_SESSION_TOKEN:
SCCACHE_BUCKET:
SCCACHE_REGION:
SCCACHE_S3_KEY_PREFIX: ${SCCACHE_S3_KEY_PREFIX:-sccache}
diff --git a/dev/tasks/docker-tests/github.linux.yml
b/dev/tasks/docker-tests/github.linux.yml
index a893432cf9..afb0e1569e 100644
--- a/dev/tasks/docker-tests/github.linux.yml
+++ b/dev/tasks/docker-tests/github.linux.yml
@@ -19,6 +19,11 @@
{{ macros.github_header() }}
+permissions:
+ contents: read
+ # For OIDC token for sccache
+ id-token: write
+
jobs:
test:
name: |
@@ -31,6 +36,7 @@ jobs:
{{ macros.github_free_space()|indent }}
{{ macros.github_install_archery()|indent }}
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Execute Docker Build
shell: bash
env:
diff --git a/dev/tasks/macros.jinja b/dev/tasks/macros.jinja
index 8e9a41b46b..5b74999d95 100644
--- a/dev/tasks/macros.jinja
+++ b/dev/tasks/macros.jinja
@@ -26,6 +26,7 @@ on:
env:
ARCHERY_DEBUG: 1
+ SCCACHE_ENABLED: {{ "${{ secrets.SCCACHE_REGION != '' }}" }}
{% endmacro %}
{%- macro github_checkout_arrow(fetch_depth=1, submodules="recursive",
action_v="4") -%}
@@ -287,10 +288,23 @@ env:
{% endif %}
{%- endmacro -%}
+{%- macro github_configure_aws_credentials() -%}
+ - name: Configure AWS credentials for sccache
+ if: env.SCCACHE_ENABLED == 'true'
+ # don't fail the build if AWS credentials are unavailable
+ continue-on-error: true
+ uses: aws-actions/configure-aws-credentials@v6
+ with:
+ aws-region: {{ '${{ secrets.SCCACHE_REGION }}' }}
+ role-to-assume: {{ '${{ secrets.AWS_ROLE_ARN }}' }}
+ role-duration-seconds: 14400
+{% endmacro %}
+
{% macro github_set_sccache_envvars(sccache_key_prefix = "sccache") %}
{% set sccache_vars = {
- "AWS_ACCESS_KEY_ID": '${{ secrets.AWS_ACCESS_KEY_ID }}',
- "AWS_SECRET_ACCESS_KEY": '${{ secrets.AWS_SECRET_ACCESS_KEY }}',
+ "AWS_ACCESS_KEY_ID": '${{ env.AWS_ACCESS_KEY_ID }}',
+ "AWS_SECRET_ACCESS_KEY": '${{ env.AWS_SECRET_ACCESS_KEY }}',
+ "AWS_SESSION_TOKEN": '${{ env.AWS_SESSION_TOKEN }}',
"SCCACHE_BUCKET": '${{ secrets.SCCACHE_BUCKET }}',
"SCCACHE_REGION": '${{ secrets.SCCACHE_REGION }}',
"SCCACHE_S3_KEY_PREFIX": sccache_key_prefix
diff --git a/dev/tasks/matlab/github.yml b/dev/tasks/matlab/github.yml
index 77f3056c36..39e79ec151 100644
--- a/dev/tasks/matlab/github.yml
+++ b/dev/tasks/matlab/github.yml
@@ -19,6 +19,11 @@
{{ macros.github_header() }}
+permissions:
+ contents: write
+ # For OIDC token for sccache
+ id-token: write
+
jobs:
ubuntu:
@@ -32,6 +37,7 @@ jobs:
uses: matlab-actions/setup-matlab@v2
with:
release: R2025b
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Build MATLAB Interface
env:
{{ macros.github_set_sccache_envvars()|indent(8) }}
@@ -74,6 +80,7 @@ jobs:
uses: matlab-actions/setup-matlab@v2
with:
release: R2025b
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Build MATLAB Interface
env:
{{ macros.github_set_sccache_envvars()|indent(8) }}
@@ -103,6 +110,7 @@ jobs:
- name: Install sccache
shell: bash
run: arrow/ci/scripts/install_sccache.sh pc-windows-msvc $(pwd)/sccache
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Build MATLAB Interface
shell: cmd
env:
diff --git a/dev/tasks/python-wheels/github.linux.yml
b/dev/tasks/python-wheels/github.linux.yml
index e2a74eb841..16ee668bff 100644
--- a/dev/tasks/python-wheels/github.linux.yml
+++ b/dev/tasks/python-wheels/github.linux.yml
@@ -21,6 +21,8 @@
permissions:
packages: write
+ # For OIDC token for sccache
+ id-token: write
jobs:
build:
@@ -65,6 +67,7 @@ jobs:
fi
echo "TEST_IMAGE_PREFIX=${test_image_prefix}" >> ${GITHUB_ENV}
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Build wheel
shell: bash
env:
diff --git a/dev/tasks/r/github.linux.arrow.version.back.compat.yml
b/dev/tasks/r/github.linux.arrow.version.back.compat.yml
index bbe88d205d..6478ab9c99 100644
--- a/dev/tasks/r/github.linux.arrow.version.back.compat.yml
+++ b/dev/tasks/r/github.linux.arrow.version.back.compat.yml
@@ -19,6 +19,11 @@
{{ macros.github_header() }}
+permissions:
+ contents: read
+ # For OIDC token for sccache
+ id-token: write
+
jobs:
write-files:
name: "Write files"
@@ -44,6 +49,7 @@ jobs:
shell: Rscript {0}
- name: Install sccache
run: arrow/ci/scripts/install_sccache.sh unknown-linux-musl
/usr/local/bin
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Install Arrow
env:
{{ macros.github_set_sccache_envvars()|indent(8) }}
diff --git a/dev/tasks/r/github.linux.cran.yml
b/dev/tasks/r/github.linux.cran.yml
index a136e1a6be..0c1ef5ae85 100644
--- a/dev/tasks/r/github.linux.cran.yml
+++ b/dev/tasks/r/github.linux.cran.yml
@@ -19,6 +19,11 @@
{{ macros.github_header() }}
+permissions:
+ contents: read
+ # For OIDC token for sccache
+ id-token: write
+
jobs:
as-cran:
name: "rhub/{{ '${{ matrix.config.r_image }}' }}"
@@ -43,6 +48,7 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_install_archery()|indent }}
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Docker Run
shell: bash
env:
diff --git a/dev/tasks/r/github.linux.offline.build.yml
b/dev/tasks/r/github.linux.offline.build.yml
index ac4ad10137..fb5862ccb7 100644
--- a/dev/tasks/r/github.linux.offline.build.yml
+++ b/dev/tasks/r/github.linux.offline.build.yml
@@ -19,6 +19,11 @@
{{ macros.github_header() }}
+permissions:
+ contents: read
+ # For OIDC token for sccache
+ id-token: write
+
jobs:
grab-dependencies:
name: "Download thirdparty dependencies"
@@ -77,6 +82,7 @@ jobs:
install.packages(c("remotes", "glue", "sys"))
remotes::install_deps("arrow/r", dependencies = TRUE)
shell: Rscript {0}
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Install
env:
ARROW_OFFLINE_BUILD: true
diff --git a/dev/tasks/r/github.linux.sanitizers.yml
b/dev/tasks/r/github.linux.sanitizers.yml
index ed31b13a08..af810cd47a 100644
--- a/dev/tasks/r/github.linux.sanitizers.yml
+++ b/dev/tasks/r/github.linux.sanitizers.yml
@@ -19,6 +19,11 @@
{{ macros.github_header() }}
+permissions:
+ contents: read
+ # For OIDC token for sccache
+ id-token: write
+
jobs:
r-linux-sanitizers:
name: "rhub/{{ '${{ matrix.config.r_image }}' }}"
@@ -41,6 +46,7 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_install_archery()|indent }}
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Docker Run
shell: bash
env:
diff --git a/dev/tasks/r/github.linux.versions.yml
b/dev/tasks/r/github.linux.versions.yml
index 644494bfba..a597d1dcb4 100644
--- a/dev/tasks/r/github.linux.versions.yml
+++ b/dev/tasks/r/github.linux.versions.yml
@@ -19,6 +19,11 @@
{{ macros.github_header() }}
+permissions:
+ contents: read
+ # For OIDC token for sccache
+ id-token: write
+
jobs:
r-versions:
name: "posit/r-base:{{ MATRIX }}-jammy"
@@ -43,6 +48,7 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_install_archery()|indent }}
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Docker Run
shell: bash
env:
diff --git a/dev/tasks/r/github.macos-linux.local.yml
b/dev/tasks/r/github.macos-linux.local.yml
index ada1c7642d..2391c97b0f 100644
--- a/dev/tasks/r/github.macos-linux.local.yml
+++ b/dev/tasks/r/github.macos-linux.local.yml
@@ -19,6 +19,11 @@
{{ macros.github_header() }}
+permissions:
+ contents: read
+ # For OIDC token for sccache
+ id-token: write
+
jobs:
local:
name: "install from local source"
@@ -78,6 +83,7 @@ jobs:
extra-packages: |
any::rcmdcheck
any::sys
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Install
env:
_R_CHECK_CRAN_INCOMING_: false
diff --git a/dev/tasks/r/github.packages.yml b/dev/tasks/r/github.packages.yml
index feeac3dba4..05da643306 100644
--- a/dev/tasks/r/github.packages.yml
+++ b/dev/tasks/r/github.packages.yml
@@ -21,6 +21,11 @@
{{ macros.github_header() }}
+permissions:
+ contents: write
+ # For OIDC token for sccache
+ id-token: write
+
jobs:
source:
# This job will change the version to either the custom_version param or
YMD format.
@@ -77,6 +82,7 @@ jobs:
brew install sccache ninja
brew install [email protected]
brew install libxml2
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Build libarrow
shell: bash
env:
@@ -132,6 +138,7 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_change_r_pkg_version(is_fork, '${{
needs.source.outputs.pkg_version }}')|indent }}
{{ macros.github_install_archery()|indent }}
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Build libarrow
shell: bash
env:
@@ -182,6 +189,7 @@ jobs:
- name: Install sccache
shell: bash
run: arrow/ci/scripts/install_sccache.sh pc-windows-msvc $(pwd)/sccache
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Build Arrow C++ with rtools40
shell: bash
env:
@@ -387,6 +395,7 @@ jobs:
run: |
cores=`nproc || sysctl -n hw.logicalcpu`
echo "MAKEFLAGS=-j$cores" >> $GITHUB_ENV
+ {{ macros.github_configure_aws_credentials()|indent }}
- name: Install arrow source package
env:
# Test source build so be sure not to download a binary
