[
https://issues.apache.org/jira/browse/CASSANALYTICS-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18081002#comment-18081002
]
Jon Haddad commented on CASSANALYTICS-155:
------------------------------------------
I've verified this works in this [easy-db-lab
branch|https://github.com/rustyrazorblade/easy-db-lab/tree/31ec4c8f4e040e0a0da257b7bdf4e564bc64284f/spark/bulk-writer-s3-iam].
[Storage
Extension|https://github.com/rustyrazorblade/easy-db-lab/blob/31ec4c8f4e040e0a0da257b7bdf4e564bc64284f/spark/bulk-writer-s3-iam/src/main/java/com/rustyrazorblade/easydblab/spark/EasyDbLabIamStorageExtension.java]
[Script to run it
all.|https://github.com/rustyrazorblade/easy-db-lab/blob/31ec4c8f4e040e0a0da257b7bdf4e564bc64284f/bin/test-spark-bulk-writer-s3-iam]
Relevant output from the spark job:
{noformat}
=== Submitting IAM bulk writer Spark job ===
Uploading bulk-writer-s3-iam-all.jar to
s3://easy-db-lab-data-6afa29e6-338a-47eb-95e2-568ec44a3792/spark/bulk-writer-s3-iam-all.jar...
Upload complete:
s3://easy-db-lab-data-6afa29e6-338a-47eb-95e2-568ec44a3792/spark/bulk-writer-s3-iam-all.jar
Submitted Spark job: s-04517261ITV6J5DMUKN9 to cluster j-UEAA3KYPRF1
Waiting for job completion...
Job state: RUNNING
Job completed successfully
=== Verifying row count ===
"count"
----------
1000000
=== Done ===
{noformat}
> support IAM when bulk writing to S3
> -----------------------------------
>
> Key: CASSANALYTICS-155
> URL: https://issues.apache.org/jira/browse/CASSANALYTICS-155
> Project: Apache Cassandra Analytics
> Issue Type: Improvement
> Reporter: Jon Haddad
> Priority: Normal
>
> When triggering a restore job via Apache Cassandra Sidecar, the analytics
> library currently requires explicit static S3 credentials to be passed in the
> job request. This prevents deployments where the Sidecar node authenticates
> to S3 via IAM instance profiles or other non-static credential providers.
> CASSSIDECAR-415 adds support on the Sidecar side for a credentialType field
> in the restore job request, allowing the Sidecar to use its own IAM role when
> credentials are not provided.
> This ticket tracks the corresponding changes in Cassandra Analytics to allow
> users to configure a CredentialType on StorageCredentialPair, selecting how
> the Sidecar should authenticate to S3. The initial supported types are STATIC
> (existing behavior, no change) and IAM. The design leaves room for additional
> credential types in the future without structural changes.
> All existing restore job functionality (coordinated writes, consistency
> levels, cross-region support, lease management) is unaffected — only the
> authentication method passed to the Sidecar changes.
> The same authentication method passed to sidecar will also need to be used to
> push the files to S3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
