dwsmith1983 opened a new issue, #6575: URL: https://github.com/apache/datafusion-comet/issues/6575
### Describe the bug When `fs.s3a.aws.credentials.provider` names an AWS SDK profile provider, `software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider` or `com.amazonaws.auth.profile.ProfileCredentialsProvider`, native scans resolve the profile with aws-config's `ProfileFileCredentialsProvider`. With Hadoop 3.4 and later (Spark 4.x), Hadoop maps both names to the Java SDK v2 `ProfileCredentialsProvider`, and the two SDKs resolve profiles differently. Some of these differences change which identity signs requests or which STS endpoint is called: - a source profile with both static keys and `credential_process`: Java uses `credential_process`, aws-config the keys; - assume-role STS region: Java uses each role's own `region`, else the default region chain, else the global `sts.amazonaws.com`; the native side uses one region, from aws-config's default region chain, for every role in the chain; - a role whose `source_profile` names itself: Java reports a cycle, aws-config assumes the role with the profile's own keys; - SSO keys together with `role_arn`: Java uses SSO, aws-config assumes the role. This is the current behavior on `main` and `branch-1.1`. Hadoop's own `org.apache.hadoop.fs.s3a.auth.ProfileAWSCredentialsProvider` is handled in #5872 and is not part of this issue. ### Steps to reproduce On Spark 4.x, set `fs.s3a.aws.credentials.provider=software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider` and select an assume-role profile whose source profile has both `aws_access_key_id` and `credential_process`. A native read signs `AssumeRole` with the static key, while Spark with Comet disabled signs with the process credentials. ### Expected behavior Native scans pick the same credentials and STS endpoints as Hadoop for the SDK profile provider names. One option is to route them through `HadoopS3ACredentialProviderAdapter`, which builds Hadoop's own provider list on the executor. That would add the adapter's classpath requirement and a JNI call per request for static keys to configurations that work today. ### Additional context Found while reviewing #5872. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
