sunchao commented on code in PR #5872:
URL: https://github.com/apache/datafusion-comet/pull/5872#discussion_r4012497151
##########
native/core/src/parquet/objectstore/s3.rs:
##########
@@ -814,11 +882,22 @@ impl CredentialProviderMetadata {
.build();
Ok(Arc::new(credential_provider))
}
- CredentialProviderMetadata::Profile => {
- let credential_provider =
ProfileFileCredentialsProvider::builder()
- .configure(&ProviderConfig::with_default_region().await)
- .build();
- Ok(Arc::new(credential_provider))
+ CredentialProviderMetadata::Profile { name, file } => {
+ let mut builder = ProfileFileCredentialsProvider::builder()
+ .configure(&ProviderConfig::with_default_region().await);
+ if let Some(name) = name {
+ builder = builder.profile_name(name);
+ }
+ if let Some(file) = file {
Review Comment:
### Correctness
[P2] Keep Hadoop's default profile source credentials-only
Could we preserve Hadoop's file selection when `fs.s3a.auth.profile.file` is
unset too? The newly supported `HADOOP_PROFILE` arm reaches this branch with
`file: None`, so it leaves the Rust SDK defaults in place. In the locked SDK
those defaults merge `~/.aws/config` with `~/.aws/credentials`. [Hadoop's
provider](https://github.com/apache/hadoop/blob/22a6ad99379850e1fc43e3032a9d676fff7abbf2/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/ProfileAWSCredentialsProvider.java#L57-L100)
instead selects only `AWS_SHARED_CREDENTIALS_FILE` or `~/.aws/credentials` in
this case.
For example, with `auth.profile.name=analytics`, static credentials in the
credentials file and a same-name config profile containing `role_arn` plus
`source_profile=analytics`, Hadoop uses the static identity while the native
SDK merges in the role and assumes it. That can change the identity or fail
native reads that Hadoop can perform. Please retain the Hadoop/SDK provider
distinction and select a credentials-only default for the Hadoop spelling,
while preserving normal SDK defaults for the SDK spellings. A test of the
selected file set when the override is absent would cover this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
