dwsmith1983 commented on code in PR #5872:
URL: https://github.com/apache/datafusion-comet/pull/5872#discussion_r4081289124
##########
native/core/src/parquet/objectstore/s3.rs:
##########
@@ -814,11 +908,35 @@ impl CredentialProviderMetadata {
.build();
Ok(Arc::new(credential_provider))
}
- CredentialProviderMetadata::Profile => {
- let credential_provider =
ProfileFileCredentialsProvider::builder()
- .configure(&ProviderConfig::with_default_region().await)
- .build();
- Ok(Arc::new(credential_provider))
+ CredentialProviderMetadata::Profile {
+ name,
+ file,
+ credentials_only,
+ } => {
+ let mut builder = ProfileFileCredentialsProvider::builder()
+ .configure(&ProviderConfig::with_default_region().await);
+ if let Some(name) = name {
+ builder = builder.profile_name(name);
+ }
+ // Hadoop's ProfileAWSCredentialsProvider loads the configured
file, or the
+ // shared credentials file, as a credentials-format file and
reads nothing
+ // else, so a same-name role profile in the SDK's config file
never applies.
+ let credentials_file = match (file, credentials_only) {
+ (Some(file), _) => Some(file.clone()),
+ (None, true) => Some(default_shared_credentials_file(
+ std::env::var("AWS_SHARED_CREDENTIALS_FILE").ok(),
+ std::env::var("HOME").ok(),
+ )),
Review Comment:
> Could we pass Hadoop's resolved default file into this branch instead of
deriving it from the native process's `HOME`?
In f3758e59f. The executor resolves `fs.s3a.comet.default.profile.file`
against its own `user.home` or `AWS_SHARED_CREDENTIALS_FILE` when a native plan
is created, native keeps it as a session extension and overlays it onto each
scan's options, and the Hadoop provider arm takes a configured
`fs.s3a.auth.profile.file` first, then that path. A driver and executor with
different homes resolve the executor's.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
