sunchao commented on code in PR #5872:
URL: https://github.com/apache/datafusion-comet/pull/5872#discussion_r4167676503
##########
native/core/src/parquet/objectstore/s3.rs:
##########
@@ -943,11 +1103,35 @@ impl CredentialProviderMetadata {
.build();
Ok(Arc::new(credential_provider))
}
- CredentialProviderMetadata::Profile => {
- let credential_provider =
ProfileFileCredentialsProvider::builder()
- .configure(&ProviderConfig::with_default_region().await)
- .build();
- Ok(Arc::new(credential_provider))
+ CredentialProviderMetadata::Profile {
+ name,
+ file,
+ credentials_only,
+ } => {
+ let mut builder = ProfileFileCredentialsProvider::builder()
+ .configure(&ProviderConfig::with_default_region().await);
Review Comment:
[P2] Resolve the region after selecting the Hadoop profile. With
`fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.auth.ProfileAWSCredentialsProvider`,
`fs.s3a.auth.profile.name=analytics`, and `fs.s3a.auth.profile.file` pointing
to an assume-role profile containing `region=us-west-2`, native credential
resolution fails when the default SDK profile and environment supply no region.
`with_default_region()` runs before the new name/file overrides, and those
setters do not recompute the region. Hadoop’s AWS SDK reads the region from the
selected role profile, while native returns `Invalid Configuration: Missing
Region` before issuing an STS request. Consequently, this newly supported
configuration cannot perform native reads. Please derive the provider’s region
from the selected profile with Hadoop’s precedence and retain SDK-alias
behavior.
Evidence: Reproduced using the verbatim profile-construction block and
locked `aws-config 1.12.0`/`aws-runtime 1.9.2`. The synthetic credentials file
contains `[analytics]` with
`role_arn=arn:aws:iam::123456789012:role/synthetic`, `source_profile=source`,
and `region=us-west-2`, plus synthetic static credentials under `[source]`.
With `AWS_PROFILE=default`, empty default SDK files, region environment
variables unset, and IMDS disabled, `provide_credentials()` returns
`ResolveEndpointError: Invalid Configuration: Missing Region`. Reproduction:
`/tmp/comet5872-oct02-role-repro/src/main.rs`; output:
`/tmp/comet5872-oct02-role-repro-no-endpoint.log`. Setting
`AWS_REGION=us-west-2` resolves the same profile successfully against a
synthetic loopback STS server. Hadoop 3.4.2’s `ProfileAWSCredentialsProvider`
passes the selected file/name to Java’s provider, and AWS SDK 2.29.52
`StsProfileCredentialsProviderFactory.configureEndpoint` explicitly selects
`profile.property(ProfilePropert
y.REGION)` first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
