gaborgsomogyi commented on PR #27187: URL: https://github.com/apache/flink/pull/27187#issuecomment-3586645049
I've just picked this up and intend to test it in-depth. At the first touch without any special config entries the app blows up with invalid region. After some digging I've found out the following: Hadoop Connector (S3A with AWS SDK v1) The Hadoop connector uses AWS SDK v1 which has automatic region detection through a comprehensive fallback chain: 1. Explicit Configuration: Checks fs.s3a.endpoint.region Hadoop configuration 2. Environment Variables: AWS_REGION or AWS_DEFAULT_REGION 3. EC2 Instance Metadata: Queries EC2 metadata service if running on AWS 4. AWS Config/Credentials Files: ~/.aws/config region settings 5. Bucket Location API: As a last resort, calls GetBucketLocation API to determine the bucket's region This is handled by AWS SDK v1's `DefaultAwsRegionProviderChain` class, which is automatically used when no region is explicitly configured. Native Connector (AWS SDK v2) The Native connector uses AWS SDK v2 which is more strict: - Hardcoded default: us-east-1 (as we saw in NativeS3FileSystemFactory.java:63) - No automatic bucket location detection: SDK v2 does not call GetBucketLocation API by default - Requires explicit configuration: Must set `s3.region` in Flink config I think we must use `DefaultAwsRegionProviderChain` in the native lib too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
