[
https://issues.apache.org/jira/browse/HADOOP-19470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18025096#comment-18025096
]
Steve Loughran commented on HADOOP-19470:
-----------------------------------------
This has surfaced in the wild again.
I propose some new regions
* "sdk": hand off to the sdk
* "auto": do whatever we think is clever. May change from version to version
* "ec2" code is running in EC2,
> S3A: region resolution within AWS infra always goes to us-east-2
> ----------------------------------------------------------------
>
> Key: HADOOP-19470
> URL: https://issues.apache.org/jira/browse/HADOOP-19470
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.1
> Reporter: Steve Loughran
> Priority: Major
>
> I think this is new to the V2 SDK, or at least our region logic there.
> When you try to connect to a bucket without specifying the region, then even
> if you're running in the same region of the store a HEAD request is still
> made to US Central. This adds latency, makes us-central a SPoF and if the
> VM/container has network rules which blocks such access, we actually timeout
> and eventually fail.
> While Machine configurations should ideally have the fs.s3a.endpoint.region
> setting configured, that information is actually provided as IAM metadata.
> Therefore it would be possible
> This is actually included in the default region chain according to the SDK
> docs "If running in EC2, check the EC2 metadata service for the region", so
> maybe this isn't being picked up because
> # cross region access is being checked for first.
> # the region chain we are setting off doesn't check the EC2 metadata service.
> The SDK region chain does do the right thing within AWS infra. How do we
> restore that while still supporting remote deployments?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
