[
https://issues.apache.org/jira/browse/HADOOP-19470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19470:
------------------------------------
Description:
I think this is new to the V2 SDK, or at least our region logic there.
When you try to connect to a bucket without specifying the region, then even if
you're running in the same region of the store a HEAD request is still made to
US Central. This adds latency, makes us-central a SPoF and if the VM/container
has network rules which blocks such access, we actually timeout and eventually
fail.
While Machine configurations should ideally have the fs.s3a.endpoint.region
setting configured, that information is actually provided as IAM metadata.
Therefore it would be possible
This is actually included in the default region chain according to the SDK docs
"If running in EC2, check the EC2 metadata service for the region", so maybe
this isn't being picked up because
# cross region access is being checked for first.
# the region chain we are setting off doesn't check the EC2 metadata service.
The SDK region chain does do the right thing within AWS infra. How do we
restore that while still supporting remote deployments?
was:
S3A: region resolution within AWS infra always goes to us-east-2
I think this is new to the V2 SDK, or at least our region logic there.
When you try to connect to a bucket without specifying the region, then even if
you're running in the same region of the store a
While Machine configurations should ideally have the fs.s3a.endpoint.region
setting configured, that information is actually pro
This is actually included in the default region chain according to the SDK docs
"If running in EC2, check the EC2 metadata servi
# cross region access is being checked for first.
# the region chain we are setting off doesn't check the EC2 metadata service.
The SDK region chain does do the right thing within AWS infra. How do we
restore that while still supporting remote deployments?
> S3A: region resolution within AWS infra always goes to us-east-2
> ----------------------------------------------------------------
>
> Key: HADOOP-19470
> URL: https://issues.apache.org/jira/browse/HADOOP-19470
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.1
> Reporter: Steve Loughran
> Priority: Major
>
> I think this is new to the V2 SDK, or at least our region logic there.
> When you try to connect to a bucket without specifying the region, then even
> if you're running in the same region of the store a HEAD request is still
> made to US Central. This adds latency, makes us-central a SPoF and if the
> VM/container has network rules which blocks such access, we actually timeout
> and eventually fail.
> While Machine configurations should ideally have the fs.s3a.endpoint.region
> setting configured, that information is actually provided as IAM metadata.
> Therefore it would be possible
> This is actually included in the default region chain according to the SDK
> docs "If running in EC2, check the EC2 metadata service for the region", so
> maybe this isn't being picked up because
> # cross region access is being checked for first.
> # the region chain we are setting off doesn't check the EC2 metadata service.
> The SDK region chain does do the right thing within AWS infra. How do we
> restore that while still supporting remote deployments?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
