Steve Loughran created HADOOP-18095:
---------------------------------------
Summary: s3a connector to fully support AWS partitions,
Key: HADOOP-18095
URL: https://issues.apache.org/jira/browse/HADOOP-18095
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.2
Reporter: Steve Loughran
There are some minor issues in using the S3A connector's more advanced features
in china
see https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html
Specifically, that "arn:aws:" prefix we use for all arns needs to be
configurable so that aws-cn can be used instead.
This means finding where we create and use these in production code
(dynamically creating IAM role policies) and in tests, and making it
configurable.
proposed
* add an option {{fs.s3a.aws.partition}}, default aws.
* new StoreContext methods to query this, and create the arn for the current
bucket (string concat or from the bucket's ARN if created with an AP ARN)
* docs
I remember ABFS had a problem with oauth endpoints, that was a lot more serious.
Can't think of real tests for this, other than verifying that if you create an
invalid partition "aws-mars" some things break.
someone needs to run all our existing tests in china, including those with IAM
roles and SSE-KMS.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
