[
https://issues.apache.org/jira/browse/HADOOP-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513502#comment-17513502
]
Steve Loughran commented on HADOOP-18095:
-----------------------------------------
* whatever is needed for the sse tests to be happy...wildcards are fine.
* adding a partition field to the StoreContext too. is there any way to
dynamically determine this, or will it just need to be configured
DelegationTokens are trouble, but given the lack of complaints I'm not sure how
many people use that assumed role feature in production. (we use DTs and the
plugin point, but with our own tokens and auth service).
ideally all test configs should be retargetable, as people doing private store
testing need this. generally the more aws-specific features (STS, SSE) are
skipped. If there are some paths/buckets/endpoints which aren't,. thats
something to fix
i like the idea of a pubilc dataset class incorporating all of this. we could
have something which takes a path to a hadoop XML config file; we'd have a
default resource but the maven build could be pointed at another via a command
line property. this file could contain all the settings for a test against a
partition or internal s3-compatible store
> s3a connector to fully support AWS partitions,
> ----------------------------------------------
>
> Key: HADOOP-18095
> URL: https://issues.apache.org/jira/browse/HADOOP-18095
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.2
> Reporter: Steve Loughran
> Priority: Minor
>
> There are some minor issues in using the S3A connector's more advanced
> features in china
> see https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html
> Specifically, that "arn:aws:" prefix we use for all arns needs to be
> configurable so that aws-cn can be used instead.
> This means finding where we create and use these in production code
> (dynamically creating IAM role policies) and in tests, and making it
> configurable.
> proposed
> * add an option {{fs.s3a.aws.partition}}, default aws.
> * new StoreContext methods to query this, and create the arn for the current
> bucket (string concat or from the bucket's ARN if created with an AP ARN)
> * docs
> I remember ABFS had a problem with oauth endpoints, that was a lot more
> serious.
> Can't think of real tests for this, other than verifying that if you create
> an invalid partition "aws-mars" some things break.
> someone needs to run all our existing tests in china, including those with
> IAM roles and SSE-KMS.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
