[
https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812375#comment-15812375
]
Larry McCay commented on HADOOP-13336:
--------------------------------------
Hi [[email protected]] -
This looks like a good approach - couple comments:
* In propagateBucketOptions it seems that there may be a redundant loop:
{noformat}
+ for (String unmodifiable : UNMODIFIABLE_PREFIX) {
+ if (stripped.startsWith(unmodifiable)) {
+ //tell user off
+ LOG.debug("Ignoring bucket option {}", key);
+ skip = true;
+ }
+ }
+ for (String unmodifiable : UNMODIFIABLE_VALUE) {
+ if (stripped.equals(unmodifiable)) {
+ //tell user off
+ LOG.debug("Ignoring bucket option {}", key);
+ skip = true;
+ }
+ }
{noformat}
* would it also be better to have the log message mention why it is being
ignored?
* would the following code in the same method result in passwords being logged
by any chance:
{noformat}
+ // propagate the value, building a new origin field.
+ // to track overwrites, the generic key is overwritten even if
+ // already matches the new one.
+ if (!skip) {
+ final String generic = FS_S3A_PREFIX + stripped;
+ LOG.debug("Setting {} to value of {} : {} (was {})",
+ generic, key, value, source.get(generic));
+ dest.set(generic, value, key);
+ }
{noformat}
* I don't see any evidence that this will work with credential provider API
through the use of Configuration.getPassword. Am I missing some other nuance
here?
> S3A to support per-bucket configuration
> ---------------------------------------
>
> Key: HADOOP-13336
> URL: https://issues.apache.org/jira/browse/HADOOP-13336
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch,
> HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch,
> HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch,
> HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch
>
>
> S3a now supports different regions, by way of declaring the endpoint —but you
> can't do things like read in one region, write back in another (e.g. a distcp
> backup), because only one region can be specified in a configuration.
> If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt
> s3a://b2.seol , then this would be possible.
> Swift does this with a full filesystem binding/config: endpoints, username,
> etc, in the XML file. Would we need to do that much? It'd be simpler
> initially to use a domain suffix of a URL to set the region of a bucket from
> the domain and have the aws library sort the details out itself, maybe with
> some config options for working with non-AWS infra
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
