[
https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831590#comment-15831590
]
Steve Loughran commented on HADOOP-13876:
-----------------------------------------
That's going to be a separate topic. Looking at this patch, there's a new
property doc in s3guard.md (which is good), and changes to two tests —changes
which aren't needed
We can guarantee that s3guard is disabled by explicitly disabling in
test/resources/core-site.xml. This is what I think we should do, looking at my
auth-keys.xml file (indeed, making me suspect that the csv files are only
working because of the .impl property).
Why not force it in the tests? Because the reason for making the CSV URL
configurable is so that people testing against their own S3 infra (e.g the WDC
team) have the option to point to a local file. Now, they don't need s3guard,
having a consistent store, but it may be that others (me?) want to test against
a different file with s3guard turned on, just to see what happens.
I think we could just add these values to the test core-site and rely on it to
set things up...people can override if they want
{code}
<property>
<name>fs.s3a.bucket.landsat-pds.endpoint</name>
<value>${central.endpoint}</value>
<description>The endpoint for s3a://landsat-pds URLs</description>
</property>
<property>
<name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name>
<value>${s3guard.null}</value>
<description>The read-only landsat-pds repository isn't
managed by s3guard</description>
</property>
<property>
<name>s3guard.null</name>
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
</property>
<property>
<name>s3guard.dynamo</name>
<value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
</property>
<property>
<name>frankfurt.endpoint</name>
<value>s3.eu-central-1.amazonaws.com</value>
</property>
<property>
<name>ireland.endpoint</name>
<value>s3-eu-west-1.amazonaws.com</value>
</property>
<property>
<name>central.endpoint</name>
<value>s3.amazonaws.com</value>
</property>
{code}
> S3Guard: better support for multi-bucket access including read-only
> -------------------------------------------------------------------
>
> Key: HADOOP-13876
> URL: https://issues.apache.org/jira/browse/HADOOP-13876
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
> Attachments: HADOOP-13876-HADOOP-13345.000.patch
>
>
> HADOOP-13449 adds support for DynamoDBMetadataStore.
> The code currently supports two options for choosing DynamoDB table names:
> 1. Use name of each s3 bucket and auto-create a DynamoDB table for each.
> 2. Configure a table name in the {{fs.s3a.s3guard.ddb.table}} parameter.
> One of the issues is with accessing read-only buckets. If a user accesses a
> read-only bucket with credentials that do not have DynamoDB write
> permissions, they will get errors when trying to access the read-only bucket.
> This manifests causes test failures for {{ITestS3AAWSCredentialsProvider}}.
> Goals for this JIRA:
> - Fix {{ITestS3AAWSCredentialsProvider}} in a way that makes sense for the
> real use-case.
> - Allow for a "one DynamoDB table per cluster" configuration with a way to
> chose which credentials are used for DynamoDB.
> - Document limitations etc. in the s3guard.md site doc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
