[
https://issues.apache.org/jira/browse/HADOOP-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403083#comment-16403083
]
Aaron Fabbri commented on HADOOP-15314:
---------------------------------------
The DynamoDB implementation (DynamoDBMetadataStore) does not persist the scheme
(just /bucket/path/etc) so this could be feasible. We'd need to spell out in
the MetadataStore API semantics that scheme must be ignored, if that is what we
wanted, or make that behavior a property of the MetadataStore implementation
that we could query at runtime and enforce with some new contract test cases.
Mixing hadoop FS clients on a single bucket could be bad? I.e. if they have
different ways of emulating directories (e.g. how empty directories are
represented and invariants around that).
One idea is to inspect fs.s3.impl at runtime and if set to S3AFileSystem, add
s3 to list of allowed schemes. I'm not sure concurrent access to same bucket
with proprietary s3:// client code and also the apache hadoop s3a:// client is
safe though. That would need an explanation. Do you guys know?
> Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily
> restrictive
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-15314
> URL: https://issues.apache.org/jira/browse/HADOOP-15314
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: DJ Hoffman
> Priority: Major
>
> In version 3.0.0, the checkPath method for dealing with paths prevents us
> from using the s3:// scheme when utilizing S3Guard. However, in our
> core-site.xml we have included
> {noformat}
> <property>
> <name>fs.s3.impl</name>
> <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
> </property>{noformat}
> which should enforce that s3 prefixed paths go through s3a and are properly
> compatible with s3guard. We removed the assertion that paths use the s3a
> scheme (some of our paths use the s3 scheme) and our testing thus far with
> S3Guard enabled have been positive. We believe the assertion in checkPath is
> unnecessary and could be expanded to include the s3 and s3n schemes if not
> dropped altogether or altered in some other way. We're happy to develop and
> test a patch if the community is amenable to the change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
