[
https://issues.apache.org/jira/browse/HADOOP-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404986#comment-16404986
]
DJ Hoffman commented on HADOOP-15314:
-------------------------------------
We're not dealing with EMR at all. We're running into this problem due to
having many platforms, all of which share the same set of URL constants, some
of which go through Hadoop and therefore via s3a, and others of which access s3
via other AWS APIs. We're exploring what we can do internally, but we
definitely appreciate you taking the time to respond here, and based on your
comments so far we do think that we're not going to run into any large
compatibility issues, which is a sigh of relief from our end.
> Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily
> restrictive
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-15314
> URL: https://issues.apache.org/jira/browse/HADOOP-15314
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: DJ Hoffman
> Priority: Major
>
> In version 3.0.0, the checkPath method for dealing with paths prevents us
> from using the s3:// scheme when utilizing S3Guard. However, in our
> core-site.xml we have included
> {noformat}
> <property>
> <name>fs.s3.impl</name>
> <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
> </property>{noformat}
> which should enforce that s3 prefixed paths go through s3a and are properly
> compatible with s3guard. We removed the assertion that paths use the s3a
> scheme (some of our paths use the s3 scheme) and our testing thus far with
> S3Guard enabled have been positive. We believe the assertion in checkPath is
> unnecessary and could be expanded to include the s3 and s3n schemes if not
> dropped altogether or altered in some other way. We're happy to develop and
> test a patch if the community is amenable to the change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
