[
https://issues.apache.org/jira/browse/HADOOP-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564394#comment-16564394
]
Steve Loughran commented on HADOOP-15314:
-----------------------------------------
Reviewing this again.
I think you are both right: we don't need to be checking the schema. All we
care about is a path is being registered -it is up to the owning FS to worry
about schema equivalence, because path qualification should have done that
already.
However, the static
{{PathMetadataDynamoDBTranslation.itemToPathMetadata(Item)}} method is
prepending that s3a scheme when it builds paths, so removing the check in
{{checkPath}} will only get the path in: when you get FileStatus entries back
they'll be of the s3a:// scheme.
This issue here then is really one of "support URI schemes other than S3a in
DDB", as the instance of the metastore will need to know which scheme to use
when creating paths from items.
> Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily
> restrictive
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-15314
> URL: https://issues.apache.org/jira/browse/HADOOP-15314
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: DJ Hoffman
> Priority: Major
>
> In version 3.0.0, the checkPath method for dealing with paths prevents us
> from using the s3:// scheme when utilizing S3Guard. However, in our
> core-site.xml we have included
> {noformat}
> <property>
> <name>fs.s3.impl</name>
> <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
> </property>{noformat}
> which should enforce that s3 prefixed paths go through s3a and are properly
> compatible with s3guard. We removed the assertion that paths use the s3a
> scheme (some of our paths use the s3 scheme) and our testing thus far with
> S3Guard enabled have been positive. We believe the assertion in checkPath is
> unnecessary and could be expanded to include the s3 and s3n schemes if not
> dropped altogether or altered in some other way. We're happy to develop and
> test a patch if the community is amenable to the change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
