[ https://issues.apache.org/jira/browse/HADOOP-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403690#comment-16403690 ]
Steve Loughran commented on HADOOP-15314: ----------------------------------------- aaron: w.r.t mixing clients, I was thinking "different versions of ASF hadoop s3a client" What [~djhoffman] has probably done is just registered S3A as the implementor of s3://, so that any code from EMR will work without changing URLs. You are right, trying to share a bucket with EMR s3 and s3a could cause problems, but if it's just the fs schema in the JVM, that's a non-issue. > Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily > restrictive > ----------------------------------------------------------------------------------------- > > Key: HADOOP-15314 > URL: https://issues.apache.org/jira/browse/HADOOP-15314 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.0.0 > Reporter: DJ Hoffman > Priority: Major > > In version 3.0.0, the checkPath method for dealing with paths prevents us > from using the s3:// scheme when utilizing S3Guard. However, in our > core-site.xml we have included > {noformat} > <property> > <name>fs.s3.impl</name> > <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value> > </property>{noformat} > which should enforce that s3 prefixed paths go through s3a and are properly > compatible with s3guard. We removed the assertion that paths use the s3a > scheme (some of our paths use the s3 scheme) and our testing thus far with > S3Guard enabled have been positive. We believe the assertion in checkPath is > unnecessary and could be expanded to include the s3 and s3n schemes if not > dropped altogether or altered in some other way. We're happy to develop and > test a patch if the community is amenable to the change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org