Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8294 )
Change subject: IMPALA-6067: Enable s3 access via IAM roles for EC2 VMs ...................................................................... Patch Set 3: > Patch Set 3: > > >[...]Please do answer the question about changing > > the FE tests to just use s3a so that we can remove the hackery > > altogether.[...] > > In short: it depends on your end goal. > If the end goal is to get rid of the s3n: provider and its supporting > menagerie (jets3t, etc.), then yes, it can be done with a small price: a > single negative test in AnalyzeStmtsTest.java will have to go. The code there > in > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java#L3000 > performs a test step that should fail for s3n -- but if I remove the s3n > properties from core-site.xml, this line blows up complaining about missing > credentials. The stack trace reveals that Impala's HdfsUri::analyze() calls > into Hadoop's Path::getFileSystem() to find out what kind of filesystem the > URI points to; this in turn calls down all the way to > org.apache.hadoop.fs.s3.S3Credentials.initialize(S3Credentials.java:74) being > called from Jets3t. > > If your end goal is to remove dummy credentials from core-site.xml.tmpl, this > will probably not be complete success: the s3a: provider behaves in a similar > () though not identical) way. When the front-end tests are run without an S3 > container in sight (e.g. your local dev box with no S3 access set up), > AnalyzeDDLTest's s3a: URIs still require similar dummy credentials, but with > s3a: these can be supplied in environment variables. This means that > core-site.xml[.tmpl] could be cleaned up, but dummy credentials in > envireonment variables would still be there. Hadoop has essentially deprecated s3n and it's removed entirely in Hadoop-3.0. (See HADOOP-14738.) The distribution of Hadoop we're depending on also prefers s3a. I think we can do away with s3n entirely, but I'm happy to take that on as a separate change after yours here. -- To view, visit http://gerrit.cloudera.org:8080/8294 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14cd9d4453a91baad3c379aa7e4944993fca95ae Gerrit-Change-Number: 8294 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: David Knupp <[email protected]> Gerrit-Reviewer: Jim Apple <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Laszlo Gaal <[email protected]> Gerrit-Reviewer: Michael Brown <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Comment-Date: Wed, 22 Nov 2017 21:25:56 +0000 Gerrit-HasComments: No
