steveloughran opened a new pull request #685: HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true URL: https://github.com/apache/hadoop/pull/685 This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start. Production code changes: * S3AFileStatus Adds "true" to the superclass's encrypted flag during construction. Tests * Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts? * Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads * And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up Docs: * Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch. Contributed by Steve Loughran. Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
