toddlipcon opened a new pull request #40: TEZ-1348. Fix local mode with non-local default FS URL: https://github.com/apache/tez/pull/40 This fixes local mode to be able to run when fs.defaultFS is not local. The main change is that, when setting up the working directory for the AM, we need to consider that the staging directory may not be on the local file system, and appropriately construct the right FileSystem instances to copy the data into the working directory. This also fixes the loading of the conf PB file to come from that same local working directory instead of attempting to read it off the staging directory. Along the way I did a bit of cleanup: - simplified TezUtilsInternal.readUserSpecifiedTezConfiguration to use a try-with-resources and to throw an exception instead of returning null if the file doesn't exist. All of the call sites of this method assumed a non-null return value, so any case that it would have returned null previously would have resulted in an NPE anyway. - removed a commented-out variant of the same method - removed LocalClient.getPluginDescriptorInfo. This had some unnecessary complexity and checks and the necessary code takes just one line at the call site. To test, I modified the existing TestLocalMode to parameterize it: it now runs once with fs.defaultFS as the local filesystem, and once with it as HDFS. The modified test fails prior to the patch and passes now.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
