Merge pull request #149 from tgravescs/fixSecureHdfsAccess Fix secure hdfs access for spark on yarn
https://github.com/apache/incubator-spark/pull/23 broke secure hdfs access. Not sure if it works with secure hdfs on standalone. Fixing it at least for spark on yarn. The broadcasting of jobconf change also broke secure hdfs access as it didn't take into account things calling the getPartitions before sparkContext is initialized. The DAGScheduler does this as it tries to getShuffleMapStage. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/8af99f23 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/8af99f23 Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/8af99f23 Branch: refs/heads/master Commit: 8af99f2356ed19fe43d722ada02f8802cbd46d40 Parents: 72a601e 13a1950 Author: Matei Zaharia <[email protected]> Authored: Sat Nov 9 13:48:00 2013 -0800 Committer: Matei Zaharia <[email protected]> Committed: Sat Nov 9 13:48:00 2013 -0800 ---------------------------------------------------------------------- .../apache/spark/deploy/SparkHadoopUtil.scala | 21 +++++++++++++++----- .../scala/org/apache/spark/rdd/HadoopRDD.scala | 2 ++ 2 files changed, 18 insertions(+), 5 deletions(-) ----------------------------------------------------------------------
