Merge pull request #220 from rxin/zippart Memoize preferred locations in ZippedPartitionsBaseRDD
so preferred location computation doesn't lead to exponential explosion. This was a problem in GraphX where we have a whole chain of RDDs that are ZippedPartitionsRDD's, and the preferred locations were taking eternity to compute. (cherry picked from commit e36fe55a031d2c01c9d7c5d85965951c681a0c74) Signed-off-by: Reynold Xin <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/87676a6a Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/87676a6a Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/87676a6a Branch: refs/heads/scala-2.10 Commit: 87676a6af2c8fc33c5b5d4e7eb45e3e8558f3c33 Parents: 0780498 9cf7f31 Author: Matei Zaharia <[email protected]> Authored: Fri Dec 6 11:01:42 2013 -0800 Committer: Matei Zaharia <[email protected]> Committed: Fri Dec 6 11:01:42 2013 -0800 ---------------------------------------------------------------------- .../apache/spark/rdd/ZippedPartitionsRDD.scala | 27 ++++++++------------ 1 file changed, 11 insertions(+), 16 deletions(-) ----------------------------------------------------------------------
