Merge pull request #139 from aarondav/shuffle-next Never store shuffle blocks in BlockManager
After the BlockId refactor (PR #114), it became very clear that ShuffleBlocks are of no use within BlockManager (they had a no-arg constructor!). This patch completely eliminates them, saving us around 100-150 bytes per shuffle block. The total, system-wide overhead per shuffle block is now a flat 8 bytes, excluding state saved by the MapOutputTracker. Note: This should *not* be merged directly into 0.8.0 -- see #138 Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/81065321 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/81065321 Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/81065321 Branch: refs/heads/master Commit: 81065321c073fa35028b8aa110704385cba6d5b2 Parents: 0b26a39 93c9084 Author: Reynold Xin <[email protected]> Authored: Mon Nov 4 20:47:14 2013 -0800 Committer: Reynold Xin <[email protected]> Committed: Mon Nov 4 20:47:14 2013 -0800 ---------------------------------------------------------------------- .../org/apache/spark/storage/BlockInfo.scala | 18 +----------------- .../org/apache/spark/storage/BlockManager.scala | 10 ++-------- .../apache/spark/storage/BlockObjectWriter.scala | 12 +----------- 3 files changed, 4 insertions(+), 36 deletions(-) ----------------------------------------------------------------------
