Merge pull request #57 from aarondav/bid Refactor BlockId into an actual type
Converts all of our BlockId strings into actual BlockId types. Here are some advantages of doing this now: + Type safety + Code clarity - it's now obvious what the key of a shuffle or rdd block is, for instance. Additionally, appearing in tuple/map type signatures is a big readability bonus. A Seq[(String, BlockStatus)] is not very clear. Further, we can now use more Scala features, like matching on BlockId types. + Explicit usage - we can now formally tell where various BlockIds are being used (without doing string searches); this makes updating current BlockIds a much clearer process, and compiler-supported. (I'm looking at you, shuffle file consolidation.) + It will only get harder to make this change as time goes on. Downside is, of course, that this is a very invasive change touching a lot of different files, which will inevitably lead to merge conflicts for many. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/3b11f43e Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/3b11f43e Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/3b11f43e Branch: refs/heads/master Commit: 3b11f43e36e2aca2346db7542c52fcbbeee70da2 Parents: 9979690 4a45019 Author: Reynold Xin <r...@apache.org> Authored: Mon Oct 14 14:20:01 2013 -0700 Committer: Reynold Xin <r...@apache.org> Committed: Mon Oct 14 14:20:01 2013 -0700 ---------------------------------------------------------------------- .../spark/network/netty/FileClientHandler.java | 3 +- .../spark/network/netty/FileServerHandler.java | 6 +- .../apache/spark/BlockStoreShuffleFetcher.scala | 11 +- .../scala/org/apache/spark/CacheManager.scala | 8 +- .../spark/broadcast/BitTorrentBroadcast.scala | 4 +- .../apache/spark/broadcast/HttpBroadcast.scala | 11 +- .../apache/spark/broadcast/TreeBroadcast.scala | 8 +- .../org/apache/spark/executor/Executor.scala | 4 +- .../apache/spark/network/netty/FileHeader.scala | 22 ++-- .../spark/network/netty/ShuffleCopier.scala | 27 ++--- .../spark/network/netty/ShuffleSender.scala | 9 +- .../scala/org/apache/spark/rdd/BlockRDD.scala | 6 +- .../apache/spark/scheduler/DAGScheduler.scala | 6 +- .../org/apache/spark/scheduler/TaskResult.scala | 3 +- .../spark/serializer/KryoSerializer.scala | 10 +- .../apache/spark/storage/BlockException.scala | 2 +- .../spark/storage/BlockFetcherIterator.scala | 24 ++-- .../org/apache/spark/storage/BlockId.scala | 96 ++++++++++++++++ .../org/apache/spark/storage/BlockManager.scala | 95 +++++++--------- .../spark/storage/BlockManagerMaster.scala | 8 +- .../spark/storage/BlockManagerMasterActor.scala | 21 ++-- .../spark/storage/BlockManagerMessages.scala | 16 +-- .../spark/storage/BlockManagerWorker.scala | 4 +- .../org/apache/spark/storage/BlockMessage.scala | 38 +++---- .../spark/storage/BlockMessageArray.scala | 7 +- .../spark/storage/BlockObjectWriter.scala | 2 +- .../org/apache/spark/storage/BlockStore.scala | 14 +-- .../org/apache/spark/storage/DiskStore.scala | 37 +++--- .../org/apache/spark/storage/MemoryStore.scala | 34 +++--- .../spark/storage/ShuffleBlockManager.scala | 16 +-- .../org/apache/spark/storage/StorageUtils.scala | 47 ++++---- .../apache/spark/storage/ThreadingTest.scala | 6 +- .../org/apache/spark/ui/storage/RDDPage.scala | 23 ++-- .../org/apache/spark/CacheManagerSuite.scala | 12 +- .../org/apache/spark/CheckpointSuite.scala | 6 +- .../org/apache/spark/DistributedSuite.scala | 16 +-- .../spark/scheduler/DAGSchedulerSuite.scala | 15 +-- .../cluster/TaskResultGetterSuite.scala | 3 +- .../org/apache/spark/storage/BlockIdSuite.scala | 114 +++++++++++++++++++ .../spark/storage/BlockManagerSuite.scala | 102 +++++++++-------- .../spark/streaming/NetworkInputTracker.scala | 11 +- .../streaming/dstream/NetworkInputDStream.scala | 14 +-- .../streaming/dstream/RawInputDStream.scala | 4 +- .../streaming/receivers/ActorReceiver.scala | 4 +- 44 files changed, 544 insertions(+), 385 deletions(-) ----------------------------------------------------------------------
