GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/2030
[SPARK-3119] Re-implementation of TorrentBroadcast.
This is a re-implementation of TorrentBroadcast, with the following changes:
1. Removes most of the mutable, transient state from TorrentBroadcast (e.g.
totalBytes, num of blocks fetched).
2. Removes TorrentInfo and TorrentBlock
3. Replaces the BlockManager.getSingle call in readObject with a getLocal,
resuling in one less RPC call to the BlockManagerMasterActor to find the
location of the block.
4. Removes the metadata block, resulting in one less block to fetch.
5. Removes an extra memory copy for deserialization (by using Java's
SequenceInputStream).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark torrentBroadcast
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2030.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2030
----
commit c1185cded8224470f06abb4e345df5ebbd87db68
Author: Reynold Xin <[email protected]>
Date: 2014-08-19T06:40:27Z
[SPARK-3119] Re-implementation of TorrentBroadcast.
This is a re-implementation of TorrentBroadcast, with the following changes:
1. Removes most of the mutable, transient state from TorrentBroadcast (e.g.
totalBytes, num of blocks fetched).
2. Removes TorrentInfo and TorrentBlock
3. Replaces the BlockManager.getSingle call in readObject with a getLocal,
resuling in one less RPC call to the BlockManagerMasterActor to find the
location of the block.
4. Removes the metadata block, resulting in one less block to fetch.
5. Removes an extra memory copy for deserialization (by using Java's
SequenceInputStream).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
