GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/2240
[SPARK-3019] Pluggable block transfer interface (BlockTransferService)
This pull request creates a new BlockTransferService interface for block
fetch/upload.
The pull request also refactors the existing ConnectionManager to implement
BlockTransferService (CMBlockTransferService).
TODOs that should be part of this PR:
- [ ] Bring this up to date with master
- [ ] Look at the removed unit tests and see if they still make sense to be
added back
TODOs that should be separate PRs:
- [ ] Implement NettyBlockTransferService
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark blockTransferService
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2240.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2240
----
commit 9ef279ccfc6b8fedf4e3eb24b9cd7b5bc4ce7424
Author: Reynold Xin <[email protected]>
Date: 2014-08-28T21:20:12Z
Initial refactoring to move ConnectionManager to use the
BlockTransferService.
commit d8d595c161fd215c3b2e39f5130df94077880fd3
Author: Reynold Xin <[email protected]>
Date: 2014-08-28T21:21:57Z
Merge branch 'master' of github.com:apache/spark into blockTransferService
Conflicts:
core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala
core/src/main/scala/org/apache/spark/storage/BlockManager.scala
core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
commit ae05fcd47b52f0da3db669d74888f3cc0780f33b
Author: Reynold Xin <[email protected]>
Date: 2014-09-02T08:06:26Z
Updated tests, although DistributedSuite is hanging.
commit 98c668ae98e6e7d3d22504b3607527ad162356fc
Author: Reynold Xin <[email protected]>
Date: 2014-09-02T22:03:58Z
Added failure handling and fixed unit tests.
commit 07ccf0db4c250c0160e42cf6efb030e7502d30e7
Author: Reynold Xin <[email protected]>
Date: 2014-09-02T22:12:54Z
Added init check to CMBlockTransferService.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
