GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/3974
[SPARK-5124][Core][WIP] A standard internal RPC interface and an Akka
implementation
This PR added a standard internal RPC interface for Spark and an Akka
implementation based on @rxin's initial work. See the design document for more
details. I replaced the most usages of Akka with the new RPC interface to prove
it can cover all our requirements.
I sent this PR for discussing the new RPC interface. Please focus on
1. core/src/main/scala/org/apache/spark/rpc/RpcEnv.scala
2. core/src/main/scala/org/apache/spark/rpc/akka/AkkaRpcEnv.scala
and ignore naming issues in other files (I avoid to change them for a clean
diff)
Some problems need to discuss:
1. We create ActorSystem without host and port. The Actor addresses in such
ActorSystem don't contain host and port. The new RPC mechanism requires that a
RpcEnv must bind to a host and port. Is it too strict?
2. Because Sreaming uses ActorSystem in SparkEnv, the current PR cannot
totally decouple Akka. So should Streaming create ActorSystme by itself, or use
the new RPC interface?
TODO:
1. Update the variable names from `***Actor` to `***Endpoint`
2. Totally decouple Akka from Spark and make the Akka dependency optional.
3. A test framework.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark rpc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3974.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3974
----
commit 63d2d8d936e3d3a532f39ed0c375e134361f2a34
Author: Reynold Xin <[email protected]>
Date: 2014-12-18T23:58:09Z
rpc temp checkin
commit 85bafc483ca3172c2a9235bf5ad21d3166c454c9
Author: Reynold Xin <[email protected]>
Date: 2014-12-19T20:57:59Z
Merge branch 'master' into rpc
Conflicts:
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
core/src/main/scala/org/apache/spark/executor/Executor.scala
commit c9f2cc929777befb1ce428c4d90293edda13a684
Author: Reynold Xin <[email protected]>
Date: 2014-12-20T09:26:03Z
First version that ran.
commit bdc67b56033c10a4e3bff0ab7999886ff39a5481
Author: Reynold Xin <[email protected]>
Date: 2014-12-24T18:58:00Z
Minor update.
commit 1a195d5f72e4e1d9067219282ed637544278753e
Author: Reynold Xin <[email protected]>
Date: 2014-12-24T18:58:24Z
Merge branch 'master' into rpc
commit 2f4b9d864ade386b171ae7983449ae86bc55eae6
Author: zsxwing <[email protected]>
Date: 2014-12-25T03:41:43Z
abstract class => trait
commit 0f7f0325ecef99e0cf758c0675c44bddc46a0541
Author: zsxwing <[email protected]>
Date: 2014-12-25T04:03:50Z
Move Akka classes to org.apache.spark.rpc.akka
commit 2d2cba373ee3a0a93c34514c9cd5b26f37c8f06e
Author: zsxwing <[email protected]>
Date: 2014-12-25T08:56:03Z
Update the APIs to support to get the `sender` RpcEndPointRef when sending
a message and add some unit tests
commit 1fc4a0176021e1397b0e3639f9d97f3f236a5329
Author: zsxwing <[email protected]>
Date: 2014-12-25T09:15:08Z
endPoint => endpoint
commit 0627986d232e0257f5ee12a6b3340af9713aff01
Author: zsxwing <[email protected]>
Date: 2014-12-26T03:28:27Z
Remove log api from RpcEndpoint trait
commit ee988d325ed726886a785ceef4862cf95847f688
Author: zsxwing <[email protected]>
Date: 2014-12-26T03:37:28Z
Throw an exception if RpcEndpoint is not in RpcEnv
commit 2e99b4a7cc688119449ce523c90e9ce3c702738f
Author: zsxwing <[email protected]>
Date: 2014-12-26T10:48:07Z
Add tests for HeartbeatReceiver
commit 73db9e56fe86ac1a3ecbb7d16a80c1a3dd5b5011
Author: zsxwing <[email protected]>
Date: 2014-12-26T11:13:53Z
Minor changes
commit 19053ca1ff95d6c001befe6107de5e15f96e9f13
Author: zsxwing <[email protected]>
Date: 2014-12-26T12:07:30Z
Add test for ExecutorActor
commit c8d8ac1e8c358c32908ea745098afdbd512ba323
Author: zsxwing <[email protected]>
Date: 2014-12-26T15:00:17Z
Merge branch 'master' into rpc
commit a435eb234a6be4fdbdb1f899ac4ffb22d031df72
Author: zsxwing <[email protected]>
Date: 2014-12-27T10:11:51Z
Fault tolerance for RpcEndpoint
commit 12bc1c1ccc4f10a2fe5fa24676efc7b39fb7cc9b
Author: zsxwing <[email protected]>
Date: 2014-12-27T10:13:43Z
Merge branch 'master' into rpc
commit 2636664922701272408a6af487125c260c6ad416
Author: zsxwing <[email protected]>
Date: 2014-12-29T08:08:39Z
Change CoarseGrainedExecutorBackend to a RpcEndpoint
commit 85cfb333fe157d6eebcb4292fb0abe5af8624969
Author: zsxwing <[email protected]>
Date: 2014-12-29T09:37:49Z
Change CoarseMesosSchedulerBackend to use RpcEndpoint
commit 21586d056f5e37f04b87cf28631eefe35fb47bde
Author: zsxwing <[email protected]>
Date: 2014-12-30T05:57:14Z
Merge branch 'master' into rpc
commit 8e561b49cbf48519680cbf5809ed699cf910eb61
Author: zsxwing <[email protected]>
Date: 2014-12-30T07:19:34Z
Change MapOutputTrackerMasterActor to use RpcEndpoint
commit a067228c1188e202fbdc49eaf1578b87f82183fc
Author: zsxwing <[email protected]>
Date: 2014-12-30T07:23:14Z
Fix the code style
commit b13fbd9e2e29153cd5eea533a791dff47b1b9ca7
Author: zsxwing <[email protected]>
Date: 2014-12-30T09:10:43Z
Change DAGScheduler to use RpcEndpoint
commit 0b17ccd883168ad5313b6e34e64a467c6afed5cf
Author: zsxwing <[email protected]>
Date: 2014-12-31T07:52:55Z
Merge branch 'master' into rpc
commit 595fb6194134f465825730f9d1c98ef5735253d7
Author: zsxwing <[email protected]>
Date: 2015-01-04T02:54:03Z
Merge branch 'master' into rpc
commit 811b6b8f6a27e950fef456476f35221281ee8446
Author: zsxwing <[email protected]>
Date: 2015-01-04T04:17:32Z
Change BlockManager to use RpcEndpoint
commit c1d3df82a7f9f055e675db067a868c47eea7c2f3
Author: zsxwing <[email protected]>
Date: 2015-01-04T09:55:14Z
Change WorkerWatcher to use RpcEndpoint
commit 20682d1fd4a211b58742f0effeb611df0a0d00ba
Author: zsxwing <[email protected]>
Date: 2015-01-04T11:46:50Z
Change Master to use RpcEndpoint
commit acb18fbb0a3146170ddd6be01255182d29775b27
Author: zsxwing <[email protected]>
Date: 2015-01-05T05:30:11Z
Merge branch 'master' into rpc
Conflicts:
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
commit 7b43e395e0528a0184cc60e80b69226cd7844a37
Author: zsxwing <[email protected]>
Date: 2015-01-05T08:57:38Z
Add RpcAddress and change AppClient to use RpcEndpoint
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
