[
https://issues.apache.org/jira/browse/SPARK-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14334166#comment-14334166
]
Marcelo Vanzin edited comment on SPARK-5124 at 2/24/15 1:08 AM:
----------------------------------------------------------------
Hi [~zsxwing], I briefly looked over the design and parts of the
implementation, and I have some comments. Overall this looks OK, even if it
kinda does look a lot like akka's actor system, but I don't think there would
be a problem implementing it over a different RPC library.
One thing that I found very confusing is {{RpcEndpoint}} vs.
{{NetworkRpcEndpoint}}. The "R" in RPC means "remote", which sort of implies
networking. I think Reynold touched on this when he talked about using an event
loop for "local actors". Perhaps a more flexible approach would be something
like:
- A generic {{Endpoint}} that defines the {{receive()}} method and other
generic interfaces
- {{RpcEnv}} takes an {{Endpoint}} and a name to register endpoints for remote
connections. Things like "onConnected" and "onDisassociated" become messages
passed to {{receive()}}, kinda like in akka today. This means that there's no
need for a specialized {{RpcEndpoint}} interface.
- "local" Endpoints could be exposed directly, without the need for an
{{RpcEnv}}. For example, as a getter in {{SparkEnv}}. You could have some
wrapper to expose {{send()}} and {{ask()}} so that the client interface looks
similar for remote and local endpoints.
- The default {{Endpoint}} has no thread-safety guarantees. You can wrap an
{{Endpoint}} in an {{EventLoop}} if you want messages to be handled using a
queue, or synchronize your {{receive()}} method (although that can block the
dispatcher thread, which could be bad). But this would easily allow actors to
process multiple messages concurrently if desired.
What do you think?
was (Author: vanzin):
Hi [~zsxwing], I briefly looked over the design and parts of the
implementation, and I have some comments. Overall this looks OK, even if it
kinda does look a lot like akka's actor system, but I don't think there would
be a problem implementing it over a different RPC library.
One thing that I found very confusing is {{RpcEndpoint}} vs.
{{NetworkRpcEndpoint}}. The "R" in RPC means "remote", which sort of implies
networking. I think Reynold touched on this when he talked about using an event
loop for "local actors". Perhaps a more flexible approach would be something
like:
- A generic {{Endpoint}} that defines the {{receive()}} method and other
generic interfaces
- {{RpcEnv}} takes an {{Endpoint}} and a name to register endpoints for remote
connections. Things like "onConnected" and "onDisassociated" become messages
passed to {{receive()}}, kinda like in akka today. This means that there's no
need for a specialized {{RpcEndpoint}} interface.
- "local" Endpoints could be exposed directly, without the need for an
{{RpcEnv}}. For example, as a getter in {{SparkEnv}}. You could have some
wrapper to expose {{send()}} and {{ask()}} so that the client interface looks
similar for remote and local endpoints.
- The default {{Endpoint}} has no thread-safety guarantees. You can wrap an
{{Endpoint}} in an {{EventLoop}} if you want messages to be handled using a
queue, or synchronize your {{receive()}} method (although that can block the
dispatcher thread, which could be bad). But this would easily allow actors to
process multiple messages concurrently is desired.
What do you think?
> Standardize internal RPC interface
> ----------------------------------
>
> Key: SPARK-5124
> URL: https://issues.apache.org/jira/browse/SPARK-5124
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Reporter: Reynold Xin
> Assignee: Shixiong Zhu
> Attachments: Pluggable RPC - draft 1.pdf, Pluggable RPC - draft 2.pdf
>
>
> In Spark we use Akka as the RPC layer. It would be great if we can
> standardize the internal RPC interface to facilitate testing. This will also
> provide the foundation to try other RPC implementations in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
