[
https://issues.apache.org/jira/browse/SYSTEMML-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LI Guobao updated SYSTEMML-2420:
--------------------------------
Description: It aims to implement the parameter exchange between ps and
workers. We could leverage spark RPC to setup a ps endpoint in driver node
which means that the ps service could be discovered by workers in the network.
And then the workers could invoke the pull/push method via RPC using the
registered endpoint of ps service. Hence, in details, this tasks consists of
registering the ps endpoint in spark rpc framework and using rpc to invoke
target method in worker side. We can learn that the spark rpc is implemented in
Scala. Hence we need to wrap them in in order to be used in Java. Overall, we
could register the ps service with _RpcEndpoint_ and invoke the service with
_RpcEndpointRef_. (was: It aims to implement the parameter exchange between ps
and workers. We could leverage spark RPC to setup a ps endpoint in driver node
which means that the ps service could be discovered by workers in the network.
And then the workers could invoke the pull/push method via RPC using the
registered endpoint of ps service. Hence, in details, this tasks consists of
registering the ps endpoint in spark rpc framework and using rpc to invoke
target method in worker side. We can learn that the spark rpc is implemented in
Scala. But we could easily wrap them in java class to be reused. And then the
two methods (push, pull) could be wrapped into this defined endpoint.)
> Communication between ps and workers
> ------------------------------------
>
> Key: SYSTEMML-2420
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2420
> Project: SystemML
> Issue Type: Sub-task
> Reporter: LI Guobao
> Assignee: LI Guobao
> Priority: Major
>
> It aims to implement the parameter exchange between ps and workers. We could
> leverage spark RPC to setup a ps endpoint in driver node which means that the
> ps service could be discovered by workers in the network. And then the
> workers could invoke the pull/push method via RPC using the registered
> endpoint of ps service. Hence, in details, this tasks consists of registering
> the ps endpoint in spark rpc framework and using rpc to invoke target method
> in worker side. We can learn that the spark rpc is implemented in Scala.
> Hence we need to wrap them in in order to be used in Java. Overall, we could
> register the ps service with _RpcEndpoint_ and invoke the service with
> _RpcEndpointRef_.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
