[
https://issues.apache.org/jira/browse/SYSTEMML-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LI Guobao updated SYSTEMML-2420:
--------------------------------
Description: It aims to implement the parameter exchange between ps and
workers. We could leverage spark RPC to setup a ps endpoint in driver node
which means that the ps service could be discovered by workers in the network.
And then the workers could invoke the pull/push method via RPC using the
registered endpoint of ps service. Hence, in details, this tasks consists of
registering the ps endpoint in spark rpc framework and using rpc to invoke
target method in worker side. We can learn that the spark rpc is implemented in
Scala. But we could easily wrap them in java class to be reused. And then the
two methods (push, pull) could be wrapped into this defined endpoint. (was: It
aims to implement the parameter exchange between ps and workers. We could
leverage spark RPC to setup a ps endpoint in driver node which means that the
ps service could be discovered by workers in the network. And then the workers
could invoke the pull/push method via RPC using the registered endpoint of ps
service. Hence, in details, this tasks consists of registering the ps endpoint
in spark rpc framework and using rpc to invoke target method in worker side. We
can learn that the spark rpc is implemented in Scala. But we could easily wrap
them in java class to be reused.)
> Communication between ps and workers
> ------------------------------------
>
> Key: SYSTEMML-2420
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2420
> Project: SystemML
> Issue Type: Sub-task
> Reporter: LI Guobao
> Assignee: LI Guobao
> Priority: Major
>
> It aims to implement the parameter exchange between ps and workers. We could
> leverage spark RPC to setup a ps endpoint in driver node which means that the
> ps service could be discovered by workers in the network. And then the
> workers could invoke the pull/push method via RPC using the registered
> endpoint of ps service. Hence, in details, this tasks consists of registering
> the ps endpoint in spark rpc framework and using rpc to invoke target method
> in worker side. We can learn that the spark rpc is implemented in Scala. But
> we could easily wrap them in java class to be reused. And then the two
> methods (push, pull) could be wrapped into this defined endpoint.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
