[
https://issues.apache.org/jira/browse/SYSTEMML-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LI Guobao updated SYSTEMML-2420:
--------------------------------
Attachment: systemml_rpc_class_diagram.png
> Communication between ps and workers
> ------------------------------------
>
> Key: SYSTEMML-2420
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2420
> Project: SystemML
> Issue Type: Sub-task
> Reporter: LI Guobao
> Assignee: LI Guobao
> Priority: Major
> Attachments: systemml_rpc_2_seq_diagram.png,
> systemml_rpc_class_diagram.png, systemml_rpc_sequence_diagram.png
>
>
> It aims to implement the parameter exchange between ps and workers. We could
> leverage netty framework to implement our own Rpc framework. In general, the
> netty {{TransportClient}} and {{TransportServer}} provides the sending and
> receiving service for ps and workers. Extending the {{RpcHandler}} allows to
> invoke the corresponding ps method (i.e., push/pull method) by handling the
> different input Rpc call object. And then the {{SparkPsProxy}} wrapping
> {{TransportClient}} allows the workers to execute the push/pull call to
> server. At the same time, the ps netty server also provides the file
> repository service which allows the workers to download the partitioned
> training data, so that the workers could rebuild the matrix object with the
> transfered file instead of broadcasting all the files with spark which are
> not all necessary for each worker.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
