LI Guobao created SYSTEMML-2465:
-----------------------------------
Summary: Keep data consistency for a pre-trained model
Key: SYSTEMML-2465
URL: https://issues.apache.org/jira/browse/SYSTEMML-2465
Project: SystemML
Issue Type: Sub-task
Reporter: LI Guobao
Assignee: LI Guobao
In distributed spark backend, pass a given pre-trained model to the paramserv
function may cause the data inconsistency. Because the pre-trained model would
be cached in driver's memory. In this case, when kicking off the paramserv
func, the workers firstly will try to read the data from HDFS where the dirty
data in pre-trained model has not been persisted. This leads to a
inconsistency. So the idea is to export the dirty data to HDFS before kicking
off the remote workers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
