[ https://issues.apache.org/jira/browse/SYSTEMML-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150232#comment-16150232 ]
Matthias Boehm commented on SYSTEMML-1879: ------------------------------------------ FYI [~Tenma] - this will also help for the deep learning scenarios (weights are just read once per process not once per core, and probably less GC overhead). > Parfor remote spark w/ reuse of shared inputs > --------------------------------------------- > > Key: SYSTEMML-1879 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1879 > Project: SystemML > Issue Type: Sub-task > Components: APIs, Runtime > Reporter: Matthias Boehm > Fix For: SystemML 1.0 > > > Currently, we read shared inputs redundantly in each parfor worker. This > causes redundant read and is unnecessarily memory-inefficient. > This task aims to read shared inputs once per process and reuse them across > threads. The most elegant way of handling this is to reuse initially parsed > symbol table entries (instances of matrix objects), except for result > variables. Then the result happens automatically over the shared per-process > buffer pool. -- This message was sent by Atlassian JIRA (v6.4.14#64029)