[
https://issues.apache.org/jira/browse/SYSTEMML-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16535629#comment-16535629
]
Matthias Boehm commented on SYSTEMML-2419:
------------------------------------------
Yes, this can happen if the input sizes to functions are unknown. However,
there is a simple fix. Whenever we force parfor into {{MR}} or {{SPARK}}
execution modes, we use {{ParForProgramBlock.checkMRAndRecompileToCP}} to
compile all operations to CP. Maybe you could move this function to a central
place (e.g., {{Recompiler}}) and call it both from the paramserv and parfor
functionality.
> Setup and cleanup of remote workers
> -----------------------------------
>
> Key: SYSTEMML-2419
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2419
> Project: SystemML
> Issue Type: Sub-task
> Reporter: LI Guobao
> Assignee: LI Guobao
> Priority: Major
>
> In the context of distributed spark env, we need to firstly ship the
> necessary functions and variables to the remote workers and then to
> initialize and register the cleanup of buffer pool for each remote worker.
> All these are inspired by the parfor implementation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
