[
https://issues.apache.org/jira/browse/SYSTEMML-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16543682#comment-16543682
]
Matthias Boehm commented on SYSTEMML-2419:
------------------------------------------
Indeed we usually run Spark with 1 executor per node and k cores per executor,
which means that k tasks will run concurrently in the same executor JVM. I
think it would be a natural first step to similarly run k paramserv workers per
executor - this will happen automatically if you use the current {{foreach}}
approach.
Code generation is an advanced feature (by default still disabled) that
automatically fuses multiple operators into composite operators by generating
the respective source code, as well as compiling and loading the respective
classes. To ensure robustness with regarding to heterogeneous JVMs in a
cluster, we actually ship the source code and compile the classes at each
executor. The classes are compiled and loaded and subsequently shared across
tasks (concurrent and subsequent) via a basic class cache.
> Setup and cleanup of remote workers
> -----------------------------------
>
> Key: SYSTEMML-2419
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2419
> Project: SystemML
> Issue Type: Sub-task
> Reporter: LI Guobao
> Assignee: LI Guobao
> Priority: Major
>
> In the context of distributed spark env, we need to firstly ship the
> necessary functions and variables to the remote workers and then to
> initialize and register the cleanup of buffer pool for each remote worker.
> All these are inspired by the parfor implementation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
