[
https://issues.apache.org/jira/browse/SYSTEMML-1309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-1309.
------------------------------------
> Parfor spark buffer pool handling
> ---------------------------------
>
> Key: SYSTEMML-1309
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1309
> Project: SystemML
> Issue Type: Sub-task
> Components: APIs, Runtime
> Reporter: Matthias Boehm
> Assignee: Matthias Boehm
> Fix For: SystemML 1.0
>
>
> In contrast to parfor mr jobs, where every task has its own, process-local
> buffer pool, on spark with multi-threaded executors, multiple tasks share a
> common buffer pool. This is advantageous because common inputs are just read
> once. However, it also requires a synchronized buffer pool initialization and
> cleanup per executor. Especially the cleanup (e.g., of created cache
> directories) is tricky because spark does not provide an executor close call.
> Hence, our approach is to use a robust version of deleteOnExit that is
> independent of the exit code and also removes remaining files that are
> unknown during delete registration.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
