[
https://issues.apache.org/jira/browse/SYSTEMML-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-2188.
------------------------------------
Resolution: Fixed
Assignee: Matthias Boehm
Fix Version/s: SystemML 1.1
> Unnecessary evictions on rdd collect
> -------------------------------------
>
> Key: SYSTEMML-2188
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2188
> Project: SystemML
> Issue Type: Sub-task
> Reporter: Matthias Boehm
> Assignee: Matthias Boehm
> Priority: Major
> Fix For: SystemML 1.1
>
>
> For robustness regarding potential OOMs we already have functionality for
> guarded collects that write the RDD to hdfs and read it into memory instead
> of collect because the latter requires twice the memory of a simple read.
> However, there are scenarios, where we collect an RDD and because its size
> exceeds the buffer pool, we immediately evict to local file system in a
> single-threaded manner. This task aims to consolidate this and use the
> guarded collect whenever the data is known to exceed the buffer pool size.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
