Matthias Boehm created SYSTEMML-2188:
----------------------------------------
Summary: Unnecessary evictions on rdd collect
Key: SYSTEMML-2188
URL: https://issues.apache.org/jira/browse/SYSTEMML-2188
Project: SystemML
Issue Type: Sub-task
Reporter: Matthias Boehm
For robustness regarding potential OOMs we already have functionality for
guarded collects that write the RDD to hdfs and read it into memory instead of
collect because the latter requires twice the memory of a simple read. However,
there are scenarios, where we collect an RDD and because its size exceeds the
buffer pool, we immediately evict to local file system in a single-threaded
manner. This task aims to consolidate this and use the guarded collect whenever
the data is known to exceed the buffer pool size.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
