[
https://issues.apache.org/jira/browse/SYSTEMML-1423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-1423.
------------------------------------
> OOM on generating ultra-sparse rand data
> ----------------------------------------
>
> Key: SYSTEMML-1423
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1423
> Project: SystemML
> Issue Type: Bug
> Reporter: Matthias Boehm
> Assignee: Matthias Boehm
> Fix For: SystemML 1.0
>
>
> In order to ensure consistency across backends, we first determine the number
> of non-zeros per block and subsequently generate random data accordingly.
> However, in case of ultra-sparse data sets, this temporary array can be
> almost as large as the dataset. Since this memory consumption is unaccounted
> and even required for distributed operations, there are various possible
> scenarios where this would cause OOMs.
> This task aims to solve this issue for all backends, by determining the nnz
> per block in a streaming manner without materialization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
