[
https://issues.apache.org/jira/browse/SYSTEMML-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-1753.
------------------------------------
> OOM on parfor local in-memory result merge
> ------------------------------------------
>
> Key: SYSTEMML-1753
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1753
> Project: SystemML
> Issue Type: Bug
> Reporter: Matthias Boehm
> Assignee: Matthias Boehm
> Fix For: SystemML 1.0
>
>
> Consider a scenario with relatively large parfor result variable (e.g., many
> iterations with relatively large output vector each). There are conditions
> under which the parfor local in-memory result merge runs unnecessarily out of
> memory (as shown below) due to allocating the result in sparse, collecting
> all (sparse) outputs and finally converting this result to dense. For result
> merge without compare, the target number of non-zeros are exactly known,
> allowing us to directly allocate the result in the correct format, which
> reduces memory pressure by more than 2x (dense matrix in sparse format).
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.sysml.runtime.matrix.data.MatrixBlock.allocateDenseBlock(MatrixBlock.java:362)
> at
> org.apache.sysml.runtime.matrix.data.MatrixBlock.sparseToDense(MatrixBlock.java:1136)
> at
> org.apache.sysml.runtime.matrix.data.MatrixBlock.examSparsity(MatrixBlock.java:1019)
> at
> org.apache.sysml.runtime.controlprogram.parfor.ResultMergeLocalMemory.executeSerialMerge(ResultMergeLocalMemory.java:114)
> at
> org.apache.sysml.runtime.controlprogram.ParForProgramBlock.consolidateAndCheckResults(ParForProgramBlock.java:1751)
> at
> org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeLocalParFor(ParForProgramBlock.java:814)
> at
> org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:635)
> at
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
