Matthias Boehm created SYSTEMML-1753:
----------------------------------------
Summary: OOM on parfor local in-memory result merge
Key: SYSTEMML-1753
URL: https://issues.apache.org/jira/browse/SYSTEMML-1753
Project: SystemML
Issue Type: Bug
Reporter: Matthias Boehm
Consider a scenario with relatively large parfor result variable (e.g., many
iterations with relatively large output vector each). There are conditions
under which the parfor local in-memory result merge runs unnecessarily out of
memory (as shown below) due to allocating the result in sparse, collecting all
(sparse) outputs and finally converting this result to dense. For result merge
without compare, the target number of non-zeros are exactly known, allowing us
to directly allocate the result in the correct format, which reduces memory
pressure by more than 2x (dense matrix in sparse format).
{code}
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at
org.apache.sysml.runtime.matrix.data.MatrixBlock.allocateDenseBlock(MatrixBlock.java:362)
at
org.apache.sysml.runtime.matrix.data.MatrixBlock.sparseToDense(MatrixBlock.java:1136)
at
org.apache.sysml.runtime.matrix.data.MatrixBlock.examSparsity(MatrixBlock.java:1019)
at
org.apache.sysml.runtime.controlprogram.parfor.ResultMergeLocalMemory.executeSerialMerge(ResultMergeLocalMemory.java:114)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.consolidateAndCheckResults(ParForProgramBlock.java:1751)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeLocalParFor(ParForProgramBlock.java:814)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:635)
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
