[
https://issues.apache.org/jira/browse/SYSTEMML-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999702#comment-15999702
]
Matthias Boehm commented on SYSTEMML-1588:
------------------------------------------
cc [~nakul02]
> Perftest: Datagen descriptive statistics fails on parfor result merge
> ---------------------------------------------------------------------
>
> Key: SYSTEMML-1588
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1588
> Project: SystemML
> Issue Type: Bug
> Reporter: Matthias Boehm
>
> The perftest data generation script for descriptive statistics fails for the
> 80GB scenario with the following stacktrace
> {code}
> Caused by: org.apache.sysml.runtime.DMLRuntimeException:
> org.apache.sysml.runtime.DMLRuntimeException:
> org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
> hdfs://<node>:<port>/user/biuser/scratch_space/_p123456789_1.2.34.56/_t16/temp863_830
> at
> org.apache.sysml.runtime.controlprogram.parfor.ResultMergeRemoteSpark.executeParallelMerge(ResultMergeRemoteSpark.java:116)
> at
> org.apache.sysml.runtime.controlprogram.parfor.ResultMergeRemoteSpark.executeSerialMerge(ResultMergeRemoteSpark.java:71)
> at
> org.apache.sysml.runtime.controlprogram.ParForProgramBlock.consolidateAndCheckResults(ParForProgramBlock.java:1756)
> at
> org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeLocalParFor(ParForProgramBlock.java:818)
> at
> org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:639)
> ... 14 more
> {code}
> This issue is caused by incorrect export handling of inputs which only works
> for existing inputs on hdfs or dirty in-memory objects but not for pending
> rdd operations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
