[
https://issues.apache.org/jira/browse/SYSTEMML-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm updated SYSTEMML-1627:
-------------------------------------
Affects Version/s: SystemML 0.14
> Mlogreg fails with file not found on MNIST480m and certain mem configs
> ----------------------------------------------------------------------
>
> Key: SYSTEMML-1627
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1627
> Project: SystemML
> Issue Type: Bug
> Affects Versions: SystemML 0.14
> Reporter: Matthias Boehm
>
> Scenario: MultiLogReg over MNIST480m (480M rows x 784, sparse) fails for
> certain memory configurations (where unary operations over 480Mx2
> intermediates run in CP and binary operations in SPARK), with the following
> exception:
> {code}
> Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error
> in program block generated from statement block between lines 261 and 273 --
> Error evaluating instruction:
> SPARK°tak+*°Y·MATRIX·DOUBLE°_mVar432·MATRIX·DOUBLE°1·SCALAR·INT·true°_Var437·SCALAR·DOUBLE
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:322)
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:221)
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:167)
> at
> org.apache.sysml.runtime.controlprogram.WhileProgramBlock.execute(WhileProgramBlock.java:165)
> ... 14 more
> Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does
> not exist:
> hdfs://larry.almaden.ibm.com:8020/user/biuser/scratch_space/_p684936_9.1.44.28/_t0/temp154_56
> at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
> at
> org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:45)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
> {code}
> The root cause is a missing export on guarded parallelize (as introduced in
> the 0.14 release) of cached matrices which have previously been collected
> from input rdds. These matrix objects are not marked dirty and hence not
> exported although they do not have an associated hdfs file yet.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
