[ https://issues.apache.org/jira/browse/SYSTEMML-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090325#comment-16090325 ]
Fei Hu commented on SYSTEMML-1774: ---------------------------------- The errors when running the [distributed MNIST LeNet example] | https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet_distrib_sgd.dml] at the local machine: {code:java} Exception in thread "main" org.apache.sysml.api.mlcontext.MLContextException: Exception when executing script at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:320) at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:293) at org.apache.sysml.api.mlcontext.Script.execute(Script.java:717) at org.apache.sysml.scripts.nn.examples.Mnist_lenet_distrib_sgd.train(Mnist_lenet_distrib_sgd.java) at org.apache.sysml.examples.MNIST_Distrib_Sgd$.main(MNIST_Distrib_Sgd.scala:41) at org.apache.sysml.examples.MNIST_Distrib_Sgd.main(MNIST_Distrib_Sgd.scala) Caused by: org.apache.sysml.api.mlcontext.MLContextException: Exception occurred while executing runtime program at org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(ScriptExecutor.java:397) at org.apache.sysml.api.mlcontext.ScriptExecutor.execute(ScriptExecutor.java:311) at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:316) ... 5 more Caused by: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 11 and 11 -- Error evaluating instruction: CP°extfunct°./scripts/nn/examples/mnist_lenet_distrib_sgd.dml°train°10°8°X·MATRIX·DOUBLE°Y·MATRIX·DOUBLE°X_val·MATRIX·DOUBLE°Y_val·MATRIX·DOUBLE°C·SCALAR·INT·false°Hin·SCALAR·INT·false°Win·SCALAR·INT·false°batch_size·SCALAR·INT·false°parallel_batches·SCALAR·INT·false°epochs·SCALAR·INT·false°W1°b1°W2°b2°W3°b3°W4°b4 at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:130) at org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:95) at org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:54) at org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(ScriptExecutor.java:395) ... 7 more Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 11 and 11 -- Error evaluating instruction: CP°extfunct°./scripts/nn/examples/mnist_lenet_distrib_sgd.dml°train°10°8°X·MATRIX·DOUBLE°Y·MATRIX·DOUBLE°X_val·MATRIX·DOUBLE°Y_val·MATRIX·DOUBLE°C·SCALAR·INT·false°Hin·SCALAR·INT·false°Win·SCALAR·INT·false°batch_size·SCALAR·INT·false°parallel_batches·SCALAR·INT·false°epochs·SCALAR·INT·false°W1°b1°W2°b2°W3°b3°W4°b4 at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:316) at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:217) at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:163) at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123) ... 10 more Caused by: org.apache.sysml.runtime.DMLRuntimeException: error executing function ./scripts/nn/examples/mnist_lenet_distrib_sgd.dml::train at org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:187) at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286) ... 13 more Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 36 and 271 -- Error evaluating function program block at org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:122) at org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:180) ... 14 more Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 115 and 270 -- Error evaluating for program block at org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:157) at org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:115) ... 15 more Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 117 and 265 -- Error evaluating for program block at org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:157) at org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:145) ... 16 more Caused by: org.apache.sysml.runtime.DMLRuntimeException: PARFOR: Failed to execute loop in parallel. at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:660) at org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:145) ... 17 more Caused by: org.apache.sysml.runtime.DMLRuntimeException: Not supported: Instructions of type other than CP instructions org.apache.sysml.runtime.instructions.spark.RandSPInstruction SPARK°rand°¶_Var3092¶°512°1000°1000°1°1°0.5°-1°scratch_space/_p10640_9.108.91.12//_t0/°uniform°1.0°_mVar3093·MATRIX·DOUBLE at org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.serializeInstructions(ProgramConverter.java:937) at org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.rSerializeProgramBlock(ProgramConverter.java:1297) at org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.rSerializeProgramBlocks(ProgramConverter.java:1125) at org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.rSerializeProgramBlock(ProgramConverter.java:1264) at org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.rSerializeFunctionProgramBlocks(ProgramConverter.java:1106) at org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.serializeProgram(ProgramConverter.java:805) at org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.serializeParForBody(ProgramConverter.java:763) at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeRemoteSparkParFor(ParForProgramBlock.java:1034) at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:647) ... 18 more {code} > Improve Parfor parallelism for deep learning > -------------------------------------------- > > Key: SYSTEMML-1774 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1774 > Project: SystemML > Issue Type: Improvement > Components: Algorithms, Compiler, ParFor > Affects Versions: SystemML 1.0 > Reporter: Fei Hu > Labels: deeplearning > > When running the [distributed MNIST LeNet example | > https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet_distrib_sgd.dml], > each mini-batch could ideally run in parallel without interaction. We try to > force {{parfor (j in 1:parallel_batches)}} at line 137 of > {{nn/examples/mnist_lenet_distrib_sgd.dml}} to be {{parfor (j in > 1:parallel_batches, mode=REMOTE_SPARK, opt=CONSTRAINED)}} use > {{REMOTE_SPARK}} mode, but got some errors about > {{org.apache.sysml.runtime.DMLRuntimeException: Not supported: Instructions > of type other than CP instructions}}. More log information can be found at > the following comments. One example of the errors is that at the convolution > layer, we need to randomly generate a matrix, but SystemML choose > {{RandSPInstruction}} instead of {{DataGenCPInstruction}}, which may be > because SystemML could not determine the row number of the matrix. For this > distributed MNIST LeNet example, using CPInstruction may achieve better > performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029)