[
https://issues.apache.org/jira/browse/SYSTEMML-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090325#comment-16090325
]
Fei Hu edited comment on SYSTEMML-1774 at 7/17/17 7:15 PM:
-----------------------------------------------------------
The errors when running the [distributed MNIST LeNet example |
https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet_distrib_sgd.dml]
at the local machine are as following. One example of the errors is that at
the convolutional layer, we need to randomly generate some matrixes, but
SystemML choose {{RandSPInstruction}} instead of {{DataGenCPInstruction}},
which may be because SystemML could not determine the row number of the matrix.
For this distributed MNIST LeNet example, using CPInstruction may achieve
better performance.
{code:java}
Exception in thread "main" org.apache.sysml.api.mlcontext.MLContextException:
Exception when executing script
at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:320)
at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:293)
at org.apache.sysml.api.mlcontext.Script.execute(Script.java:717)
at
org.apache.sysml.scripts.nn.examples.Mnist_lenet_distrib_sgd.train(Mnist_lenet_distrib_sgd.java)
at
org.apache.sysml.examples.MNIST_Distrib_Sgd$.main(MNIST_Distrib_Sgd.scala:41)
at
org.apache.sysml.examples.MNIST_Distrib_Sgd.main(MNIST_Distrib_Sgd.scala)
Caused by: org.apache.sysml.api.mlcontext.MLContextException: Exception
occurred while executing runtime program
at
org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(ScriptExecutor.java:397)
at
org.apache.sysml.api.mlcontext.ScriptExecutor.execute(ScriptExecutor.java:311)
at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:316)
... 5 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program
block generated from statement block between lines 11 and 11 -- Error
evaluating instruction:
CP°extfunct°./scripts/nn/examples/mnist_lenet_distrib_sgd.dml°train°10°8°X·MATRIX·DOUBLE°Y·MATRIX·DOUBLE°X_val·MATRIX·DOUBLE°Y_val·MATRIX·DOUBLE°C·SCALAR·INT·false°Hin·SCALAR·INT·false°Win·SCALAR·INT·false°batch_size·SCALAR·INT·false°parallel_batches·SCALAR·INT·false°epochs·SCALAR·INT·false°W1°b1°W2°b2°W3°b3°W4°b4
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:130)
at
org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:95)
at
org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:54)
at
org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(ScriptExecutor.java:395)
... 7 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error
in program block generated from statement block between lines 11 and 11 --
Error evaluating instruction:
CP°extfunct°./scripts/nn/examples/mnist_lenet_distrib_sgd.dml°train°10°8°X·MATRIX·DOUBLE°Y·MATRIX·DOUBLE°X_val·MATRIX·DOUBLE°Y_val·MATRIX·DOUBLE°C·SCALAR·INT·false°Hin·SCALAR·INT·false°Win·SCALAR·INT·false°batch_size·SCALAR·INT·false°parallel_batches·SCALAR·INT·false°epochs·SCALAR·INT·false°W1°b1°W2°b2°W3°b3°W4°b4
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:316)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:217)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:163)
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
... 10 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: error executing
function ./scripts/nn/examples/mnist_lenet_distrib_sgd.dml::train
at
org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:187)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286)
... 13 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error
in function program block generated from function statement block between lines
36 and 271 -- Error evaluating function program block
at
org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:122)
at
org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:180)
... 14 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error
in for program block generated from for statement block between lines 115 and
270 -- Error evaluating for program block
at
org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:157)
at
org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:115)
... 15 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error
in for program block generated from for statement block between lines 117 and
265 -- Error evaluating for program block
at
org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:157)
at
org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:145)
... 16 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: PARFOR: Failed to
execute loop in parallel.
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:660)
at
org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:145)
... 17 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Not supported:
Instructions of type other than CP instructions
org.apache.sysml.runtime.instructions.spark.RandSPInstruction
SPARK°rand°¶_Var3092¶°512°1000°1000°1°1°0.5°-1°scratch_space/_p10640_9.108.91.12//_t0/°uniform°1.0°_mVar3093·MATRIX·DOUBLE
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.serializeInstructions(ProgramConverter.java:937)
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.rSerializeProgramBlock(ProgramConverter.java:1297)
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.rSerializeProgramBlocks(ProgramConverter.java:1125)
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.rSerializeProgramBlock(ProgramConverter.java:1264)
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.rSerializeFunctionProgramBlocks(ProgramConverter.java:1106)
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.serializeProgram(ProgramConverter.java:805)
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.serializeParForBody(ProgramConverter.java:763)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeRemoteSparkParFor(ParForProgramBlock.java:1034)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:647)
... 18 more
{code}
was (Author: tenma):
The errors when running the [distributed MNIST LeNet example |
https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet_distrib_sgd.dml]
at the local machine:
{code:java}
Exception in thread "main" org.apache.sysml.api.mlcontext.MLContextException:
Exception when executing script
at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:320)
at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:293)
at org.apache.sysml.api.mlcontext.Script.execute(Script.java:717)
at
org.apache.sysml.scripts.nn.examples.Mnist_lenet_distrib_sgd.train(Mnist_lenet_distrib_sgd.java)
at
org.apache.sysml.examples.MNIST_Distrib_Sgd$.main(MNIST_Distrib_Sgd.scala:41)
at
org.apache.sysml.examples.MNIST_Distrib_Sgd.main(MNIST_Distrib_Sgd.scala)
Caused by: org.apache.sysml.api.mlcontext.MLContextException: Exception
occurred while executing runtime program
at
org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(ScriptExecutor.java:397)
at
org.apache.sysml.api.mlcontext.ScriptExecutor.execute(ScriptExecutor.java:311)
at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:316)
... 5 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program
block generated from statement block between lines 11 and 11 -- Error
evaluating instruction:
CP°extfunct°./scripts/nn/examples/mnist_lenet_distrib_sgd.dml°train°10°8°X·MATRIX·DOUBLE°Y·MATRIX·DOUBLE°X_val·MATRIX·DOUBLE°Y_val·MATRIX·DOUBLE°C·SCALAR·INT·false°Hin·SCALAR·INT·false°Win·SCALAR·INT·false°batch_size·SCALAR·INT·false°parallel_batches·SCALAR·INT·false°epochs·SCALAR·INT·false°W1°b1°W2°b2°W3°b3°W4°b4
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:130)
at
org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:95)
at
org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:54)
at
org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(ScriptExecutor.java:395)
... 7 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error
in program block generated from statement block between lines 11 and 11 --
Error evaluating instruction:
CP°extfunct°./scripts/nn/examples/mnist_lenet_distrib_sgd.dml°train°10°8°X·MATRIX·DOUBLE°Y·MATRIX·DOUBLE°X_val·MATRIX·DOUBLE°Y_val·MATRIX·DOUBLE°C·SCALAR·INT·false°Hin·SCALAR·INT·false°Win·SCALAR·INT·false°batch_size·SCALAR·INT·false°parallel_batches·SCALAR·INT·false°epochs·SCALAR·INT·false°W1°b1°W2°b2°W3°b3°W4°b4
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:316)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:217)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:163)
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
... 10 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: error executing
function ./scripts/nn/examples/mnist_lenet_distrib_sgd.dml::train
at
org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:187)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286)
... 13 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error
in function program block generated from function statement block between lines
36 and 271 -- Error evaluating function program block
at
org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:122)
at
org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:180)
... 14 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error
in for program block generated from for statement block between lines 115 and
270 -- Error evaluating for program block
at
org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:157)
at
org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:115)
... 15 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error
in for program block generated from for statement block between lines 117 and
265 -- Error evaluating for program block
at
org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:157)
at
org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:145)
... 16 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: PARFOR: Failed to
execute loop in parallel.
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:660)
at
org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:145)
... 17 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Not supported:
Instructions of type other than CP instructions
org.apache.sysml.runtime.instructions.spark.RandSPInstruction
SPARK°rand°¶_Var3092¶°512°1000°1000°1°1°0.5°-1°scratch_space/_p10640_9.108.91.12//_t0/°uniform°1.0°_mVar3093·MATRIX·DOUBLE
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.serializeInstructions(ProgramConverter.java:937)
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.rSerializeProgramBlock(ProgramConverter.java:1297)
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.rSerializeProgramBlocks(ProgramConverter.java:1125)
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.rSerializeProgramBlock(ProgramConverter.java:1264)
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.rSerializeFunctionProgramBlocks(ProgramConverter.java:1106)
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.serializeProgram(ProgramConverter.java:805)
at
org.apache.sysml.runtime.controlprogram.parfor.ProgramConverter.serializeParForBody(ProgramConverter.java:763)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeRemoteSparkParFor(ParForProgramBlock.java:1034)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:647)
... 18 more
{code}
> Improve Parfor parallelism for deep learning
> --------------------------------------------
>
> Key: SYSTEMML-1774
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1774
> Project: SystemML
> Issue Type: Improvement
> Components: Algorithms, Compiler, ParFor
> Affects Versions: SystemML 1.0
> Reporter: Fei Hu
> Labels: deeplearning
>
> When running the [distributed MNIST LeNet example |
> https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet_distrib_sgd.dml],
> each mini-batch could ideally run in parallel without interaction. We try to
> force {{parfor (j in 1:parallel_batches)}} at line 137 of
> {{nn/examples/mnist_lenet_distrib_sgd.dml}} to be {{parfor (j in
> 1:parallel_batches, mode=REMOTE_SPARK, opt=CONSTRAINED)}} use
> {{REMOTE_SPARK}} mode, but got some errors about
> {{org.apache.sysml.runtime.DMLRuntimeException: Not supported: Instructions
> of type other than CP instructions}} on the local machine and the error
> {{java.lang.NullPointerException}} on the Spark cluster. More log information
> can be found at the following comments.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
