[ https://issues.apache.org/jira/browse/SYSTEMML-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090590#comment-16090590 ]
Mike Dusenberry edited comment on SYSTEMML-1774 at 7/17/17 10:14 PM: --------------------------------------------------------------------- [~Tenma] Can you include a script with the updated parfor loop that could be used to replicate this issue, as well as the memory settings? Additionally, can you include the runtime plan ({{ml.setExplain(True)}} in MLContext) for each setup (local machine, Spark cluster)? [~mboehm7] Can you please assist in fixing this issue? I suspect that we have two issues here: (1) the sizes aren't being propagated to the rand op thus causing as conservative Spark op to start with, and (2) a lack of a guard against compiling Spark ops within a REMOTE_SPARK parfor op. EDIT: Part of the issue may actually be due to the SPARK vs HYBRID_SPARK modes. was (Author: mwdus...@us.ibm.com): [~Tenma] Can you include a script with the updated parfor loop that could be used to replicate this issue, as well as the memory settings? Additionally, can you include the runtime plan ({{ml.setExplain(True)}} in MLContext) for each setup (local machine, Spark cluster)? [~mboehm7] Can you please assist in fixing this issue? I suspect that we have two issues here: (1) the sizes aren't being propagated to the rand op thus causing as conservative Spark op to start with, and (2) a lack of a guard against compiling Spark ops within a REMOTE_SPARK parfor op. > Improve Parfor parallelism for deep learning > -------------------------------------------- > > Key: SYSTEMML-1774 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1774 > Project: SystemML > Issue Type: Improvement > Components: Algorithms, Compiler, ParFor > Affects Versions: SystemML 1.0 > Reporter: Fei Hu > Labels: deeplearning > Attachments: mnist_lenet_distrib_sgd.dml > > > When running the [distributed MNIST LeNet example | > https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet_distrib_sgd.dml], > each mini-batch could ideally run in parallel without interaction. We try to > force {{parfor (j in 1:parallel_batches)}} at line 137 of > {{nn/examples/mnist_lenet_distrib_sgd.dml}} to be {{parfor (j in > 1:parallel_batches, mode=REMOTE_SPARK, opt=CONSTRAINED)}} use > {{REMOTE_SPARK}} mode, but got some errors about > {{org.apache.sysml.runtime.DMLRuntimeException: Not supported: Instructions > of type other than CP instructions}} using the mode {{SPARK}}, and the error > {{java.lang.NullPointerException}} using the mode {{HYBRID_SPARK}}. More log > information can be found at the following comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029)