[
https://issues.apache.org/jira/browse/SYSTEMML-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706722#comment-15706722
]
Felix Schüler commented on SYSTEMML-1131:
-----------------------------------------
The stack trace:
```
16/11/29 14:16:55 ERROR DMLScript: Failed to execute DML script.
org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: PARFOR: Failed to execute loop in
parallel.
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:130)
at org.apache.sysml.api.DMLScript.execute(DMLScript.java:675)
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:358)
at org.apache.sysml.api.DMLScript.main(DMLScript.java:200)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.sysml.runtime.DMLRuntimeException: PARFOR: Failed to
execute loop in parallel.
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:630)
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
... 12 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile
program block hierarchy to CP.
at
org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy2Forced(Recompiler.java:554)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.releaseForcedRecompile(ParForProgramBlock.java:1561)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeRemoteSparkParFor(ParForProgramBlock.java:1028)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:617)
... 13 more
Caused by: java.lang.NullPointerException
at
org.apache.sysml.hops.recompile.Recompiler.recompileHopsDag2Forced(Recompiler.java:407)
at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock2Forced(Recompiler.java:1228)
at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock2Forced(Recompiler.java:1237)
at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock2Forced(Recompiler.java:1268)
at
org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy2Forced(Recompiler.java:549)
... 16 more
Exception in thread "main" org.apache.sysml.api.DMLException:
org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: PARFOR: Failed to execute loop in
parallel.
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:372)
at org.apache.sysml.api.DMLScript.main(DMLScript.java:200)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: PARFOR: Failed to execute loop in
parallel.
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:130)
at org.apache.sysml.api.DMLScript.execute(DMLScript.java:675)
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:358)
... 10 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: PARFOR: Failed to
execute loop in parallel.
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:630)
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
... 12 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile
program block hierarchy to CP.
at
org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy2Forced(Recompiler.java:554)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.releaseForcedRecompile(ParForProgramBlock.java:1561)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeRemoteSparkParFor(ParForProgramBlock.java:1028)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:617)
... 13 more
Caused by: java.lang.NullPointerException
at
org.apache.sysml.hops.recompile.Recompiler.recompileHopsDag2Forced(Recompiler.java:407)
at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock2Forced(Recompiler.java:1228)
at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock2Forced(Recompiler.java:1237)
at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock2Forced(Recompiler.java:1268)
at
org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy2Forced(Recompiler.java:549)
... 16 more
```
> NPE in executeRemoteSparkParFor
> -------------------------------
>
> Key: SYSTEMML-1131
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1131
> Project: SystemML
> Issue Type: Bug
> Reporter: Felix Schüler
>
> The method ParForProgramBlock.releaseForcedRecompile(long tid) calls
> recompileProgramBlockHierarchy2Forced with execution type (et) null. This
> leads to a NullPointerException.
> I haven't fully figured out under which circumstances this occurs but it
> happens when calling an external function inside a forced parfor_spark.
> The ParForProgramBlock.executeRemoteSparkParFor method sets the flagForced to
> true which then in turn calls the above method with et==null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
