When running Linear Regression example described at
https://apache.github.io/incubator-systemml/standalone-guide.html#linear-regression-example
, everything works well with spark-submit under hybrid_spark mode.
However, when running the very last step GLM-predict with -exec spark, a
null pointer exception occurs.
spark-submit --master yarn --deploy-mode client --class
org.apache.sysml.api.DMLScript systemml.jar -f GLM-predict.dml -exec spark
-nvargs X=linRegData.test.data.csv Y=linRegData.test.labels.csv B=betas.csv
fmt=csv
16/09/02 15:35:10 ERROR DMLScript: Failed to execute DML script.
org.apache.sysml.lops.LopsException: ERROR: line 240, column 30 -- Problem
generating simple inst - SPARK°cdf°dist=normal°target=¶_Var237¶°
_Var238·SCALAR·DOUBLE
at org.apache.sysml.lops.compile.Dag.generateControlProgramJobs
(Dag.java:1529)
at org.apache.sysml.lops.compile.Dag.doGreedyGrouping(Dag.java:1194)
at org.apache.sysml.lops.compile.Dag.getJobs(Dag.java:268)
at org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock
(DMLProgram.java:530)
at org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock
(DMLProgram.java:327)
at org.apache.sysml.parser.DMLProgram.getRuntimeProgram
(DMLProgram.java:196)
at org.apache.sysml.api.DMLScript.execute(DMLScript.java:625)
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:337)
at org.apache.sysml.api.DMLScript.main(DMLScript.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1
(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NullPointerException
at
org.apache.sysml.runtime.instructions.SPInstructionParser.parseSingleInstruction
(SPInstructionParser.java:284)
at
org.apache.sysml.runtime.instructions.InstructionParser.parseSingleInstruction
(InstructionParser.java:47)
at org.apache.sysml.lops.compile.Dag.generateControlProgramJobs
(Dag.java:1516)
... 17 more
Exception in thread "main" org.apache.sysml.api.DMLException:
org.apache.sysml.lops.LopsException: ERROR: line 240, column 30 -- Problem
generating simple inst - SPARK°cdf°dist=normal°target=¶_Var237¶°
_Var238·SCALAR·DOUBLE
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:351)
at org.apache.sysml.api.DMLScript.main(DMLScript.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1
(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.sysml.lops.LopsException: ERROR: line 240, column 30
-- Problem generating simple inst - SPARK°cdf°dist=normal°target=¶_Var237¶°
_Var238·SCALAR·DOUBLE
at org.apache.sysml.lops.compile.Dag.generateControlProgramJobs
(Dag.java:1529)
at org.apache.sysml.lops.compile.Dag.doGreedyGrouping(Dag.java:1194)
at org.apache.sysml.lops.compile.Dag.getJobs(Dag.java:268)
at org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock
(DMLProgram.java:530)
at org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock
(DMLProgram.java:327)
at org.apache.sysml.parser.DMLProgram.getRuntimeProgram
(DMLProgram.java:196)
at org.apache.sysml.api.DMLScript.execute(DMLScript.java:625)
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:337)
... 10 more
Caused by: java.lang.NullPointerException
at
org.apache.sysml.runtime.instructions.SPInstructionParser.parseSingleInstruction
(SPInstructionParser.java:284)
at
org.apache.sysml.runtime.instructions.InstructionParser.parseSingleInstruction
(InstructionParser.java:47)
at org.apache.sysml.lops.compile.Dag.generateControlProgramJobs
(Dag.java:1516)
... 17 more
16/09/02 15:35:10 INFO SparkContext: Invoking stop() from shutdown hook
Although I first hit the null pointer exception running newer code against
Spark 2.0, the stack trace above corresponds to 0.10.0 release against
Spark 1.6.1. No problem occurs if run with -exec hybrid_spark. I only
used -exec spark to force different scenario and just checking if anyone
else has seen this before opening a JIRA.
Thanks,
Glenn
