This has been resolved in https://issues.apache.org/jira/browse/SYSTEMML-955.
Thanks, Glenn From: Glenn Weidner/Silicon Valley/IBM@IBMUS To: [email protected] Date: 09/02/2016 04:00 PM Subject: NPE running GLM-predict.dml in spark mode When running Linear Regression example described at https://apache.github.io/incubator-systemml/standalone-guide.html#linear-regression-example , everything works well with spark-submit under hybrid_spark mode. However, when running the very last step GLM-predict with -exec spark, a null pointer exception occurs. spark-submit --master yarn --deploy-mode client --class org.apache.sysml.api.DMLScript systemml.jar -f GLM-predict.dml -exec spark -nvargs X=linRegData.test.data.csv Y=linRegData.test.labels.csv B=betas.csv fmt=csv 16/09/02 15:35:10 ERROR DMLScript: Failed to execute DML script. org.apache.sysml.lops.LopsException: ERROR: line 240, column 30 -- Problem generating simple inst - SPARK°cdf°dist=normal°target=¶_Var237¶° _Var238·SCALAR·DOUBLE at org.apache.sysml.lops.compile.Dag.generateControlProgramJobs (Dag.java:1529) at org.apache.sysml.lops.compile.Dag.doGreedyGrouping (Dag.java:1194) at org.apache.sysml.lops.compile.Dag.getJobs(Dag.java:268) at org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock (DMLProgram.java:530) at org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock (DMLProgram.java:327) at org.apache.sysml.parser.DMLProgram.getRuntimeProgram (DMLProgram.java:196) at org.apache.sysml.api.DMLScript.execute(DMLScript.java:625) at org.apache.sysml.api.DMLScript.executeScript (DMLScript.java:337) at org.apache.sysml.api.DMLScript.main(DMLScript.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark $deploy $SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1 (SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit (SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main (SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.NullPointerException at org.apache.sysml.runtime.instructions.SPInstructionParser.parseSingleInstruction (SPInstructionParser.java:284) at org.apache.sysml.runtime.instructions.InstructionParser.parseSingleInstruction (InstructionParser.java:47) at org.apache.sysml.lops.compile.Dag.generateControlProgramJobs (Dag.java:1516) ... 17 more Exception in thread "main" org.apache.sysml.api.DMLException: org.apache.sysml.lops.LopsException: ERROR: line 240, column 30 -- Problem generating simple inst - SPARK°cdf°dist=normal°target=¶_Var237¶° _Var238·SCALAR·DOUBLE at org.apache.sysml.api.DMLScript.executeScript (DMLScript.java:351) at org.apache.sysml.api.DMLScript.main(DMLScript.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark $deploy $SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1 (SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit (SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main (SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.apache.sysml.lops.LopsException: ERROR: line 240, column 30 -- Problem generating simple inst - SPARK°cdf°dist=normal°target=¶_Var237¶° _Var238·SCALAR·DOUBLE at org.apache.sysml.lops.compile.Dag.generateControlProgramJobs (Dag.java:1529) at org.apache.sysml.lops.compile.Dag.doGreedyGrouping (Dag.java:1194) at org.apache.sysml.lops.compile.Dag.getJobs(Dag.java:268) at org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock (DMLProgram.java:530) at org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock (DMLProgram.java:327) at org.apache.sysml.parser.DMLProgram.getRuntimeProgram (DMLProgram.java:196) at org.apache.sysml.api.DMLScript.execute(DMLScript.java:625) at org.apache.sysml.api.DMLScript.executeScript (DMLScript.java:337) ... 10 more Caused by: java.lang.NullPointerException at org.apache.sysml.runtime.instructions.SPInstructionParser.parseSingleInstruction (SPInstructionParser.java:284) at org.apache.sysml.runtime.instructions.InstructionParser.parseSingleInstruction (InstructionParser.java:47) at org.apache.sysml.lops.compile.Dag.generateControlProgramJobs (Dag.java:1516) ... 17 more 16/09/02 15:35:10 INFO SparkContext: Invoking stop() from shutdown hook Although I first hit the null pointer exception running newer code against Spark 2.0, the stack trace above corresponds to 0.10.0 release against Spark 1.6.1. No problem occurs if run with -exec hybrid_spark. I only used -exec spark to force different scenario and just checking if anyone else has seen this before opening a JIRA. Thanks, Glenn
