Hello, I got an error when running the systemML/scripts/Univar-Stats.dml script on a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is at the bottom of the email. The same script ran fine on a smaller sample (several MB) of the same data set, when MR was not invoked.
The main error was java.lang.NoSuchMethodError: org.apache.hadoop.mapred.JobConf.getDouble() Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't have the JobConf.getDouble() method. The hadoop-core jar of CDH4.2.1 can be found here: https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/ The calling line of SystemML is line 1194 of https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java I was wondering, if the finding is accurate, is there a potential fix, or does this mean the current version of SystemML is not compatible with CDH4.2.1? Thank you, Ethan hadoop jar $sysDir/target/SystemML.jar -f $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs X=$baseDirHDFS/original-coded.csv TYPES=$baseDirHDFS/original-coded-type.csv STATS=$baseDirHDFS/univariate-summary.csv 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file (./SystemML-config.xml) found 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in DMLConfig 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded text read for 'text' and 'csv' due to thread contention on JRE < 1.8 (java.version=1.7.0_71). SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics: Total execution time: 0.880 sec. Number of executed MR Jobs: 0. 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07 Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D at org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195) at org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129) at org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307) at org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289) at org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275) at org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257) at org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143) at org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500) at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309) at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227) at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169) at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146) at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676) at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338) at org.apache.sysml.api.DMLScript.main(DMLScript.java:197) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
