well, we did indeed not run on MR v1 for a while now. However, I don't want to get that far and say we don't support it anymore. I'll fix this particular issue by tomorrow.
In the next couple of weeks we should run our full performance testsuite (for broad coverage) over an MR v1 cluster and systematically remove unnecessary incompatibility like this instance. Any volunteers? Regards, Matthias From: Ethan Xu <ethan.yifa...@gmail.com> To: dev@systemml.incubator.apache.org Date: 02/04/2016 05:51 PM Subject: Compatibility with MR1 Cloudera cdh4.2.1 Hello, I got an error when running the systemML/scripts/Univar-Stats.dml script on a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is at the bottom of the email. The same script ran fine on a smaller sample (several MB) of the same data set, when MR was not invoked. The main error was java.lang.NoSuchMethodError: org.apache.hadoop.mapred.JobConf.getDouble() Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't have the JobConf.getDouble() method. The hadoop-core jar of CDH4.2.1 can be found here: https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/ The calling line of SystemML is line 1194 of https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java I was wondering, if the finding is accurate, is there a potential fix, or does this mean the current version of SystemML is not compatible with CDH4.2.1? Thank you, Ethan hadoop jar $sysDir/target/SystemML.jar -f $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs X=$baseDirHDFS/original-coded.csv TYPES=$baseDirHDFS/original-coded-type.csv STATS=$baseDirHDFS/univariate-summary.csv 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file (./SystemML-config.xml) found 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in DMLConfig 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded text read for 'text' and 'csv' due to thread contention on JRE < 1.8 (java.version=1.7.0_71). SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics: Total execution time: 0.880 sec. Number of executed MR Jobs: 0. 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07 Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D at org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs (MRJobConfiguration.java:1195) at org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs (MRJobConfiguration.java:1129) at org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob (CSVReblockMR.java:307) at org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob (CSVReblockMR.java:289) at org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275) at org.apache.sysml.lops.runtime.RunMRJobs.submitJob (RunMRJobs.java:257) at org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob (RunMRJobs.java:143) at org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction (MRJobInstruction.java:1500) at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction (ProgramBlock.java:309) at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions (ProgramBlock.java:227) at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute (ProgramBlock.java:169) at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146) at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676) at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338) at org.apache.sysml.api.DMLScript.main(DMLScript.java:197) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
