Hi Keith, My answers inline.
On 4/15/10 12:57 AM, "Keith Wiley" <[email protected]> wrote: How do I use a nondefault Java InputFormat/RecordReader with a Pipes program. I realize I can set: <property> <name>hadoop.pipes.java.recordreader</name> <value>true</value> </property> or alterntively "-D hadoop.pipes.java.recordreader=true" ...to get the default reader (and that works and my pipes program runs), but how do I use my own (Java) InputFormat/RecordReader? I tried jarring up my InputFormat and RecordReader, then passing the jar via -libjars, and specifying it with hadoop.pipes.java.recordreader thus: <property> <name>mapred.input.format.class</name> <value>FitsHeaderInputFormat</value> </property> <property> <name>hadoop.pipes.java.recordreader</name> <value>FitsHDURecordReader</value> </property> Here, the value for hadoop.pipes.java.recordreader should be true or false, not the recordreader itself. $ pipes -libjars [PATH]/pipes.jar ...rest of command... Can you run the following command without specifying any other configuration ? $pipes -libjars [PATH]/pipes.jar inputformat FitsHeaderInputFormat .. rest of the command. where pipes.jar contains: $ jar tvf [PATH]/pipes.jar 0 Wed Apr 14 11:49:40 PDT 2010 META-INF/ 60 Wed Apr 14 11:49:40 PDT 2010 META-INF/MANIFEST.MF 5167 Thu Apr 08 14:26:30 PDT 2010 FitsHDURecordReader.java 1816 Thu Apr 08 14:26:30 PDT 2010 FitsHeaderInputFormat.java ...but I get this error: Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: FitsHeaderInputFormat at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:900) at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:399) at org.apache.hadoop.mapred.pipes.Submitter.setupPipesJob(Submitter.java:289) at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:247) at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479) at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: FitsHeaderInputFormat at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:892) ... 5 more Caused by: java.lang.ClassNotFoundException: FitsHeaderInputFormat at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:315) at java.lang.ClassLoader.loadClass(ClassLoader.java:250) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:398) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866) ... 6 more Lastly, I also tried specifying the class by the package at the top of the java file. Thus, where FitsHeaderInputFormat.java contains: package org.uw.astro.util; I tried: <property> <name>mapred.input.format.class</name> <value>org.uw.astro.utilFitsHeaderInputFormat</value> </property> Here, the value should be org.uw.astro.util.FitsHeaderInputFormat. Looks like you missed '.' after util. ...but I get the exact same error. I am rapidly running out of ideas here. Thanks for any help. ________________________________________________________________________________ Keith Wiley [email protected] www.keithwiley.com "What I primarily learned in grad school is how much I *don't* know. Consequently, I left grad school with a higher ignorance to knowledge ratio than when I entered." -- Keith Wiley ________________________________________________________________________________
