How do I use a nondefault Java InputFormat/RecordReader with a Pipes program.
I realize I can set:
<property>
<name>hadoop.pipes.java.recordreader</name>
<value>true</value>
</property>
or alterntively "-D hadoop.pipes.java.recordreader=true"
...to get the default reader (and that works and my pipes program runs), but
how do I use my own (Java) InputFormat/RecordReader? I tried jarring up my
InputFormat and RecordReader, then passing the jar via -libjars, and specifying
it with hadoop.pipes.java.recordreader thus:
<property>
<name>mapred.input.format.class</name>
<value>FitsHeaderInputFormat</value>
</property>
<property>
<name>hadoop.pipes.java.recordreader</name>
<value>FitsHDURecordReader</value>
</property>
$ pipes -libjars [PATH]/pipes.jar ...rest of command...
where pipes.jar contains:
$ jar tvf [PATH]/pipes.jar
0 Wed Apr 14 11:49:40 PDT 2010 META-INF/
60 Wed Apr 14 11:49:40 PDT 2010 META-INF/MANIFEST.MF
5167 Thu Apr 08 14:26:30 PDT 2010 FitsHDURecordReader.java
1816 Thu Apr 08 14:26:30 PDT 2010 FitsHeaderInputFormat.java
...but I get this error:
Exception in thread "main" java.lang.RuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException:
FitsHeaderInputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:900)
at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:399)
at
org.apache.hadoop.mapred.pipes.Submitter.setupPipesJob(Submitter.java:289)
at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:247)
at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479)
at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
FitsHeaderInputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:892)
... 5 more
Caused by: java.lang.ClassNotFoundException: FitsHeaderInputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:315)
at java.lang.ClassLoader.loadClass(ClassLoader.java:250)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:398)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
... 6 more
Lastly, I also tried specifying the class by the package at the top of the java
file. Thus, where FitsHeaderInputFormat.java contains:
package org.uw.astro.util;
I tried:
<property>
<name>mapred.input.format.class</name>
<value>org.uw.astro.utilFitsHeaderInputFormat</value>
</property>
...but I get the exact same error. I am rapidly running out of ideas here.
Thanks for any help.
________________________________________________________________________________
Keith Wiley [email protected] www.keithwiley.com
"What I primarily learned in grad school is how much I *don't* know.
Consequently, I left grad school with a higher ignorance to knowledge ratio than
when I entered."
-- Keith Wiley
________________________________________________________________________________
