On Apr 14, 2010, at 20:55 , Amareshwari Sri Ramadasu wrote: > Hi Keith, > > My answers inline. > > On 4/15/10 12:57 AM, "Keith Wiley" <[email protected]> wrote: > > How do I use a nondefault Java InputFormat/RecordReader with a Pipes program. > I realize I can set: > > <property> > <name>hadoop.pipes.java.recordreader</name> > <value>true</value> > </property> > > or alterntively "-D hadoop.pipes.java.recordreader=true" > > ...to get the default reader (and that works and my pipes program runs), but > how do I use my own (Java) InputFormat/RecordReader? I tried jarring up my > InputFormat and RecordReader, then passing the jar via -libjars, and > specifying it with hadoop.pipes.java.recordreader thus: > > <property> > <name>mapred.input.format.class</name> > <value>FitsHeaderInputFormat</value> > </property> > > <property> > <name>hadoop.pipes.java.recordreader</name> > <value>FitsHDURecordReader</value> > </property> > > >>>>> Here, the value for hadoop.pipes.java.recordreader should be true or > >>>>> false, not the recordreader itself.
I agree that the recordreader parameter is usually true or false, I was just trying stuff out...so which should it be if I am using my own recordreader, but which is in java? Does the hadoop.pipes.java.recordreader parameter state whether a java recordreader of any kind is being used with C++ pipes, or only whether the built-in default recordreader is being used? More to the point, what is the correct setting for this parameter in my situation, the one described in my original post? > $ pipes -libjars [PATH]/pipes.jar ...rest of command... > > >>>>> Can you run the following command without specifying any other > >>>>> configuration ? > $pipes -libjars [PATH]/pipes.jar inputformat FitsHeaderInputFormat .. rest of > the command. I assume you meant "-inputformat", with the hyphen. I still get: Exception in thread "main" java.lang.ClassNotFoundException: FitsHeaderInputFormat > where pipes.jar contains: > > $ jar tvf [PATH]/pipes.jar > 0 Wed Apr 14 11:49:40 PDT 2010 META-INF/ > 60 Wed Apr 14 11:49:40 PDT 2010 META-INF/MANIFEST.MF > 5167 Thu Apr 08 14:26:30 PDT 2010 FitsHDURecordReader.java > 1816 Thu Apr 08 14:26:30 PDT 2010 FitsHeaderInputFormat.java > > ...but I get this error: > > Exception in thread "main" java.lang.RuntimeException: > java.lang.RuntimeException: java.lang.ClassNotFoundException: > FitsHeaderInputFormat > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:900) > at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:399) > at > org.apache.hadoop.mapred.pipes.Submitter.setupPipesJob(Submitter.java:289) > at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:247) > at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479) > at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494) > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: > FitsHeaderInputFormat > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:892) > ... 5 more > Caused by: java.lang.ClassNotFoundException: FitsHeaderInputFormat > at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > at java.lang.ClassLoader.loadClass(ClassLoader.java:315) > at java.lang.ClassLoader.loadClass(ClassLoader.java:250) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:398) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866) > ... 6 more > > Lastly, I also tried specifying the class by the package at the top of the > java file. Thus, where FitsHeaderInputFormat.java contains: > > package org.uw.astro.util; > > I tried: > > <property> > <name>mapred.input.format.class</name> > <value>org.uw.astro.utilFitsHeaderInputFormat</value> > </property> > > >>>>> Here, the value should be org.uw.astro.util.FitsHeaderInputFormat. > >>>>> Looks like you missed '.' after util. Nah, that's just a typo in the email, sorry. The .xml file didn't have that error in it, so that's not the cause of the problem. Why do I keep getting a ClassNotFoundException on the input format class. There has got to be a way to get this to work. Any other suggestions would really help. Thank you. ________________________________________________________________________________ Keith Wiley [email protected] www.keithwiley.com "What I primarily learned in grad school is how much I *don't* know. Consequently, I left grad school with a higher ignorance to knowledge ratio than when I entered." -- Keith Wiley ________________________________________________________________________________
