I think I pinned down the problem, but still can't find a solution. It seems like 'hadoop.pipes.java.recordreader' is not the correct property name to use? Because I tried setting other properties (e.g. hadoop.pipes.executable) in conf.xml and they were effectively passed in to pipes when the program runs.
Where could I find the source code of pipes reading conf.xml file? Where could I find a list of all properties (for pipes) and their default values? Best I could find is this http://hadoop.apache.org/docs/r1.1.2/commands_manual.html but it does not list all properties. Any suggestion? Thanks, Alice On Tue, Apr 23, 2013 at 10:14 PM, Xun TANG <[email protected]> wrote: > I implemented my own InputFormat/RecordReader, and I try to run it with > Hadoop Pipes. I understand I could pass in properties to Pipes program by > either: > > <property> > <name>hadoop.pipes.java.recordreader</name> > <value>false</value> > </property> > > or alterntively "-D hadoop.pipes.java.recordreader=false". > > However, when I ran with above configuration and with my record reader, I > got error > Hadoop Pipes Exception: RecordReader defined when not needed. at > impl/HadoopPipes.cc:798 in virtual void > HadoopPipes::TaskContextImpl::runMap(std::string, int, bool) > > It pipes did not seem to pick up my setting of > hadoop.pipes.java.recordreader as false. > > I've tried using conf.xml or putting -D or the combine of both. None > worked. I've googled the whole day but could not find the reason. Did I > miss something here? > > I am using hadoop-1.0.4. > > Here is my conf.xml > > <?xml version="1.0"?> > <configuration> > <property> > <name>hadoop.pipes.executable</name> > <value>bin/cpc</value> > </property> > <property> > <name>hadoop.pipes.java.recordreader</name> > <value>false</value> > </property> > <property> > <name>hadoop.pipes.java.recordwriter</name> > <value>true</value> > </property> > </configuration> > > Here is the command > > $HADOOP pipes \ > -conf $CONF \ > -files 0 \ > -libjars $HADOOP_HOME/build/hadoop-test-1.0.4-SNAPSHOT.jar \ > -input $IN \ > -output $OUT \ > -program bin/$NAME \ > -reduces 0 -reduce org.apache.hadoop.mapred.lib.IdentityReducer \ > -inputformat org.apache.hadoop.mapred.pipes.WordCountInputFormat > > where $CONF is full path to conf.xml > > I could provide more info if that hellps to determine the reason. > >
