I implemented my own InputFormat/RecordReader, and I try to run it with
Hadoop Pipes. I understand I could pass in properties to Pipes program by
either:
<property>
<name>hadoop.pipes.java.recordreader</name>
<value>false</value>
</property>
or alterntively "-D hadoop.pipes.java.recordreader=false".
However, when I ran with above configuration and with my record reader, I
got error
Hadoop Pipes Exception: RecordReader defined when not needed. at
impl/HadoopPipes.cc:798 in virtual void
HadoopPipes::TaskContextImpl::runMap(std::string, int, bool)
It pipes did not seem to pick up my setting of
hadoop.pipes.java.recordreader as false.
I've tried using conf.xml or putting -D or the combine of both. None
worked. I've googled the whole day but could not find the reason. Did I
miss something here?
I am using hadoop-1.0.4.
Here is my conf.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>hadoop.pipes.executable</name>
<value>bin/cpc</value>
</property>
<property>
<name>hadoop.pipes.java.recordreader</name>
<value>false</value>
</property>
<property>
<name>hadoop.pipes.java.recordwriter</name>
<value>true</value>
</property>
</configuration>
Here is the command
$HADOOP pipes \
-conf $CONF \
-files 0 \
-libjars $HADOOP_HOME/build/hadoop-test-1.0.4-SNAPSHOT.jar \
-input $IN \
-output $OUT \
-program bin/$NAME \
-reduces 0 -reduce org.apache.hadoop.mapred.lib.IdentityReducer \
-inputformat org.apache.hadoop.mapred.pipes.WordCountInputFormat
where $CONF is full path to conf.xml
I could provide more info if that hellps to determine the reason.