On 2/18/10 10:39 AM, Rob Stewart wrote:
Hi there,
I am using Hadoop 0.20.1 and I am trying to submit jobs to the cluster
as jars.
Here's what I'm trying to do:
> hadoop jar $hadoopPath/hadoop-*-examples.jar join
-Dmapred.reduce.tasks=10 -inFormat
org.apache.hadoop.mapred.KeyValueTextInputFormat -outKey
org.apache.hadoop.io.Text file1.dat file2.dat output.dat
However, my parameter setting of attempting to state 10 reducers for the
job is not being honoured by Hadoop, instead choosing some other number.
Where am I going wrong here? I do not want to have to change this value
in hadoop/conf.*.xml files, as I am attempting to show the expressive
power of Hadoop. Note: The power to specify the number or reducers in
possible in both Pig and Hive.
Thanks,
Rob Stewart
Rob:
It's possible that something inside the jar is calling
JobConf.setNumReducers(x) after it parses the command line args. That
would cause this type of behavior. I haven't looked at the source for
the join example to confirm this, though.
Regards.
--
Eric Sammer
[email protected]
http://esammer.blogspot.com
