Brendan, Are these jobs (whose split behavior has changed) via Hive/etc. or plain Java MR?
In case its the former, do you have users using newer versions of them? On 04-Nov-2011, at 8:03 PM, Brendan W. wrote: > Hi, > > In the jobs running on my cluster of 20 machines, I used to run jobs (via > "hadoop jar ...") that would spawn around 4000 map tasks. Now when I run > the same jobs, that number is 20; and I notice that in the job > configuration, the parameter mapred.map.tasks is set to 20, whereas it > never used to be present at all in the configuration file. > > Changing the input split size in the job doesn't affect this--I get the > size split I ask for, but the *number* of input splits is still capped at > 20--i.e., the job isn't reading all of my data. > > The mystery to me is where this parameter could be getting set. It is not > present in the mapred-site.xml file in <hadoop home>/conf on any machine in > the cluster, and it is not being set in the job (I'm running out of the > same jar I always did; no updates). > > Is there *anywhere* else this parameter could possibly be getting set? > I've stopped and restarted map-reduce on the cluster with no effect...it's > getting re-read in from somewhere, but I can't figure out where. > > Thanks a lot, > > Brendan
