Depends on the distribution of the keys and how the partitioner is assigning keys to reducers. (Remember that pairs with the same key have to go to the same reducer). Chris On Jan 8, 2014 2:33 AM, "ch huang" <[email protected]> wrote:
> hi,maillist: > i look the containers log from " hadoop fs -cat > /var/log/hadoop-yarn/apps/root/logs/application_1388730279827_2770/CHBM221_50853" > > and log say it get 25 map output , and assiging 7 to fetcher 5, assiging 7 > to fetcher 4 and assiging 11 to fetcher 3,my question is why not > " assiging 8 to fetcher 5, assiging 8 to fetcher 4 and assiging 9 to > fetcher 3 " ? > > 2014-01-08 11:28:00,346 INFO [EventFetcher for fetching Map Completion > Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: > attempt_1388730279827_2770_r_000000_0: Got 25 new map-outputs > 2014-01-08 11:28:00,348 INFO [fetcher#5] > org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging > CHBM223:8080 with 7 to fetcher#5 > 2014-01-08 11:28:00,349 INFO [fetcher#5] > org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7 > to CHBM223:8080 to fetcher#5 > 2014-01-08 11:28:00,349 INFO [fetcher#4] > org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging > CHBM222:8080 with 7 to fetcher#4 > 2014-01-08 11:28:00,349 INFO [fetcher#4] > org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7 > to CHBM222:8080 to fetcher#4 > 2014-01-08 11:28:00,352 INFO [fetcher#3] > org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging > CHBM221:8080 with 11 to fetcher#3 > 2014-01-08 11:28:00,352 INFO [fetcher#3] > org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 11 of 11 > to CHBM221:8080 to fetcher#3 >
