On Nov 13, 2008, at 1:59 PM, Brian Bockelman wrote:

Hey all,

When we run a large task (lots of intermediate output), we have a real problem with it "beating up" nodes during the shuffle phase. About half of our nodes are completely overwhelmed by the number of reducers requesting data to be copied (somewhere between 50 and 75 files appear to be open); this causes wait_io on the node to go up to about 50-75% of the CPU. The number of active threads is about 45. The amount of intermediate data is about 10GB per node.

Anyone else run into this problem? Any hints? My first thought is to ask the user to use a combiner so there are less files to work with in the first place.


Combiners are useful, they help in cutting down the amount of data which needs to be shuffled.

Have you tried compressing map-outputs?
http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Data+Compression

A heavy-handed option is to cut down the no. of per-reducer threads which do the shuffle: mapred.reduce.parallel.copies. We recommend ~20, try with a lesser number than that...

Arun

Reply via email to