On Nov 13, 2008, at 4:23 PM, Arun C Murthy wrote:
On Nov 13, 2008, at 1:59 PM, Brian Bockelman wrote:
Hey all,
When we run a large task (lots of intermediate output), we have a
real problem with it "beating up" nodes during the shuffle phase.
About half of our nodes are completely overwhelmed by the number of
reducers requesting data to be copied (somewhere between 50 and 75
files appear to be open); this causes wait_io on the node to go up
to about 50-75% of the CPU. The number of active threads is about
45. The amount of intermediate data is about 10GB per node.
Anyone else run into this problem? Any hints? My first thought is
to ask the user to use a combiner so there are less files to work
with in the first place.
Combiners are useful, they help in cutting down the amount of data
which needs to be shuffled.
I poked around the user's code, and he indeed had 36 billion outputs,
and no real need to not use a Combiner. I'll sit and have a good talk
with him tomorrow...
Have you tried compressing map-outputs?
http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Data+Compression
A heavy-handed option is to cut down the no. of per-reducer threads
which do the shuffle: mapred.reduce.parallel.copies. We recommend
~20, try with a lesser number than that...
Ah, I might think about this one. What's the thinking behind the ~20
number?
Brian