This is technically a java issue but I thought other hadoop users would find it interesting.
When we upgraded from old jvm to a newer (32bit) jvm of 1.6.0_21, we started seeing users' tasks having issues at random places 1. Tasks running 2-3 times slower 2. Tasks failing with OutOfMemory 3. Reducer failing with OutOfMemory while pulling map outpus Our first response was to increase the heapsize but what actually was happening was that NewRatio has changed from 8 to 2 after we upgraded jvm to 1.6.0_21. http://forums.oracle.com/forums/thread.jspa?threadID=2167755 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6887571 So with -Xmx640m, older jvm (1.6.0_17 or before) old generation went up to PS Old Generation capacity = 596574208 (568.9375MB) used = 596574200 (568.9374923706055MB) free = 8 (7.62939453125E-6MB) 99.99999865901007% used but newer jvm(1.6.0_21) only went up to PS Old Generation capacity = 447414272 (426.6875MB) used = 447414264 (426.68749237060547MB) free = 8 (7.62939453125E-6MB) 99.99999821194797% used causing all sorts of issues. To bring back the old behavior, we needed to add -XX:NewRatio=8 to jvm param. If you have MAPREDUCE-1207 in your version, you can set <property> <name>mapreduce.admin.map.child.java.opts</name> <value>-XX:NewRatio=8</value> </property> <property> <name>mapreduce.admin.reduce.child.java.opts</name> <value>-XX:NewRatio=8</value> </property> Otherwise, you can set it on 'mapred.child.java.opts' and ask users to also set it whenever they overwrite that conf. Maybe this is a known issue to everyone but thought I would share it since we spent so much time on this simple(?) issue :) You can check the NewRatio value by running jmap -histo on the jvm. Koji