On Dec 17, 2008, at 10:44 AM, Philip wrote:
I've been trying to trouble shoot an OOME we've been having.
When we run the job over a dataset that about 700GB (~9000 files) or
larger
we will get an OOME on the map jobs. However if we run the job over
smaller
set of the data then everything works out fine. So my question is:
What
changes in Hadoop as the size of the input set increases?
We are on hadoop 0.18.0.
I suspect the reason is that larger data-sets result in more maps and
we seem to have a memory leak at the TaskTracker which depends on the
number of maps being run on a given TaskTracker.
I've opened https://issues.apache.org/jira/browse/HADOOP-4906 to track
this.
As a workaround you could try increasing the heapsize for the
TaskTracker via HADOOP_TASKTRACKER_OPTS in conf/hadoop-env.sh.
Arun
