Mapper runs out of memory
-------------------------
Key: HADOOP-4976
URL: https://issues.apache.org/jira/browse/HADOOP-4976
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.19.0
Environment: Amazon EC2 Extra Large instance (4 cores, 15 GB RAM), Sun
Java 6 (1.6.0_10); 1 Master, 4 Slaves (all the same); each Java process takes
the argument "-Xmx700m" (2 Java processes per Instance)
Reporter: Richard J. Zak
Fix For: 0.19.1
The hadoop job has the task of processing 4 directories in HDFS, each with 15
files. This is sample data, a test run, before I go to the needed 5
directories of about 800 documents each. The mapper takes in nearly 200 pages
(not files) and throws an OutOfMemory exception. The largest file is 17 MB.
If this problem is something on my end and not truly a bug, I apologize.
However, after Googling a bit, I did see many threads of people running out of
memory with small data sets.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
