[
https://issues.apache.org/jira/browse/HADOOP-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HADOOP-2053:
----------------------------------
Attachment: HADOOP-2053_1_20071015.patch
Here is a patch which frees the reference to the large {{DataOutputBuffer}}
that {{BasicTypeSorterBase}} has in it's {{close}} method... this lets the GC
collect away the keyValBuffer.
In absence of this patch, there is a window where both the currently active
keyValBuffer and the one that should have been freed in the previous iteration
are both active i.e. doubling the required amount of memory, which leads to the
OutOfMemoryException.
All credit to this goes to Koji!
> OutOfMemoryError : Java heap space errors in hadoop 0.14
> --------------------------------------------------------
>
> Key: HADOOP-2053
> URL: https://issues.apache.org/jira/browse/HADOOP-2053
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0, 0.14.1, 0.14.2
> Reporter: lohit vijayarenu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.3
>
> Attachments: HADOOP-2053_1_20071015.patch
>
>
> In recent hadoop 0.14 we are seeing few jobs where map taskf fail with
> java.lang.OutOfMemoryError: Java heap space problem
> These were the same jobs which used to work fine with 0.13
> <stack>
> task_200710112103_0001_m_000015_1: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:243)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:340)
> </stack>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
