memory model is not accurate enough for map side sorts
------------------------------------------------------
Key: HADOOP-875
URL: https://issues.apache.org/jira/browse/HADOOP-875
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.10.0
Reporter: Owen O'Malley
Assigned To: Devaraj Das
I configured a sort (IdentityMapper) with large compressed values with the map
output buffer (io.sort.mb) set to 300mb and the child jvm heap size set to
900mb and some of the maps were running out of memory deterministically. I
suspect that some part of the data path is not handling large values well and
is consuming large amounts of ram.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
