Scott Oaks created MAPREDUCE-7336:
-------------------------------------
Summary: MapTask leaks sort buffer into merge increasing GC
pressure in merge
Key: MAPREDUCE-7336
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7336
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: performance
Affects Versions: 3.2.2
Reporter: Scott Oaks
When running with a large value for io.sort.mb, we see lots of GC pressure (and
often GC failures) during the merge phase as the array of spill records are
allocated. This is because the kvbuffer allocated during the sort phase is
still allocated and consuming memory.
This doesn't seem to be the intention; line 1539 of MapTask.java reads:
{code:java}
// release sort buffer before the merge
kvbuffer = null;{code}
But this isn't the only place that has a reference to kvbuffer; line 1002 wraps
it into kvmeta:
{code:java}
kvmeta = ByteBUffer.wrap(kvbuffer){code}
The code needs to deference kvmeta so that kvbuffer can be garbage collected as
desired.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
