[
https://issues.apache.org/jira/browse/MAPREDUCE-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billie Rinaldi updated MAPREDUCE-7265:
--------------------------------------
Attachment: MAPREDUCE-7265.01.patch
> Buffer corruption with spill percent 1.0
> ----------------------------------------
>
> Key: MAPREDUCE-7265
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7265
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Billie Rinaldi
> Assignee: Billie Rinaldi
> Priority: Minor
> Attachments: MAPREDUCE-7265.01.patch
>
>
> I encountered a variety of issues on a cluster where the spill percent was
> set to 1.0. Under some conditions, MapTask will not detect that its in memory
> spill buffer is already full and will keep collecting k/v pairs, causing
> corruption of the buffer.
> I have been able to track at least some of the problems to a condition where
> adding a key/value pair to the buffer fills the buffer with fewer than 16
> bytes remaining (the kv metadata size). When this happens, the next metadata
> index (kvindex) passes over the data index (bufindex), which causes some of
> the index and length calculations to be incorrect in the collect and write
> methods. It can allow data to keep being written to the buffer even though it
> is already full, with data overwriting metadata in the buffer and vice versa.
> I have seen this manifest as the NegativeArraySizeException seen in
> MAPREDUCE-6907 as well as in ArrayIndexOutOfBoundsException and EOFException.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
