Billie Rinaldi created MAPREDUCE-7265:
-----------------------------------------
Summary: Buffer corruption with spill percent 1.0
Key: MAPREDUCE-7265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7265
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
I encountered a variety of issues on a cluster where the spill percent was set
to 1.0. Under some conditions, MapTask will not detect that its in memory spill
buffer is already full and will keep collecting k/v pairs, causing corruption
of the buffer.
I have been able to track at least some of the problems to a condition where
adding a key/value pair to the buffer fills the buffer with fewer than 16 bytes
remaining (the kv metadata size). When this happens, the next metadata index
(kvindex) passes over the data index (bufindex), which causes some of the index
and length calculations to be incorrect in the collect and write methods. It
can allow data to keep being written to the buffer even though it is already
full, with data overwriting metadata in the buffer and vice versa. I have seen
this manifest as the NegativeArraySizeException seen in MAPREDUCE-6907 as well
as in ArrayIndexOutOfBoundsException and EOFException.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
