Muhammad Samir Khan created TEZ-3809:
----------------------------------------
Summary: The buffer size allocated for InMemoryMapOutput can be
optimized
Key: TEZ-3809
URL: https://issues.apache.org/jira/browse/TEZ-3809
Project: Apache Tez
Issue Type: Bug
Reporter: Muhammad Samir Khan
Assignee: Muhammad Samir Khan
Related jiras: TEZ-3752 and TEZ-3732.
-When shuffling input to memory, the decompressed length is used to create the
InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 bytes
less (the IFile header). These 4 bytes can optimized and, in an extreme case of
10,000,000 fetches, can save ~38 MB (TEZ-3732).
-Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers to
allocate the new InMemoryMapOutput. However, each input has two EOF_MARKERs
while only two are needed at the end.
-InMemoryWriter wraps the output BoundedByteArrayOutputStream in
IFileOutputStream which will write checksum at close. This can create a
mismatch between the primary input buffers which don't have checksum and the
merged buffers which do. IFileOutputStream wrap can be removed to save 4 bytes
per merged buffers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
