[
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Muhammad Samir Khan updated TEZ-3809:
-------------------------------------
Description:
Related jiras: TEZ-3752 and TEZ-3732.
-When shuffling input to memory, the decompressed length is used to create the
InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 bytes
less (the IFile header). These 4 bytes can optimized and, in an extreme case of
10,000,000 fetches, can save ~38 MB (TEZ-3732).
-Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers to
allocate the new InMemoryMapOutput. However, each input has two EOF_MARKERs
while only two are needed at the end.
-InMemoryWriter wraps the output BoundedByteArrayOutputStream in
IFileOutputStream which will write checksum at close. This can create a
mismatch between the primary input buffers which don't have checksum and the
merged buffers which do. IFileOutputStream wrap can be removed to save 4 bytes
per merged buffers.
-InMemoryWriter does not account for two EOF_MARKERs written at close() in its
accounting so that the getRawLength() method is off by two bytes.
was:
Related jiras: TEZ-3752 and TEZ-3732.
-When shuffling input to memory, the decompressed length is used to create the
InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 bytes
less (the IFile header). These 4 bytes can optimized and, in an extreme case of
10,000,000 fetches, can save ~38 MB (TEZ-3732).
-Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers to
allocate the new InMemoryMapOutput. However, each input has two EOF_MARKERs
while only two are needed at the end.
-InMemoryWriter wraps the output BoundedByteArrayOutputStream in
IFileOutputStream which will write checksum at close. This can create a
mismatch between the primary input buffers which don't have checksum and the
merged buffers which do. IFileOutputStream wrap can be removed to save 4 bytes
per merged buffers.
> The buffer size allocated for InMemoryMapOutput can be optimized
> ----------------------------------------------------------------
>
> Key: TEZ-3809
> URL: https://issues.apache.org/jira/browse/TEZ-3809
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Muhammad Samir Khan
> Assignee: Muhammad Samir Khan
>
> Related jiras: TEZ-3752 and TEZ-3732.
> -When shuffling input to memory, the decompressed length is used to create
> the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4
> bytes less (the IFile header). These 4 bytes can optimized and, in an extreme
> case of 10,000,000 fetches, can save ~38 MB (TEZ-3732).
> -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers
> to allocate the new InMemoryMapOutput. However, each input has two
> EOF_MARKERs while only two are needed at the end.
> -InMemoryWriter wraps the output BoundedByteArrayOutputStream in
> IFileOutputStream which will write checksum at close. This can create a
> mismatch between the primary input buffers which don't have checksum and the
> merged buffers which do. IFileOutputStream wrap can be removed to save 4
> bytes per merged buffers.
> -InMemoryWriter does not account for two EOF_MARKERs written at close() in
> its accounting so that the getRawLength() method is off by two bytes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
