[
https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991949#comment-15991949
]
Harish Jaiprakash edited comment on TEZ-3673 at 5/2/17 6:41 AM:
----------------------------------------------------------------
[~sseth] Please take a look. Couple of questions for computeNumBuffersAndSize:
* Should I subtract 64 bytes in all sizePerBuffer assignments? Currently only
if sizePerBuffer is set because of bufferLimit parameter it does the
correction. Assuming that bufferLimit was set to be some multiple of region
size. Reason being that any other value might not be at region size boundary
and correction may not be helpful.
* There is no logging or exception for wrong buffer merge percent, it silently
corrects to be within limit. Is that alright?
was (Author: harishjp):
[~seth] Please take a look. Couple of questions for computeNumBuffersAndSize:
* Should I subtract 64 bytes in all sizePerBuffer assignments? Currently only
if sizePerBuffer is set because of bufferLimit parameter it does the
correction. Assuming that bufferLimit was set to be some multiple of region
size. Reason being that any other value might not be at region size boundary
and correction may not be helpful.
* There is no logging or exception for wrong buffer merge percent, it silently
corrects to be within limit. Is that alright?
> Allocate smaller buffers in UnorderedPartitionedKVWriter
> --------------------------------------------------------
>
> Key: TEZ-3673
> URL: https://issues.apache.org/jira/browse/TEZ-3673
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Harish Jaiprakash
> Assignee: Harish Jaiprakash
> Attachments: TEZ-3673.01.patch, TEZ-3673.02.patch, TEZ-3673.03.patch,
> TEZ-3673.04.patch
>
>
> UnorderedPartitionedKVWriter allocates in bigger chunks. It may or may not
> get filled up. In PipelinedSorter, we start off with 32MB chunks. But
> UnorderedPartitionedKVWriter can be worse as it allocates bigger blocks. Need
> to revisit this allocation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
