[
https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated TEZ-3673:
-----------------------------------
Attachment: TEZ-3673.03.patch
* Changed config to buffer-merge-percent.
* Sending a copy of filledBuffers.
* Added test case to test spillLimit.
* Fixed a race conditions in counters.
Found another race condition, we do not wait for tasks to finish before we
start finalMerge. I'll file a separate jira and send out a fix. We should delay
this fix until that because this patch seems to trigger the other bug more
frequently.
> Allocate smaller buffers in UnorderedPartitionedKVWriter
> --------------------------------------------------------
>
> Key: TEZ-3673
> URL: https://issues.apache.org/jira/browse/TEZ-3673
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Harish Jaiprakash
> Assignee: Harish Jaiprakash
> Attachments: TEZ-3673.01.patch, TEZ-3673.02.patch, TEZ-3673.03.patch
>
>
> UnorderedPartitionedKVWriter allocates in bigger chunks. It may or may not
> get filled up. In PipelinedSorter, we start off with 32MB chunks. But
> UnorderedPartitionedKVWriter can be worse as it allocates bigger blocks. Need
> to revisit this allocation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
