[
https://issues.apache.org/jira/browse/TEZ-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277456#comment-14277456
]
Siddharth Seth commented on TEZ-1945:
-------------------------------------
+1. Patch looks good to me. Thanks [~rajesh.balamohan].
Does the NUM_MEM_TO_DISK_MERGES counter seem incorrect ? I'd expect
SPILLED_RECORDS to be 0 in the cases where NUM_MEM_TO_DISK_MERGES,
NUM_DISK_TO_DISK_MERGES and SHUFFLE_BYTES_TO_DISK are 0. Could be because of
SHUFFLE_BYTES_DISK_DIRECT.
> Remove 2 GB memlimit restriction in MergeManager
> ------------------------------------------------
>
> Key: TEZ-1945
> URL: https://issues.apache.org/jira/browse/TEZ-1945
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: TEZ-1945.1.patch
>
>
> In certain situations (data coming in larger chunks, but yet to complete),
> fetchers might wait in MerManager.waitForShuffleToMergeMemory() for memory to
> become available.
> Removing the 2 GB resitrction on MergeManager.memlimit would help in such
> situations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
