[
https://issues.apache.org/jira/browse/TEZ-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-1945:
----------------------------------
Attachment: TEZ-1945.1.patch
Attaching initial patch to remove 2 GB check. There is a corner case wherein,
this can break Intermediate mem-to-mem merging as it relies InMemoryWriter and
InMemoryWriter is currently bound by 2 GB limit. One option could be to do
intemediate mem-to-mem merging "only" till 2 GB in createInMemorySegments() for
processing.
> Remove 2 GB memlimit restriction in MergeManager
> ------------------------------------------------
>
> Key: TEZ-1945
> URL: https://issues.apache.org/jira/browse/TEZ-1945
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Attachments: TEZ-1945.1.patch
>
>
> In certain situations (data coming in larger chunks, but yet to complete),
> fetchers might wait in MerManager.waitForShuffleToMergeMemory() for memory to
> become available.
> Removing the 2 GB resitrction on MergeManager.memlimit would help in such
> situations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
