[
https://issues.apache.org/jira/browse/TEZ-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14692489#comment-14692489
]
Rohini Palaniswamy commented on TEZ-2615:
-----------------------------------------
As a first step it would be great if we could start releasing the memory and
add support for requesting additional later. Would greatly improve the
performance of tasks when more free memory is available. The regular processing
also uses a lot of memory apart from the inputs and outputs which use memory
for fetching and sorting.
> Consider allocating completed fetcher memory to other running tasks for more
> in-memory fetches
> ----------------------------------------------------------------------------------------------
>
> Key: TEZ-2615
> URL: https://issues.apache.org/jira/browse/TEZ-2615
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
>
> - In large jobs, It might be possible to do more in-memory fetches, if tez
> can allocate more memory to the running input. After fetching is complete
> (consider multi-input scenario), its memory can be given back to memory
> distributor via context itself which can be accumulated back in distributor.
> Components needing additional memory (e.g MergeManager.reserve, In memory
> mem-to-mem etc) can register for additional memory via context. When memory
> becomes available, callback is used by framework to indicate the availability
> of memory.
> - Similar thing can be considered for pipelinedsorter as well (at a later
> stage, as there would be lots of corner cases to deal with).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
