[
https://issues.apache.org/jira/browse/TEZ-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639646#comment-14639646
]
Siddharth Seth commented on TEZ-2615:
-------------------------------------
The overall memory would have to be tracked somewhere. MemoryDistributor seems
to be the right place for that.
How would components request additional memory ? Right after the initial
distribution, they could register a request for additional memory which they
may need. Alternately, deeper inside the code they could request for smaller
chunks of memory ?
> Consider allocating completed fetcher memory to other running tasks for more
> in-memory fetches
> ----------------------------------------------------------------------------------------------
>
> Key: TEZ-2615
> URL: https://issues.apache.org/jira/browse/TEZ-2615
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
>
> - In large jobs, It might be possible to do more in-memory fetches, if tez
> can allocate more memory to the running input. After fetching is complete
> (consider multi-input scenario), its memory can be given back to memory
> distributor via context itself which can be accumulated back in distributor.
> Components needing additional memory (e.g MergeManager.reserve, In memory
> mem-to-mem etc) can register for additional memory via context. When memory
> becomes available, callback is used by framework to indicate the availability
> of memory.
> - Similar thing can be considered for pipelinedsorter as well (at a later
> stage, as there would be lots of corner cases to deal with).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
