[
https://issues.apache.org/jira/browse/TEZ-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633069#comment-14633069
]
Rajesh Balamohan commented on TEZ-2615:
---------------------------------------
-TezTaskContextImpl can have a new method, which can be used to report unused
memory by MergeManager, SimpleFetchedInputAllocator etc. MemoryDistributor can
track "freedMemory" separately.
-Whenever more memory is needed by other inputs (e.g in-memory fetching),
inputs can possibly request for additional
memory via inputcontext. If additional memory is provided by MemoryDistributor
via callback, it can be used to avoid any stall.
[~sseth] - Currently memory allocation for different inputs is done by calling
MemoryDistributor-->makeInitialAllocations() from
LogicalIOProcessorRuntimeTask. For the available unused memory, should it be
allocated on first-come first-serve model via callback? Or even these
additional memory allocations should be routed via the appropriate
memorydistributor implementations?.
> Consider allocating completed fetcher memory to other running tasks for more
> in-memory fetches
> ----------------------------------------------------------------------------------------------
>
> Key: TEZ-2615
> URL: https://issues.apache.org/jira/browse/TEZ-2615
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
>
> - In large jobs, It might be possible to do more in-memory fetches, if tez
> can allocate more memory to the running input. After fetching is complete
> (consider multi-input scenario), its memory can be given back to memory
> distributor via context itself which can be accumulated back in distributor.
> Components needing additional memory (e.g MergeManager.reserve, In memory
> mem-to-mem etc) can register for additional memory via context. When memory
> becomes available, callback is used by framework to indicate the availability
> of memory.
> - Similar thing can be considered for pipelinedsorter as well (at a later
> stage, as there would be lots of corner cases to deal with).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
