Rajesh Balamohan created TEZ-2615:
-------------------------------------
Summary: Consider allocating completed fetcher memory to other
running tasks for more in-memory fetches
Key: TEZ-2615
URL: https://issues.apache.org/jira/browse/TEZ-2615
Project: Apache Tez
Issue Type: Improvement
Reporter: Rajesh Balamohan
- In large jobs, It might be possible to do more in-memory fetches, if tez can
allocate more memory to the running input. After fetching is complete
(consider multi-input scenario), its memory can be given back to memory
distributor via context itself which can be accumulated back in distributor.
Components needing additional memory (e.g MergeManager.reserve, In memory
mem-to-mem etc) can register for additional memory via context. When memory
becomes available, callback is used by framework to indicate the availability
of memory.
- Similar thing can be considered for pipelinedsorter as well (at a later
stage, as there would be lots of corner cases to deal with).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
