Rajesh Balamohan created TEZ-3142:
-------------------------------------
Summary: Intermediate mem-to-mem enhancements
Key: TEZ-3142
URL: https://issues.apache.org/jira/browse/TEZ-3142
Project: Apache Tez
Issue Type: Improvement
Reporter: Rajesh Balamohan
-InMemoryMapOutputs is already sorted by size. This information will be
retained in the generic startMerge arrayCopy - so the inputs list should
already be sorted. We could walk the list starting from the lowest to the
highest to try and merge as many segments as possible, and then terminate,
instead of trying to look for more segments.
-InMem merged segments make it back into inMemoryMergedMapOutputs. That makes
them ineligible for subsequent memToMem merges. These could be considered as
well - especially when handling extremely small segments.
Ref: TEZ-1911
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
