[
https://issues.apache.org/jira/browse/TEZ-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534045#comment-16534045
]
Jonathan Eagles commented on TEZ-3954:
--------------------------------------
The patch design is a hybrid approach between a) optimizing for the single
partition fetch and b) optimizing for the ranged partition fetch. A single
partition fetch only stores (separate from the IndexCache) the IndexRecord and
not the whole TezSpillRecord with the ReduceContext. In this way if there are
many hundreds of outstanding fetch requests we can keep the memory demand low.
A ranged partition fetch will store the entire TezSpillRecord with the
ReduceContext. In this way we can reduce memory for ranged fetch by not
maintaining a separate list of IndexRecords for each partition fetched. In
addition, this patch "frees" the TezSpillRecord once it is no longer needed
(upstream task has only a spill record) in this way a ranged fetch over dozens
of upstream tasks can remove the memory demand for subsequent ranged partitions
fetches for sister upstream tasks.
> Reduce Tez Shuffle Handler Memory needs for holding TezIndexRecords
> -------------------------------------------------------------------
>
> Key: TEZ-3954
> URL: https://issues.apache.org/jira/browse/TEZ-3954
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Major
> Attachments: TEZ-3954.001.patch
>
>
> Tez Shuffle Handler holds on to entire SpillRecord to better accommodate for
> range fetch operation. For single reduce fetch holding on to spillRecord can
> overwhelm memory if Index Cache is overrun. In the case of single fetch, only
> keep the single tez index record needed and for range fetch, keep the entire
> spill record in memory, but free it after it is needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
