[
https://issues.apache.org/jira/browse/TEZ-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534245#comment-16534245
]
Kuhu Shukla commented on TEZ-3954:
----------------------------------
Thank you Jon. +1. I will commit this tomorrow in case someone else wants to
chime in by then.
> Reduce Tez Shuffle Handler Memory needs for holding TezIndexRecords
> -------------------------------------------------------------------
>
> Key: TEZ-3954
> URL: https://issues.apache.org/jira/browse/TEZ-3954
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Major
> Attachments: TEZ-3954.001.patch
>
>
> Tez Shuffle Handler holds on to entire SpillRecord to better accommodate for
> range fetch operation. For single reduce fetch holding on to spillRecord can
> overwhelm memory if Index Cache is overrun. In the case of single fetch, only
> keep the single tez index record needed and for range fetch, keep the entire
> spill record in memory, but free it after it is needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
