Jonathan Eagles created TEZ-3954: ------------------------------------ Summary: Reduce Tez Shuffle Handler Memory needs for holding TezIndexRecords Key: TEZ-3954 URL: https://issues.apache.org/jira/browse/TEZ-3954 Project: Apache Tez Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles
Tez Shuffle Handler holds on to entire SpillRecord to better accommodate for range fetch operation. For single reduce fetch holding on to spillRecord can overwhelm memory if Index Cache is overrun. In the case of single fetch, only keep the single tez index record needed and for range fetch, keep the entire spill record in memory, but free it after it is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)