[
https://issues.apache.org/jira/browse/TEZ-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated TEZ-711:
-------------------------------
Attachment: TEZ-711.wip.4.txt
Updated to change ScatterGather shuffle to use HttpClient. Also fixes some
other shutdown issues with SG shuffle.
> Fix memory leak when not reading from inputs due to caching
> -----------------------------------------------------------
>
> Key: TEZ-711
> URL: https://issues.apache.org/jira/browse/TEZ-711
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.2.0
> Reporter: Rohini Palaniswamy
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: TEZ-711.wip.1.txt, TEZ-711.wip.2.txt, TEZ-711.wip.3.txt,
> TEZ-711.wip.4.txt
>
>
> When you are reading from inputs and caching objects with vertex scope, you
> don't have to read the input again when container is reused. But it allocates
> memory and that leaks causing OOM. KeyValueReader does not have a API to
> close the reader to clear allotted memory without reading from it. Also if
> there was a option to pre-close inputs in Processor and not fetch input at
> all over the wire and do shuffle/sort it would be a good optimization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
