[jira] [Commented] (TEZ-711) Fix memory leak when not reading from inputs due to caching
[ https://issues.apache.org/jira/browse/TEZ-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516264#comment-14516264 ] Hitesh Shah commented on TEZ-711: - [~rajesh.balamohan] [~sseth] is this still valid? Fix memory leak when not reading from inputs due to caching --- Key: TEZ-711 URL: https://issues.apache.org/jira/browse/TEZ-711 Project: Apache Tez Issue Type: Bug Affects Versions: 0.2.0 Reporter: Rohini Palaniswamy Assignee: Siddharth Seth Priority: Critical Attachments: OOM-threaddump-711-5-patch.txt, OOM-threaddump-till-TEZ-752.txt, TEZ-711.5.txt, TEZ-711.wip.1.txt, TEZ-711.wip.2.txt, TEZ-711.wip.3.txt, TEZ-711.wip.4.txt When you are reading from inputs and caching objects with vertex scope, you don't have to read the input again when container is reused. But it allocates memory and that leaks causing OOM. KeyValueReader does not have a API to close the reader to clear allotted memory without reading from it. Also if there was a option to pre-close inputs in Processor and not fetch input at all over the wire and do shuffle/sort it would be a good optimization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-711) Fix memory leak when not reading from inputs due to caching
[ https://issues.apache.org/jira/browse/TEZ-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516291#comment-14516291 ] Siddharth Seth commented on TEZ-711: Yes it is. Fix memory leak when not reading from inputs due to caching --- Key: TEZ-711 URL: https://issues.apache.org/jira/browse/TEZ-711 Project: Apache Tez Issue Type: Bug Affects Versions: 0.2.0 Reporter: Rohini Palaniswamy Assignee: Siddharth Seth Priority: Critical Attachments: OOM-threaddump-711-5-patch.txt, OOM-threaddump-till-TEZ-752.txt, TEZ-711.5.txt, TEZ-711.wip.1.txt, TEZ-711.wip.2.txt, TEZ-711.wip.3.txt, TEZ-711.wip.4.txt When you are reading from inputs and caching objects with vertex scope, you don't have to read the input again when container is reused. But it allocates memory and that leaks causing OOM. KeyValueReader does not have a API to close the reader to clear allotted memory without reading from it. Also if there was a option to pre-close inputs in Processor and not fetch input at all over the wire and do shuffle/sort it would be a good optimization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
