[jira] [Assigned] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter reassigned HIVE-20203: --- Assignee: Eric Wohlstadter (was: SAUVAGEAU Eric) > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch, > HIVE-20203.3.patch, HIVE-20203.4.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter reassigned HIVE-20203: --- Assignee: SAUVAGEAU Eric (was: M. Arshad Khan) > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: SAUVAGEAU Eric >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch, > HIVE-20203.3.patch, HIVE-20203.4.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter reassigned HIVE-20203: --- > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if their is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
