[
https://issues.apache.org/jira/browse/ARROW-16163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Li reassigned ARROW-16163:
--------------------------------
Assignee: Nicolas MOREAU
> [Go] IPC FileReader leaks memory when used with ZSTD compression
> ----------------------------------------------------------------
>
> Key: ARROW-16163
> URL: https://issues.apache.org/jira/browse/ARROW-16163
> Project: Apache Arrow
> Issue Type: Bug
> Components: Go
> Reporter: Nicolas MOREAU
> Assignee: Nicolas MOREAU
> Priority: Critical
> Labels: pull-request-available
> Fix For: 8.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I'll submit a PR :)
>
> When using the IPC Reader to read data compressed in zstd, we observed what
> looked like a memory leak.
> After further debugging, we observed that calling the IPC Reader created
> goroutines that were never stopped, and consumed memory.
>
> After deeper investigation, we found the following root cause:
>
> In arrow/ipc/file_reader.go, the newRecord function calls getDecompressor,
> that returns a zstd.Decoder object.
>
> When used in its stream version (calling .Read()), this decoder makes use of
> a Goroutine with a channel to receive input. This channel is closed by the
> decoder's .Close() function, which ends the Goroutine and therefore the
> collection of its memory by the GC.
>
> After using the zstd Decoder, the IPC FileReader should close it to prevent
> the leak.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
