[
https://issues.apache.org/jira/browse/PARQUET-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978802#comment-16978802
]
liupengcheng commented on PARQUET-1485:
---------------------------------------
[~yumwang] I think we currently dont backport the patch too all prior versions
is for consideration of maintenance, and hope users can keep up new versions.
Actually, you can backport this by yourself.
> Snappy Decompressor/Compressor may cause direct memory leak
> -----------------------------------------------------------
>
> Key: PARQUET-1485
> URL: https://issues.apache.org/jira/browse/PARQUET-1485
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: 1.8.1
> Environment: parquet-1.8.1
> spark2.1
> Reporter: liupengcheng
> Assignee: liupengcheng
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> In out production environment, we encountered a direct memory oom issues
> caused by the direct buffer not released in time.
> After carefully checked the code, it seems that the some methods of
> SnappyDecompressor/SnappyCompressor would not release the direct buffer
> manually. If too much direct memory allocated and no GC happens, this bug may
> result in direct memory oom.
> Moreover, if the `-XX:+DisableImplicitGC` jvm option is specified, the direct
> memory oom would happen easily for large datasets.
> Seems that the problem still exist in the latest code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
