[
https://issues.apache.org/jira/browse/PARQUET-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765908#comment-16765908
]
ASF GitHub Bot commented on PARQUET-1485:
-----------------------------------------
gszadovszky commented on pull request #581: PARQUET-1485: Fix Snappy direct
memory leak
URL: https://github.com/apache/parquet-mr/pull/581
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Snappy Decompressor/Compressor may cause direct memory leak
> -----------------------------------------------------------
>
> Key: PARQUET-1485
> URL: https://issues.apache.org/jira/browse/PARQUET-1485
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: 1.8.1
> Environment: parquet-1.8.1
> spark2.1
> Reporter: liupengcheng
> Assignee: liupengcheng
> Priority: Major
> Labels: pull-request-available
>
> In out production environment, we encountered a direct memory oom issues
> caused by the direct buffer not released in time.
> After carefully checked the code, it seems that the some methods of
> SnappyDecompressor/SnappyCompressor would not release the direct buffer
> manually. If too much direct memory allocated and no GC happens, this bug may
> result in direct memory oom.
> Moreover, if the `-XX:+DisableImplicitGC` jvm option is specified, the direct
> memory oom would happen easily for large datasets.
> Seems that the problem still exist in the latest code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
