[
https://issues.apache.org/jira/browse/PARQUET-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liupengcheng updated PARQUET-1485:
----------------------------------
Description:
In out production environment, we encountered a direct memory oom issues caused
by the direct buffer not released in time.
After carefully checked the code, it seems that the some methods of
SnappyDecompressor/SnappyCompressor would not release the direct buffer
manually. If too much direct memory allocated and no GC happens, this bug may
result in direct memory oom.
Moreover, if the `-XX:+DisableImplicitGC` jvm option is specified, the direct
memory oom would happen easily for large datasets.
Seems that the problem still exist in the latest code.
was:
In out production environment, we encountered a direct memory oom issues caused
by the direct buffer not released in time.
After carefully checked the code, it seems that the some methods of
SnappyDecompressor/SnappyCompressor would not release the direct buffer
manually, and if full gc not happen, this bug may result in direct memory oom.
Seems that the problem still exist in the latest code.
> Snappy Decompressor/Compressor may cause direct memory leak
> -----------------------------------------------------------
>
> Key: PARQUET-1485
> URL: https://issues.apache.org/jira/browse/PARQUET-1485
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: 1.8.1
> Environment: parquet-1.8.1
> spark2.1
> Reporter: liupengcheng
> Priority: Major
>
> In out production environment, we encountered a direct memory oom issues
> caused by the direct buffer not released in time.
> After carefully checked the code, it seems that the some methods of
> SnappyDecompressor/SnappyCompressor would not release the direct buffer
> manually. If too much direct memory allocated and no GC happens, this bug may
> result in direct memory oom.
> Moreover, if the `-XX:+DisableImplicitGC` jvm option is specified, the direct
> memory oom would happen easily for large datasets.
> Seems that the problem still exist in the latest code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
