[
https://issues.apache.org/jira/browse/IMPALA-11753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy updated IMPALA-11753:
---------------------------------------
Description:
We can hit this bug in several installations:
https://bugs.openjdk.org/browse/JDK-8257032
(another nice description of the issue can be found here:
https://medium.com/swlh/native-memory-the-silent-jvm-killer-595913cba8e7)
The problem is that we are creating our own Deflater object and pass it to the
constructor of DeflaterOutputStream:
https://github.com/apache/impala/blob/84fa6d210d3966e5ece8b4ac84ff8bd8780dec4e/fe/src/main/java/org/apache/impala/util/CompressionUtil.java#L47
This means that Java's DeflaterOutputStream won't assume ownership on the
Deflater, and won't invoke its end() method:
*
https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L144
*
https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L246-L247
The Deflater's methods are implemented in C and allocate native memory. This
means that until the GC doesn't destroy the unreachable Deflater objects they
can consume quite much native memory. In some scenarios it can even result in
OOMKills by the kernel.
We should solve this issue by either
* override close() of DeflaterOutputStream (like mentioned in the above blog
post)
* directly invoke end() on the Deflater object in a finally clause
was:
We can hit this bug in several installations:
https://bugs.openjdk.org/browse/JDK-8257032
(another nice description of the issue can be found here:
https://medium.com/swlh/native-memory-the-silent-jvm-killer-595913cba8e7)
The problem is that we creating our own Deflater object and pass it to the
constructor of DeflaterOutputStream:
https://github.com/apache/impala/blob/84fa6d210d3966e5ece8b4ac84ff8bd8780dec4e/fe/src/main/java/org/apache/impala/util/CompressionUtil.java#L47
This means that Java's DeflaterOutputStream won't assume ownership on the
Deflater, and won't invoke its end() method:
*
https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L144
*
https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L246-L247
The Deflater's methods are implemented in C and allocate native memory. This
means that until the GC doesn't destroy the unreachable Deflater objects they
can consume quite much native memory. In some scenarios it can even result in
OOMKills by the kernel.
We should solve this issue by either
* override close() of DeflaterOutputStream (like mentioned in the above blog
post)
* directly invoke end() on the Deflater object in a finally clause
> CatalogD OOMkilled due to natively allocated memory
> ---------------------------------------------------
>
> Key: IMPALA-11753
> URL: https://issues.apache.org/jira/browse/IMPALA-11753
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Critical
> Labels: OOM
>
> We can hit this bug in several installations:
> https://bugs.openjdk.org/browse/JDK-8257032
> (another nice description of the issue can be found here:
> https://medium.com/swlh/native-memory-the-silent-jvm-killer-595913cba8e7)
> The problem is that we are creating our own Deflater object and pass it to
> the constructor of DeflaterOutputStream:
> https://github.com/apache/impala/blob/84fa6d210d3966e5ece8b4ac84ff8bd8780dec4e/fe/src/main/java/org/apache/impala/util/CompressionUtil.java#L47
> This means that Java's DeflaterOutputStream won't assume ownership on the
> Deflater, and won't invoke its end() method:
> *
> https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L144
> *
> https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L246-L247
> The Deflater's methods are implemented in C and allocate native memory. This
> means that until the GC doesn't destroy the unreachable Deflater objects they
> can consume quite much native memory. In some scenarios it can even result in
> OOMKills by the kernel.
> We should solve this issue by either
> * override close() of DeflaterOutputStream (like mentioned in the above blog
> post)
> * directly invoke end() on the Deflater object in a finally clause
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
