[impala] branch master updated: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory

[boroknagyz]

This is an automated email from the ASF dual-hosted git repository.

boroknagyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


The following commit(s) were added to refs/heads/master by this push:
     new b816b6e39 IMPALA-11753: CatalogD OOMkilled due to natively allocated 
memory
b816b6e39 is described below

commit b816b6e3980d41d49644dc8909a9b4c845bed31a
Author: Zoltan Borok-Nagy <borokna...@cloudera.com>
AuthorDate: Mon Nov 28 18:19:10 2022 +0100

    IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
    
    CatalogD can be OOMKilled due to too much natively allocated memory.
    The bug is due to a misuse of a Java compression API:
    https://bugs.openjdk.org/browse/JDK-8257032
    
    The problem is that we create our own Deflater object and pass it
    to the constructor of DeflaterOutputStream:
    
https://github.com/apache/impala/blob/84fa6d210d3966e5ece8b4ac84ff8bd8780dec4e/fe/src/main/java/org/apache/impala/util/CompressionUtil.java#L47
    
    This means that Java's DeflaterOutputStream won't assume ownership on
    the Deflater, and won't invoke its end() method:
    
    * 
https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L144
    * 
https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L246-L247
    
    The Deflater's methods are implemented in C and allocate native memory.
    This means that until the GC doesn't destroy the unreachable Deflater
    objects they can consume quite much native memory. In some scenarios
    it can even result in OOMKills by the kernel.
    
    The fix is to override the DeflaterOutputStream's close() method so
    it invokes end() on the Deflater object.
    
    Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
    Reviewed-on: http://gerrit.cloudera.org:8080/19282
    Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
    Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
---
 .../main/java/org/apache/impala/util/CompressionUtil.java   | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/fe/src/main/java/org/apache/impala/util/CompressionUtil.java 
b/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
index f6e11a110..cfa39c914 100644
--- a/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
+++ b/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
@@ -44,7 +44,18 @@ public class CompressionUtil {
     // Deflater with 'BEST_SPEED' level provided reasonable compression ratios 
at much
     // faster speeds compared to other modes like 
BEST_COMPRESSION/DEFAULT_COMPRESSION.
     DeflaterOutputStream stream =
-        new DeflaterOutputStream(bos, new Deflater(Deflater.BEST_SPEED));
+        new DeflaterOutputStream(bos, new Deflater(Deflater.BEST_SPEED)) {
+          // IMPALA-11753: to avoid CatalogD OOM we invoke def.end() which 
frees
+          // the natively allocated memory by the Deflater. See details in 
Jira.
+          @Override
+          public void close() throws IOException {
+            try {
+              super.close();
+            } finally {
+              def.end();
+            }
+          }
+        };
     try {
       stream.write(input);
       stream.close();