Stamatis Zampetakis created HIVE-24179:
------------------------------------------
Summary: Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS
statement
Key: HIVE-24179
URL: https://issues.apache.org/jira/browse/HIVE-24179
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis
Fix For: 4.0.0
Attachments: summary.png
The problem can be reproduced by executing repeatedly a SHOW LOCK statement and
monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only takes a
few minutes before the server crashes with OutOfMemory error such as the one
shown below.
{noformat}
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:3332)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at
org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
at
org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
at
org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
at
org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
at
org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
at
org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
at
org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
at
org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
at
org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
at
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
at
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
at
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
at
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
at
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
at
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletio
at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
at
org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
at
org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2
at
org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
at
org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2008)
at
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1867)
at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
{noformat}
The heap dump shows (summary.png) that most of the memory is consumed by
{{Hashtable$Entry}} and {{ConcurrentHashMap$Node}} objects coming from Hive
configurations referenced by {{DbTxnManager}}.
The latter are not eligible for garbage collection since at
[construction|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L212]
time they are passed implicitly in a callback stored inside
ShutdownHookManager.
When the {{DbTxnManager}} is closed properly the leak is not present since the
callback is
[removed|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L882]
from ShutdownHookManager.
{{SHOW LOCKS}} statements create
([ShowDbLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java#L52],
[ShowLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowLocksAnalyzer.java#L72])
a new {{TxnManager}} and they never close it leading to the memory leak.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
