[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=486428&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486428
]
ASF GitHub Bot logged work on HIVE-24179:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Sep/20 23:12
Start Date: 18/Sep/20 23:12
Worklog Time Spent: 10m
Work Description: zabetak opened a new pull request #1509:
URL: https://github.com/apache/hive/pull/1509
### What changes were proposed in this pull request?
1. Avoid multiple shutdown hooks for the same threadpool in DbTxnManager
2. Close new HiveTxnManager in ShowLocksAnalyzer to release resources
### Why are the changes needed?
Avoid the memory leak when executing `SHOW LOCKS` statements and simplify
code.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
`mvn test -pl itests/hive-unit -Dtest=TestDbTxnManagerMemoryLeak -Pitests`
If it finishes then there is no memory leak, otherwise most of the time you
get an OutOfMemoryError.
See the JIRA case for more details.
**The commit with the test is not meant to be merged into master**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 486428)
Remaining Estimate: 0h
Time Spent: 10m
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> -------------------------------------------------------------------
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
> at
> org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletio
> at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
> at
> org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2008)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1867)
> at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
> {noformat}
> The heap dump shows (summary.png) that most of the memory is consumed by
> {{Hashtable$Entry}} and {{ConcurrentHashMap$Node}} objects coming from Hive
> configurations referenced by {{DbTxnManager}}.
> The latter are not eligible for garbage collection since at
> [construction|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L212]
> time they are passed implicitly in a callback stored inside
> ShutdownHookManager.
> When the {{DbTxnManager}} is closed properly the leak is not present since
> the callback is
> [removed|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L882]
> from ShutdownHookManager.
> {{SHOW LOCKS}} statements create
> ([ShowDbLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java#L52],
>
> [ShowLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowLocksAnalyzer.java#L72])
> a new {{TxnManager}} and they never close it leading to the memory leak.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
