[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=486849&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486849
]
ASF GitHub Bot logged work on HIVE-24179:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 21/Sep/20 09:58
Start Date: 21/Sep/20 09:58
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r491922128
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##########
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
- HiveTxnManager txnManager = null;
+ boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
Do we create txnManager instance here just to get the value of
useNewLocksFormat flag?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 486849)
Time Spent: 20m (was: 10m)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> -------------------------------------------------------------------
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
> at
> org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletio
> at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
> at
> org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2008)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1867)
> at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
> {noformat}
> The heap dump shows (summary.png) that most of the memory is consumed by
> {{Hashtable$Entry}} and {{ConcurrentHashMap$Node}} objects coming from Hive
> configurations referenced by {{DbTxnManager}}.
> The latter are not eligible for garbage collection since at
> [construction|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L212]
> time they are passed implicitly in a callback stored inside
> ShutdownHookManager.
> When the {{DbTxnManager}} is closed properly the leak is not present since
> the callback is
> [removed|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L882]
> from ShutdownHookManager.
> {{SHOW LOCKS}} statements create
> ([ShowDbLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java#L52],
>
> [ShowLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowLocksAnalyzer.java#L72])
> a new {{TxnManager}} and they never close it leading to the memory leak.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
