[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=487103&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487103
]
ASF GitHub Bot logged work on HIVE-24179:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 21/Sep/20 17:23
Start Date: 21/Sep/20 17:23
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r492225545
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##########
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
- HiveTxnManager txnManager = null;
+ boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
That's correct. Normally, we should (but I am not 100% sure) have a
TxnManager at this point so there is no need to create a new one just to obtain
the flag. I pushed commit
https://github.com/apache/hive/pull/1509/commits/297882ee80d52689a9cc1c68da9f7580918439bb
to try this out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 487103)
Time Spent: 40m (was: 0.5h)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> -------------------------------------------------------------------
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
> at
> org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletio
> at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
> at
> org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2008)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1867)
> at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
> {noformat}
> The heap dump shows (summary.png) that most of the memory is consumed by
> {{Hashtable$Entry}} and {{ConcurrentHashMap$Node}} objects coming from Hive
> configurations referenced by {{DbTxnManager}}.
> The latter are not eligible for garbage collection since at
> [construction|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L212]
> time they are passed implicitly in a callback stored inside
> ShutdownHookManager.
> When the {{DbTxnManager}} is closed properly the leak is not present since
> the callback is
> [removed|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L882]
> from ShutdownHookManager.
> {{SHOW LOCKS}} statements create
> ([ShowDbLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java#L52],
>
> [ShowLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowLocksAnalyzer.java#L72])
> a new {{TxnManager}} and they never close it leading to the memory leak.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
