[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=511446=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-511446
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 13/Nov/20 17:24
Start Date: 13/Nov/20 17:24
Worklog Time Spent: 10m
Work Description: jcamachor merged pull request #1509:
URL: https://github.com/apache/hive/pull/1509
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 511446)
Time Spent: 2h 10m (was: 2h)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
> at
> org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletio
> at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
> at
> org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2008)
> at
>
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=489100=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489100
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 23/Sep/20 04:35
Start Date: 23/Sep/20 04:35
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r492510139
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
LGTM
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 489100)
Time Spent: 2h (was: 1h 50m)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
> at
>
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=489093=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489093
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 23/Sep/20 04:34
Start Date: 23/Sep/20 04:34
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r492364790
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
Did you check the last commit? Do you have something else in mind?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 489093)
Time Spent: 1h 50m (was: 1h 40m)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at
>
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=487996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487996
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 22/Sep/20 06:55
Start Date: 22/Sep/20 06:55
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r492510139
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
LGTM
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 487996)
Time Spent: 1h 40m (was: 1.5h)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
> at
>
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=487718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487718
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 22/Sep/20 03:29
Start Date: 22/Sep/20 03:29
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r491922128
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
Do we create txnManager instance here just to get the value of
useNewLocksFormat flag?
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
Do we create txnManager instance here just to get the value of
useNewLocksFormat flag?
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
TxnManager is initialised before query compilation and has a session
scope. Correct way to access it is via SessionState. Looks like
ShowDbLocksAnalyzer was always creating new instance of TxnManager.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
TxnManager is initialized before query compilation and has a session
scope. Correct way to access it is via SessionState. Looks like
ShowDbLocksAnalyzer was always creating new instance of TxnManager.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 487718)
Time Spent: 1.5h (was: 1h 20m)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=487707=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487707
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 22/Sep/20 03:29
Start Date: 22/Sep/20 03:29
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r492225545
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
That's correct. Normally, we should (but I am not 100% sure) have a
TxnManager at this point so there is no need to create a new one just to obtain
the flag. I pushed commit
https://github.com/apache/hive/pull/1509/commits/297882ee80d52689a9cc1c68da9f7580918439bb
to try this out.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
Did you check the last commit? Do you have something else in mind?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 487707)
Time Spent: 1h 20m (was: 1h 10m)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
>
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=487298=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487298
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 21/Sep/20 21:47
Start Date: 21/Sep/20 21:47
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r492364790
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
Did you check the last commit? Do you have something else in mind?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 487298)
Time Spent: 1h 10m (was: 1h)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at
>
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=487172=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487172
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 21/Sep/20 18:29
Start Date: 21/Sep/20 18:29
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r492262911
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
TxnManager is initialized before query compilation and has a session
scope. Correct way to access it is via SessionState. Looks like
ShowDbLocksAnalyzer was always creating new instance of TxnManager.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 487172)
Time Spent: 1h (was: 50m)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
>
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=487170=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487170
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 21/Sep/20 18:28
Start Date: 21/Sep/20 18:28
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r492262911
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
TxnManager is initialised before query compilation and has a session
scope. Correct way to access it is via SessionState. Looks like
ShowDbLocksAnalyzer was always creating new instance of TxnManager.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 487170)
Time Spent: 50m (was: 40m)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
>
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=487103=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487103
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 21/Sep/20 17:23
Start Date: 21/Sep/20 17:23
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r492225545
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
That's correct. Normally, we should (but I am not 100% sure) have a
TxnManager at this point so there is no need to create a new one just to obtain
the flag. I pushed commit
https://github.com/apache/hive/pull/1509/commits/297882ee80d52689a9cc1c68da9f7580918439bb
to try this out.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 487103)
Time Spent: 40m (was: 0.5h)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
>
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=486867=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486867
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 21/Sep/20 11:24
Start Date: 21/Sep/20 11:24
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r491922128
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
Do we create txnManager instance here just to get the value of
useNewLocksFormat flag?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 486867)
Time Spent: 0.5h (was: 20m)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at
>
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=486849=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486849
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 21/Sep/20 09:58
Start Date: 21/Sep/20 09:58
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #1509:
URL: https://github.com/apache/hive/pull/1509#discussion_r491922128
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -47,14 +47,16 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
String dbName = stripQuotes(root.getChild(0).getText());
boolean isExtended = (root.getChildCount() > 1);
-HiveTxnManager txnManager = null;
+boolean useNewLocksFormat;
try {
- txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
+ HiveTxnManager txnManager =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
Review comment:
Do we create txnManager instance here just to get the value of
useNewLocksFormat flag?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 486849)
Time Spent: 20m (was: 10m)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at
>
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
[
https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=486428=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486428
]
ASF GitHub Bot logged work on HIVE-24179:
-
Author: ASF GitHub Bot
Created on: 18/Sep/20 23:12
Start Date: 18/Sep/20 23:12
Worklog Time Spent: 10m
Work Description: zabetak opened a new pull request #1509:
URL: https://github.com/apache/hive/pull/1509
### What changes were proposed in this pull request?
1. Avoid multiple shutdown hooks for the same threadpool in DbTxnManager
2. Close new HiveTxnManager in ShowLocksAnalyzer to release resources
### Why are the changes needed?
Avoid the memory leak when executing `SHOW LOCKS` statements and simplify
code.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
`mvn test -pl itests/hive-unit -Dtest=TestDbTxnManagerMemoryLeak -Pitests`
If it finishes then there is no memory leak, otherwise most of the time you
get an OutOfMemoryError.
See the JIRA case for more details.
**The commit with the test is not meant to be merged into master**
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 486428)
Remaining Estimate: 0h
Time Spent: 10m
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> ---
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement
> and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only
> takes a few minutes before the server crashes with OutOfMemory error such as
> the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at
> org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at
> org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at
> org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at
> org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at
>
