[jira] [Work logged] (HIVE-23830) Remove shutdownhook after query is completed
[ https://issues.apache.org/jira/browse/HIVE-23830?focusedWorklogId=457916=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-457916 ] ASF GitHub Bot logged work on HIVE-23830: - Author: ASF GitHub Bot Created on: 13/Jul/20 09:00 Start Date: 13/Jul/20 09:00 Worklog Time Spent: 10m Work Description: pvary merged pull request #1235: URL: https://github.com/apache/hive/pull/1235 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 457916) Time Spent: 1h 20m (was: 1h 10m) > Remove shutdownhook after query is completed > > > Key: HIVE-23830 > URL: https://issues.apache.org/jira/browse/HIVE-23830 > Project: Hive > Issue Type: Bug >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Each query registers a shutdownHook to release transactional resources in > case JVM shuts down mid query. These hooks are not cleaned up until session > is closed. Session life time is unbounded. So these hooks are a memory leak. > They should be cleaned as soon as transaction is completed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23830) Remove shutdownhook after query is completed
[ https://issues.apache.org/jira/browse/HIVE-23830?focusedWorklogId=457792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-457792 ] ASF GitHub Bot logged work on HIVE-23830: - Author: ASF GitHub Bot Created on: 12/Jul/20 21:22 Start Date: 12/Jul/20 21:22 Worklog Time Spent: 10m Work Description: mustafaiman commented on pull request #1235: URL: https://github.com/apache/hive/pull/1235#issuecomment-657277048 @pvary can you merge this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 457792) Time Spent: 1h 10m (was: 1h) > Remove shutdownhook after query is completed > > > Key: HIVE-23830 > URL: https://issues.apache.org/jira/browse/HIVE-23830 > Project: Hive > Issue Type: Bug >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Each query registers a shutdownHook to release transactional resources in > case JVM shuts down mid query. These hooks are not cleaned up until session > is closed. Session life time is unbounded. So these hooks are a memory leak. > They should be cleaned as soon as transaction is completed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23830) Remove shutdownhook after query is completed
[
https://issues.apache.org/jira/browse/HIVE-23830?focusedWorklogId=457211=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-457211
]
ASF GitHub Bot logged work on HIVE-23830:
-
Author: ASF GitHub Bot
Created on: 10/Jul/20 15:54
Start Date: 10/Jul/20 15:54
Worklog Time Spent: 10m
Work Description: mustafaiman commented on a change in pull request #1235:
URL: https://github.com/apache/hive/pull/1235#discussion_r452929806
##
File path: ql/src/java/org/apache/hadoop/hive/ql/DriverTxnHandler.java
##
@@ -553,11 +553,13 @@ private void release(boolean releaseLocks) {
LOG.warn("Exception when releasing locking in destroy: " +
e.getMessage());
}
}
-ShutdownHookManager.removeShutdownHook(shutdownRunner);
+ShutdownHookManager.removeShutdownHook(txnRollbackRunner);
}
void releaseLocksAndCommitOrRollback(boolean commit) throws LockException {
Review comment:
I would not rename to `commitAndCleanup` because this method also rolls
back transaction. I'll rename it to `endTransactionAndCleanup`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 457211)
Time Spent: 1h (was: 50m)
> Remove shutdownhook after query is completed
>
>
> Key: HIVE-23830
> URL: https://issues.apache.org/jira/browse/HIVE-23830
> Project: Hive
> Issue Type: Bug
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Each query registers a shutdownHook to release transactional resources in
> case JVM shuts down mid query. These hooks are not cleaned up until session
> is closed. Session life time is unbounded. So these hooks are a memory leak.
> They should be cleaned as soon as transaction is completed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23830) Remove shutdownhook after query is completed
[
https://issues.apache.org/jira/browse/HIVE-23830?focusedWorklogId=457046=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-457046
]
ASF GitHub Bot logged work on HIVE-23830:
-
Author: ASF GitHub Bot
Created on: 10/Jul/20 07:43
Start Date: 10/Jul/20 07:43
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #1235:
URL: https://github.com/apache/hive/pull/1235#discussion_r452677923
##
File path: ql/src/java/org/apache/hadoop/hive/ql/DriverTxnHandler.java
##
@@ -553,11 +553,13 @@ private void release(boolean releaseLocks) {
LOG.warn("Exception when releasing locking in destroy: " +
e.getMessage());
}
}
-ShutdownHookManager.removeShutdownHook(shutdownRunner);
+ShutdownHookManager.removeShutdownHook(txnRollbackRunner);
}
void releaseLocksAndCommitOrRollback(boolean commit) throws LockException {
Review comment:
After some more thought, I still was not able to come up with a really
good idea :(
The best I can think is that we should rename the method to
_commitAndCleanUp_
But I still open for any other ideas if you have some.
Thanks,
Peter
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 457046)
Time Spent: 50m (was: 40m)
> Remove shutdownhook after query is completed
>
>
> Key: HIVE-23830
> URL: https://issues.apache.org/jira/browse/HIVE-23830
> Project: Hive
> Issue Type: Bug
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Each query registers a shutdownHook to release transactional resources in
> case JVM shuts down mid query. These hooks are not cleaned up until session
> is closed. Session life time is unbounded. So these hooks are a memory leak.
> They should be cleaned as soon as transaction is completed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23830) Remove shutdownhook after query is completed
[
https://issues.apache.org/jira/browse/HIVE-23830?focusedWorklogId=457007=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-457007
]
ASF GitHub Bot logged work on HIVE-23830:
-
Author: ASF GitHub Bot
Created on: 10/Jul/20 06:19
Start Date: 10/Jul/20 06:19
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #1235:
URL: https://github.com/apache/hive/pull/1235#discussion_r452643677
##
File path: ql/src/java/org/apache/hadoop/hive/ql/DriverTxnHandler.java
##
@@ -553,11 +553,13 @@ private void release(boolean releaseLocks) {
LOG.warn("Exception when releasing locking in destroy: " +
e.getMessage());
}
}
-ShutdownHookManager.removeShutdownHook(shutdownRunner);
+ShutdownHookManager.removeShutdownHook(txnRollbackRunner);
}
void releaseLocksAndCommitOrRollback(boolean commit) throws LockException {
Review comment:
I do not have better idea than separate out to a cleanup method. I was
hoping you come up with a golden one :)
Doing cleanup in something called as "releaseLocksAndCommitOrRollback" is
definitely not a good idea IMHO.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 457007)
Time Spent: 40m (was: 0.5h)
> Remove shutdownhook after query is completed
>
>
> Key: HIVE-23830
> URL: https://issues.apache.org/jira/browse/HIVE-23830
> Project: Hive
> Issue Type: Bug
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Each query registers a shutdownHook to release transactional resources in
> case JVM shuts down mid query. These hooks are not cleaned up until session
> is closed. Session life time is unbounded. So these hooks are a memory leak.
> They should be cleaned as soon as transaction is completed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23830) Remove shutdownhook after query is completed
[
https://issues.apache.org/jira/browse/HIVE-23830?focusedWorklogId=457003=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-457003
]
ASF GitHub Bot logged work on HIVE-23830:
-
Author: ASF GitHub Bot
Created on: 10/Jul/20 06:07
Start Date: 10/Jul/20 06:07
Worklog Time Spent: 10m
Work Description: mustafaiman commented on a change in pull request #1235:
URL: https://github.com/apache/hive/pull/1235#discussion_r452639789
##
File path: ql/src/java/org/apache/hadoop/hive/ql/DriverTxnHandler.java
##
@@ -553,11 +553,13 @@ private void release(boolean releaseLocks) {
LOG.warn("Exception when releasing locking in destroy: " +
e.getMessage());
}
}
-ShutdownHookManager.removeShutdownHook(shutdownRunner);
+ShutdownHookManager.removeShutdownHook(txnRollbackRunner);
}
void releaseLocksAndCommitOrRollback(boolean commit) throws LockException {
Review comment:
This is the one method that is called from everywhere to end a
transaction in either way. Definitely more than a cleanup method. So I think we
should not rename it to something that signals it is just a cleanup. I did not
want to extract the two lines of clean up to another method either. What is
your suggestion?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 457003)
Time Spent: 0.5h (was: 20m)
> Remove shutdownhook after query is completed
>
>
> Key: HIVE-23830
> URL: https://issues.apache.org/jira/browse/HIVE-23830
> Project: Hive
> Issue Type: Bug
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Each query registers a shutdownHook to release transactional resources in
> case JVM shuts down mid query. These hooks are not cleaned up until session
> is closed. Session life time is unbounded. So these hooks are a memory leak.
> They should be cleaned as soon as transaction is completed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23830) Remove shutdownhook after query is completed
[
https://issues.apache.org/jira/browse/HIVE-23830?focusedWorklogId=456979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-456979
]
ASF GitHub Bot logged work on HIVE-23830:
-
Author: ASF GitHub Bot
Created on: 10/Jul/20 04:33
Start Date: 10/Jul/20 04:33
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #1235:
URL: https://github.com/apache/hive/pull/1235#discussion_r452616790
##
File path: ql/src/java/org/apache/hadoop/hive/ql/DriverTxnHandler.java
##
@@ -553,11 +553,13 @@ private void release(boolean releaseLocks) {
LOG.warn("Exception when releasing locking in destroy: " +
e.getMessage());
}
}
-ShutdownHookManager.removeShutdownHook(shutdownRunner);
+ShutdownHookManager.removeShutdownHook(txnRollbackRunner);
}
void releaseLocksAndCommitOrRollback(boolean commit) throws LockException {
Review comment:
This method starts to look like a cleanup.
Shall we separate out the functions to a different method, or just change
the name?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 456979)
Time Spent: 20m (was: 10m)
> Remove shutdownhook after query is completed
>
>
> Key: HIVE-23830
> URL: https://issues.apache.org/jira/browse/HIVE-23830
> Project: Hive
> Issue Type: Bug
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Each query registers a shutdownHook to release transactional resources in
> case JVM shuts down mid query. These hooks are not cleaned up until session
> is closed. Session life time is unbounded. So these hooks are a memory leak.
> They should be cleaned as soon as transaction is completed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23830) Remove shutdownhook after query is completed
[ https://issues.apache.org/jira/browse/HIVE-23830?focusedWorklogId=456931=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-456931 ] ASF GitHub Bot logged work on HIVE-23830: - Author: ASF GitHub Bot Created on: 10/Jul/20 01:05 Start Date: 10/Jul/20 01:05 Worklog Time Spent: 10m Work Description: mustafaiman opened a new pull request #1235: URL: https://github.com/apache/hive/pull/1235 Each query creates a shutdownRunner which will release transactional resources in case JVM exits mid-query. These shutdownRunners are not removed until session is closed. Session lifetime is unbounded. These shutdown hooks can cause OOM. We should remove them once query is committed or rolled back. Change-Id: I0b747bbfb2a451c2be1a28f21d4fb6354f76ee5a ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HIVE-X: Fix a typo in YYY) For more details, please see https://cwiki.apache.org/confluence/display/Hive/HowToContribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 456931) Remaining Estimate: 0h Time Spent: 10m > Remove shutdownhook after query is completed > > > Key: HIVE-23830 > URL: https://issues.apache.org/jira/browse/HIVE-23830 > Project: Hive > Issue Type: Bug >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Each query registers a shutdownHook to release transactional resources in > case JVM shuts down mid query. These hooks are not cleaned up until session > is closed. Session life time is unbounded. So these hooks are a memory leak. > They should be cleaned as soon as transaction is completed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
