Jason Dere created HIVE-22275:
---------------------------------
Summary: OperationManager.queryIdOperation does not properly clean
up multiple queryIds
Key: HIVE-22275
URL: https://issues.apache.org/jira/browse/HIVE-22275
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Jason Dere
Assignee: Jason Dere
In the case that multiple statements are run by a single Session before being
cleaned up, it appears that OperationManager.queryIdOperation is not cleaned up
properly.
See the log statements below - with the exception of the first "Removed
queryId:" log line, the queryId listed during cleanup is the same, when each of
these handles should have their own queryId. Looks like only the last queryId
executed is being cleaned up.
As a result, HS2 can run out of memory as OperationManager.queryIdOperation
grows and never cleans these queryIds/Operations up.
{noformat}
2019-09-13T08:37:36,785 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=dfed4c18-a284-4640-9f4a-1a20527105f9]
2019-09-13T08:37:38,432 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Removed queryId: hive_20190913083736_c49cf3cc-cfe8-48a1-bd22-8b924dfb0396
corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=dfed4c18-a284-4640-9f4a-1a20527105f9] with tag: null
2019-09-13T08:37:38,469 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=24d0030c-0e49-45fb-a918-2276f0941cfb]
2019-09-13T08:37:52,662 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=b983802c-1dec-4fa0-8680-d05ab555321b]
2019-09-13T08:37:56,239 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=75dbc531-2964-47b2-84d7-85b59f88999c]
2019-09-13T08:38:02,551 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=72c79076-9d67-4894-a526-c233fa5450b2]
2019-09-13T08:38:10,558 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=17b30a62-612d-4b70-9ba7-4287d2d9229b]
2019-09-13T08:38:16,930 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=ea97e99d-cc77-470b-b49a-b869c73a4615]
2019-09-13T08:38:20,440 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=a277b789-ebb8-4925-878f-6728d3e8c5fb]
2019-09-13T08:38:26,303 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=9a023ab8-aa80-45db-af88-94790cc83033]
2019-09-13T08:38:30,791 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=b697c801-7da0-4544-bcfa-442eb1d3bd77]
2019-09-13T08:39:10,187 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=bda93c8f-0822-4592-a61c-4701720a1a5c]
2019-09-13T08:39:15,471 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=24d0030c-0e49-45fb-a918-2276f0941cfb] with tag: null
2019-09-13T08:39:15,507 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=b983802c-1dec-4fa0-8680-d05ab555321b] with tag: null
2019-09-13T08:39:15,538 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=75dbc531-2964-47b2-84d7-85b59f88999c] with tag: null
2019-09-13T08:39:15,572 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=72c79076-9d67-4894-a526-c233fa5450b2] with tag: null
2019-09-13T08:39:15,600 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=17b30a62-612d-4b70-9ba7-4287d2d9229b] with tag: null
2019-09-13T08:39:15,629 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=ea97e99d-cc77-470b-b49a-b869c73a4615] with tag: null
2019-09-13T08:39:15,657 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=a277b789-ebb8-4925-878f-6728d3e8c5fb] with tag: null
2019-09-13T08:39:15,692 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=9a023ab8-aa80-45db-af88-94790cc83033] with tag: null
2019-09-13T08:39:15,723 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=b697c801-7da0-4544-bcfa-442eb1d3bd77] with tag: null
2019-09-13T08:39:15,754 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=bda93c8f-0822-4592-a61c-4701720a1a5c] with tag: null
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
