[
https://issues.apache.org/jira/browse/HIVE-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damon Cortesi updated HIVE-20600:
---------------------------------
Attachment: HIVE-20600.patch
> Metastore connection leak
> -------------------------
>
> Key: HIVE-20600
> URL: https://issues.apache.org/jira/browse/HIVE-20600
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 2.3.3
> Reporter: Damon Cortesi
> Priority: Major
> Attachments: HIVE-20600.patch, consume_threads.py
>
>
> Within the execute method of HiveServer2, there appears to be a connection
> leak. With fairly straightforward series of INSERT statements, the connection
> count in the logs continues to increase over time. Under certain loads, this
> can also consume all underlying threads of the Hive metastore and result in
> HS2 becoming unresponsive to new connections.
> The log below is the result of some python code executing a single insert
> statement, and then looping through a series of 10 more insert statements. We
> can see there's one dangling connection left open after each execution
> leaving us with 12 open connections (11 from the execute statements + 1 from
> HS2 startup).
> {code}
> 2018-09-19T17:14:32,108 INFO [main([])]: hive.metastore
> (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
> current connections: 1
> 2018-09-19T17:14:48,175 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b
> HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore
> (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
> current connections: 2
> 2018-09-19T17:15:05,543 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 1
> 2018-09-19T17:15:05,548 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 2
> 2018-09-19T17:15:05,932 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 1
> 2018-09-19T17:15:05,935 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 2
> 2018-09-19T17:15:06,123 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 1
> 2018-09-19T17:15:06,126 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 2
> ...
> 2018-09-19T17:15:20,626 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b
> HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore
> (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
> current connections: 12
> 2018-09-19T17:15:21,153 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 11
> 2018-09-19T17:15:21,155 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 12
> 2018-09-19T17:15:21,306 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 11
> 2018-09-19T17:15:21,308 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 12
> 2018-09-19T17:15:21,385 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 11
> 2018-09-19T17:15:21,387 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 12
> 2018-09-19T17:15:21,541 INFO [HiveServer2-Handler-Pool: Thread-31([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 13
> 2018-09-19T17:15:21,542 INFO [HiveServer2-Handler-Pool: Thread-31([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 12
> {code}
> Attached is a simple [impyla|https://github.com/cloudera/impyla] script that
> triggers the condition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
