snodawn created HIVE-24605:
------------------------------
Summary: Metastore link leak in hiveserver2
Key: HIVE-24605
URL: https://issues.apache.org/jira/browse/HIVE-24605
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 2.1.0
Reporter: snodawn
When we use hiveserver and connect to metastore server, we face metastore link
leak as below:
{code:java}
2021-01-08T11:10:00,196 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
metadata.Hive (:()) - db.metaStoreClient.isSameConfObj(c) is false
2021-01-08T11:10:00,196 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
hive.metastore (:()) - Mestastore configuration hive.metastore.filter.hook
changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to
org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
metadata.Hive (:()) - db.metaStoreClient.isCompatibleWith(c) is false
2021-01-08T11:10:00,196 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
hive.metastore (:()) - Mestastore configuration hive.metastore.filter.hook
changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to
org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
metadata.Hive (:()) - hive db is org.apache.hadoop.hive.ql.metadata.Hive@ec83e59
2021-01-08T11:10:00,196 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
metadata.Hive (:()) -
METASTORE_FILTER_HOOK:org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
metadata.Hive (:()) - db.metaStoreClient.isSameConfObj(c) is false
2021-01-08T11:10:00,196 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
hive.metastore (:()) - Mestastore configuration hive.metastore.filter.hook
changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to
org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
metadata.Hive (:()) - db.metaStoreClient.isCompatibleWith(c) is false
2021-01-08T11:10:00,196 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
hive.metastore (:()) - Mestastore configuration hive.metastore.filter.hook
changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to
org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
metadata.Hive (:()) - Closing current thread's connection to Hive Metastore.
2021-01-08T11:10:00,201 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
hive.metastore (:()) - Closed a connection to metastore, current connections: 0
2021-01-08T11:10:00,201 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
metadata.Hive (:()) - Closing current thread's connection to Hive Metastore.
2021-01-08T11:10:00,201 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
metadata.Hive (:()) - current Hive is
org.apache.hadoop.hive.ql.metadata.Hive@5898c9e7
2021-01-08T11:10:00,202 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
hive.metastore (:()) - Trying to connect to metastore with URI thrift://xxx
2021-01-08T11:10:00,206 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
hive.metastore (:()) - Opened a connection to metastore, current connections: 1
2021-01-08T11:10:00,206 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
hive.metastore (:()) - Connected to metastore.
2021-01-08T11:10:00,215 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
parse.SemanticAnalyzer (:()) - Completed phase 1 of Semantic Analysis
2021-01-08T11:10:00,215 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
parse.SemanticAnalyzer (:()) - Get metadata for source tables
2021-01-08T11:10:00,215 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
parse.SemanticAnalyzer (:()) - current db in getMetaData is
org.apache.hadoop.hive.ql.metadata.Hive@Hive@ec83e59
2021-01-08T11:10:00,215 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
metadata.Hive (:()) - current Hive is
org.apache.hadoop.hive.ql.metadata.Hive@5898c9e7
2021-01-08T11:10:00,215 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
hive.metastore (:()) - Trying to connect to metastore with URI thrift://xxx
2021-01-08T11:10:00,218 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
hive.metastore (:()) - Opened a connection to metastore, current connections: 2
2021-01-08T11:10:00,219 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
hive.metastore (:()) - Connected to metastore.
.....
2021-01-08T11:12:16,119 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
metadata.Hive (:()) - metadata.Hive (:()) - close current:
org.apache.hadoop.hive.ql.metadata.Hive@5898c9e7
2021-01-08T11:12:16,121 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
metadata.Hive (:()) - Closing current thread's connection to Hive Metastore,
current is org.apache.hadoop.hive.ql.metadata.Hive@5898c9e7
2021-01-08T11:12:16,121 INFO [qid=61_142_176_249_26122_20210108110957686_0]:
hive.metastore (:()) - Closed a connection to metastore, current connections: 1
{code}
because HIVE_AUTHORIZATION_MANAGER is set to
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory
when in SessionState.setAuthorizerV2Config, sessionConf is changed, so
Hive.create will change and replace the old one in Hive.hiveDB. But the old
Hive instance is already assigned in BaseSemanticAnalyzer.db, and when it's
called, metaStoreClient will be assigned to a new one and create a new
metastore link. The metastoreClient in new Hive instance can be closed, but
the old one assigned in BaseSemanticAnalyzer will not close when we call
Hive.close and Hive.closeCurrent, which may lead to a link leak.
So, can we use Hive.get() instead of using db.xxx in BaseSemanticAnalyzer, or
it's better to re-assign to the new Hive instance in BaseSemanticAnalyzer
instead of using the old one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
