[
https://issues.apache.org/jira/browse/HIVE-20844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20844:
-------------------------------
Summary: Cache Instances of LockManager in DummyTxnManager (was: Cache
Instances of CacheManager in DummyTxnManager)
> Cache Instances of LockManager in DummyTxnManager
> -------------------------------------------------
>
> Key: HIVE-20844
> URL: https://issues.apache.org/jira/browse/HIVE-20844
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Locking
> Affects Versions: 2.3.2, 3.1.0, 4.0.0
> Reporter: BELUGA BEHR
> Priority: Major
>
> I noticed that the {{DummyTxnManager}} class instantiates quite a few
> instances of {{ZooKeeperHiveLockManager}}. The ZooKeeper LM creates a
> connection to ZK for each instance created. It also does some initialization
> steps that are almost always just noise and pressure on ZooKeeper because it
> has already been initialized and the steps are therefore NOOPs.
> {{ZooKeeperHiveLockManager}} should be a singleton class with one long-lived
> connection to the ZooKeeper service. Perhaps the {{HiveLockManager}}
> interface could have a {{isSingleton()}} method which indicates that the LM
> should only be instantiated once and cached for subsequent sessions.
>
> {code:java}
> 2018-05-14 22:45:30,574 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1252389]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:27,865 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1252671]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:37,552 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1252686]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:49,046 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1252736]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:50,664 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1252742]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 23:00:54,314 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1253479]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 23:17:26,867 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1254180]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 23:24:25,426 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1255493]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> {code}
> {code:java|title=DummyTxnManager.java}
> @Override
> public HiveLockManager getLockManager() throws LockException {
> if (lockMgr == null) {
> boolean supportConcurrency =
> conf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
> if (supportConcurrency) {
> String lockMgrName =
> conf.getVar(HiveConf.ConfVars.HIVE_LOCK_MANAGER);
> if ((lockMgrName == null) || (lockMgrName.isEmpty())) {
> throw new LockException(ErrorMsg.LOCKMGR_NOT_SPECIFIED.getMsg());
> }
> try {
> // CACHE LM HERE
> LOG.info("Creating lock manager of type " + lockMgrName);
> lockMgr = (HiveLockManager)ReflectionUtils.newInstance(
> conf.getClassByName(lockMgrName), conf);
> lockManagerCtx = new HiveLockManagerCtx(conf);
> lockMgr.setContext(lockManagerCtx);
> } catch (Exception e) {
> ...
> {code}
> [https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockManager.java]
> {code:java|title=ZooKeeperHiveLockManager Initialization}
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/"
> + parent, new byte[0]);
> } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
> KeeperException.Code.NODEEXISTS) {
> LOG.warn("Unexpected ZK exception when creating parent node /" +
> parent, e);
> }
> }
> {code}
>
> https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java#L96-L106
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
