BELUGA BEHR created HIVE-20844:
----------------------------------
Summary: Cache Instances of CacheManager in DummyTxnManager
Key: HIVE-20844
URL: https://issues.apache.org/jira/browse/HIVE-20844
Project: Hive
Issue Type: Improvement
Components: HiveServer2, Locking
Affects Versions: 3.1.0, 2.3.2, 4.0.0
Reporter: BELUGA BEHR
I noticed that the {{DummyTxnManager}} class instantiates quite a few instances
of {{ZooKeeperHiveLockManager}}. The ZooKeeper LM creates a connection to ZK
for each instance created. It also does some initialization steps that are
almost always just noise and pressure on ZooKeeper because it has already been
initialized and the steps are therefore NOOPs. {{ZooKeeperHiveLockManager}}
should be a singleton class with one long-lived connection to the ZooKeeper
service. Perhaps the {{HiveLockManager}} interface could have a
{{isSingleton()}} method which indicates that the LM should only be
instantiated once and cached for subsequent sessions.
{code:java}
2018-05-14 22:45:30,574 INFO
org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
[HiveServer2-Background-Pool: Thread-1252389]: Creating lock manager of type
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 22:51:27,865 INFO
org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
[HiveServer2-Background-Pool: Thread-1252671]: Creating lock manager of type
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 22:51:37,552 INFO
org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
[HiveServer2-Background-Pool: Thread-1252686]: Creating lock manager of type
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 22:51:49,046 INFO
org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
[HiveServer2-Background-Pool: Thread-1252736]: Creating lock manager of type
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 22:51:50,664 INFO
org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
[HiveServer2-Background-Pool: Thread-1252742]: Creating lock manager of type
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 23:00:54,314 INFO
org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
[HiveServer2-Background-Pool: Thread-1253479]: Creating lock manager of type
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 23:17:26,867 INFO
org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
[HiveServer2-Background-Pool: Thread-1254180]: Creating lock manager of type
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 23:24:25,426 INFO
org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
[HiveServer2-Background-Pool: Thread-1255493]: Creating lock manager of type
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
{code}
{code:java|title=DummyTxnManager.java}
@Override
public HiveLockManager getLockManager() throws LockException {
if (lockMgr == null) {
boolean supportConcurrency =
conf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
if (supportConcurrency) {
String lockMgrName =
conf.getVar(HiveConf.ConfVars.HIVE_LOCK_MANAGER);
if ((lockMgrName == null) || (lockMgrName.isEmpty())) {
throw new LockException(ErrorMsg.LOCKMGR_NOT_SPECIFIED.getMsg());
}
try {
// CACHE LM HERE
LOG.info("Creating lock manager of type " + lockMgrName);
lockMgr = (HiveLockManager)ReflectionUtils.newInstance(
conf.getClassByName(lockMgrName), conf);
lockManagerCtx = new HiveLockManagerCtx(conf);
lockMgr.setContext(lockManagerCtx);
} catch (Exception e) {
...
{code}
[https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockManager.java]
{code:java|title=ZooKeeperHiveLockManager Initialization}
try {
curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
try{
curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" +
parent, new byte[0]);
} catch (Exception e) {
// ignore if the parent already exists
if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
KeeperException.Code.NODEEXISTS) {
LOG.warn("Unexpected ZK exception when creating parent node /" +
parent, e);
}
}
{code}
https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java#L96-L106
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
