Prasanth Jayachandran created HIVE-18348:
--------------------------------------------
Summary: Hive creates 4 different connection pools to metastore of
different size
Key: HIVE-18348
URL: https://issues.apache.org/jira/browse/HIVE-18348
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran
Enabling debug logging with HikariCP, I can see that Hive creates 4 connection
pools. {code:title=first connection pool creation stack trace} "main@1" prio=5
tid=0x1 nid=NA runnable java.lang.Thread.State: RUNNABLE at
com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:73) at
org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
at
org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
at
org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
- locked <0x102b> (a org.datanucleus.store.rdbms.ConnectionFactoryImpl) at
org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
at
org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:330)
at
org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:203)
at
org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:162)
at
org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:285)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
at
org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
at
org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
at
org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
- locked <0x1035> (a org.datanucleus.PersistenceNucleusContextImpl) at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
- locked <0x1036> (a org.datanucleus.api.jdo.JDOPersistenceManagerFactory) at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
- locked <0xeb8> (a java.lang.Class) at
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
javax.jdo.JDOHelper$16.run(JDOHelper.java:1965) at
java.security.AccessController.doPrivileged(AccessController.java:-1) at
javax.jdo.JDOHelper.invoke(JDOHelper.java:1960) at
javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) at
javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) at
org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:619) -
locked <0x957> (a java.lang.Class) at
org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:662)
at
org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:452)
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:389)
at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:344)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) at
org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59) at
org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:668)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:647)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:641)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:695)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:491)
- locked <0xe59> (a java.lang.Class) at
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7518)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:161)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:79)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4100)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4152) - locked
<0xe92> (a org.apache.hadoop.hive.ql.metadata.Hive) at
org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4132) at
org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4394) at
org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:267) at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:250)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:407) at
org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:351) at
org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:331) at
org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:307) at
org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:942)
at
org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:906)
at
org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1856)
at
org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:131)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:115) - locked
<0xe94> (a org.apache.hive.service.cli.CLIService) at
org.apache.hive.service.CompositeService.init(CompositeService.java:59) -
locked <0xe95> (a org.apache.hive.service.server.HiveServer2) at
org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:167) at
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:790)
at org.apache.hive.service.server.HiveServer2.access$900(HiveServer2.java:110)
at
org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1034)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:903) at
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.util.RunJar.run(RunJar.java:234) at
org.apache.hadoop.util.RunJar.main(RunJar.java:148) {code} {code:title=second
connection pool creation stack trace} "main@1" prio=5 tid=0x1 nid=NA runnable
java.lang.Thread.State: RUNNABLE at
com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:73) at
org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
at
org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
at
org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:134)
- locked <0x105e> (a org.datanucleus.store.rdbms.ConnectionFactoryImpl) at
org.datanucleus.store.rdbms.ConnectionFactoryImpl.createManagedConnection(ConnectionFactoryImpl.java:249)
at
org.datanucleus.store.connection.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:302)
at
org.datanucleus.store.connection.AbstractConnectionFactory.getConnection(AbstractConnectionFactory.java:84)
at
org.datanucleus.store.AbstractStoreManager.getConnection(AbstractStoreManager.java:378)
at
org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:296)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
at
org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
at
org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
at
org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
- locked <0x1035> (a org.datanucleus.PersistenceNucleusContextImpl) at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
- locked <0x1036> (a org.datanucleus.api.jdo.JDOPersistenceManagerFactory) at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
- locked <0xeb8> (a java.lang.Class) at
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
javax.jdo.JDOHelper$16.run(JDOHelper.java:1965) at
java.security.AccessController.doPrivileged(AccessController.java:-1) at
javax.jdo.JDOHelper.invoke(JDOHelper.java:1960) at
javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) at
javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) at
org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:619) -
locked <0x957> (a java.lang.Class) at
org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:662)
at
org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:452)
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:389)
at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:344)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) at
org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59) at
org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:668)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:647)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:641)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:695)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:491)
- locked <0xe59> (a java.lang.Class) at
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7518)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:161)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:79)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4100)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4152) - locked
<0xe92> (a org.apache.hadoop.hive.ql.metadata.Hive) at
org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4132) at
org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4394) at
org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:267) at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:250)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:407) at
org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:351) at
org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:331) at
org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:307) at
org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:942)
at
org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:906)
at
org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1856)
at
org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:131)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:115) - locked
<0xe94> (a org.apache.hive.service.cli.CLIService) at
org.apache.hive.service.CompositeService.init(CompositeService.java:59) -
locked <0xe95> (a org.apache.hive.service.server.HiveServer2) at
org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:167) at
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:790)
at org.apache.hive.service.server.HiveServer2.access$900(HiveServer2.java:110)
at
org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1034)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:903) at
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.util.RunJar.run(RunJar.java:234) at
org.apache.hadoop.util.RunJar.main(RunJar.java:148)
{code}
2 other instances of connection pool are explicitly created (connPool and
connPoolMutex) in TxnHandler
https://github.com/apache/hive/blob/035eca39fc0bd7aa0d9c1809a26e000ac52978d0/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L256-L264
If datanucleus.connectionPool.maxPoolSize is set to 10, then hive will create
47 (connPoolMutex creates 7 more than maxPoolSize) connections per HS2 and HMS
instance. This blows up the number of connections on DB side. Ideally, we want
a single shared connection pool for ObjectStore and TxnHandler so that it is
easily tuneable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
