Can you try use “hikaricp” connection pool manager? It seems to be using default which is no limit.
Eugene From: Igor Kuzmenko <f1she...@gmail.com> Reply-To: "user@hive.apache.org" <user@hive.apache.org> Date: Monday, March 20, 2017 at 2:17 PM To: "user@hive.apache.org" <user@hive.apache.org> Subject: Re: Hive TxnHandler::lock method run into dead lock. Sorry miss clicked. 2) TxnHandler::lock method request new connection when executing this line of code: ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst); 3) After that folowing this stacktrace: - TxnHandler::lock - TxnHandler::checkLockWithRetry - TxnHandler::checkLock In checkLock method we reach this line: handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name()); 4) acquireLock method requests another connection to DB: dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED); So all in all if i call TxnHandler::lock method in 10 threads same time at first I get all connections to db, that stored in pool, and at acquireLock I will stuck because there's no free connection. Does anyone run into this problem? How can I avoid this problem? Code was taken from here: https://github.com/hortonworks/hive-release/blob/HDP-2.5.0.0-tag/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java I guess the closest branch in apach repo is: https://github.com/apache/hive/blob/branch-2.1/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java On Tue, Mar 21, 2017 at 12:07 AM, Igor Kuzmenko <f1she...@gmail.com<mailto:f1she...@gmail.com>> wrote: Hello I'm running Hortonworks data platform 2.5.0.0 with included hive. I'm using storm hive bolt to load data into my hive. But launching many hive bolt always leads me to TimeoutException on calling hive metastore. Metastore logs full of Exception like this: 2017-03-15 18:46:12,436 ERROR [pool-5-thread-11]: txn.TxnHandler (TxnHandler.java:getDbConn(1834)) - There is a problem with a connection from the pool, retrying(rc=7): Timed out waiting for a free available connection. (SQLState=08001, ErrorCode=0) java.sql.SQLException: Timed out waiting for a free available connection. at com.jolbox.bonecp.DefaultConnectionStrategy.getConnectionInternal(DefaultConnectionStrategy.java:88) at com.jolbox.bonecp.AbstractConnectionStrategy.getConnection(AbstractConnectionStrategy.java:90) at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:553) at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:131) at org.apache.hadoop.hive.metastore.txn.TxnHandler.getDbConn(TxnHandler.java:1827) at org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:873) at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:814) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:5751) After looking through code I found out 1) TxnHandler class uses connection pool to get db connections and it's size is 10. 2) TxnHandler::lock method requset new connection whe executing this line of code:
