I increased maxConnections up to 50 and recompiled metastore jar. Its gonna
be enough for a while.
Eugene, do you know is there any Jira item on this problem?
On Sun, Mar 26, 2017 at 7:21 PM, Eugene Koifman <ekoif...@hortonworks.com>
wrote:
> I see (in HikariCP-2.5.1.jar) so perhaps upgrading the library is an
> option.
>
>
>
> *public *HikariConfig() {
> *this*.dataSourceProperties = *new *Properties();
> *this*.healthCheckProperties = *new *Properties();
> *this*.minIdle = -1;
> *this*.maxPoolSize = -1;
> *this*.maxLifetime = MAX_LIFETIME;
> *this*.connectionTimeout = CONNECTION_TIMEOUT;
> *this*.validationTimeout = VALIDATION_TIMEOUT;
> *this*.idleTimeout = IDLE_TIMEOUT;
> *this*.isAutoCommit = *true*;
> *this*.isInitializationFailFast = *true*;
> String systemProp = System.getProperty(*"hikaricp.configurationFile"*);
> *if*(systemProp != *null*) {
> *this*.loadProperties(systemProp);
> }
>
> }
>
>
>
>
>
> *From: *Igor Kuzmenko <f1she...@gmail.com>
> *Reply-To: *"user@hive.apache.org" <user@hive.apache.org>
> *Date: *Saturday, March 25, 2017 at 5:05 PM
>
> *To: *"user@hive.apache.org" <user@hive.apache.org>
> *Subject: *Re: Hive TxnHandler::lock method run into dead lock.
>
>
>
> Hi, Eugene.
>
> I've tried hicaricp, it didn't work either. Hikari pool has same 10
> maxConnections limit. When creating pool there's no explicit set of max
> pool size and in HikariConfig constructor it's hardcoded to value of 10:
>
> public HikariConfig()
> {
> dataSourceProperties = new Properties();
>
> connectionTimeout = *CONNECTION_TIMEOUT*;
> idleTimeout = *IDLE_TIMEOUT*;
> isAutoCommit = true;
> isJdbc4connectionTest = true;
> minIdle = -1;
> maxPoolSize = 10;
> maxLifetime = *MAX_LIFETIME*;
> isRecordMetrics = false;
> transactionIsolation = -1;
> metricsTrackerClassName =
> "com.zaxxer.hikari.metrics.CodaHaleMetricsTracker";
> customizer = new IConnectionCustomizer() {
> @Override
> public void customize(Connection connection) throws SQLException
> {
> }
> };
> }
>
>
>
> On Thu, Mar 23, 2017 at 8:29 PM, Eugene Koifman <ekoif...@hortonworks.com>
> wrote:
>
> Can you try use “hikaricp” connection pool manager? It seems to be using
> default which is no limit.
>
>
>
>
>
> Eugene
>
>
>
> *From: *Igor Kuzmenko <f1she...@gmail.com>
> *Reply-To: *"user@hive.apache.org" <user@hive.apache.org>
> *Date: *Monday, March 20, 2017 at 2:17 PM
> *To: *"user@hive.apache.org" <user@hive.apache.org>
> *Subject: *Re: Hive TxnHandler::lock method run into dead lock.
>
>
>
> Sorry miss clicked.
>
>
>
> 2) TxnHandler::lock method request new connection when executing this line
> of code:
>
> ConnectionLockIdPair
>
> connAndLockId =
>
> enqueueLockWithRetry(rqst);
>
>
>
> 3) After that
>
> folowing this
>
> stacktrace:
>
> - TxnHandler::lock
>
> -
>
> TxnHandler::checkLockWithRetry
>
> -
>
> TxnHandler::checkLock
>
>
>
> In
>
> checkLock method we reach this line:
>
> handle
>
> =
>
> getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
>
>
>
> 4)
>
> acquireLock method requests another connection to DB:
>
> dbConn
>
> =
>
> getDbConn(Connection.TRANSACTION_READ_COMMITTED);
>
>
>
> So all in all if i call
>
> TxnHandler::lock
>
> method in 10 threads same time at first I get all connections to db, that
> stored in pool, and at
>
> acquireLock I will stuck because there's no free connection.
>
>
>
> Does anyone run into this problem? How can I avoid this problem?
>
>
>
> Code was taken from here:
>
> https://github.com/hortonworks/hive-release/blob/
> HDP-2.5.0.0-tag/metastore/src/java/org/apache/hadoop/hive/
> metastore/txn/TxnHandler.java
>
>
>
> I guess the closest branch in
>
> apach
>
> repo is:
>
> https://github.com/apache/hive/blob/branch-2.1/
> metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>
>
>
> On Tue, Mar 21, 2017 at 12:07 AM, Igor Kuzmenko <f1she...@gmail.com>
> wrote:
>
> Hello I'm running Hortonworks data platform 2.5.0.0 with included hive.
>
> I'm using storm hive bolt to load data into my hive. But launching many
> hive bolt always leads me to TimeoutException on calling hive metastore.
> Metastore logs full of Exception like this:
>
>
>
> 2017-03-15 18:46:12,436 ERROR [pool-5-thread-11]: txn.TxnHandler
> (TxnHandler.java:getDbConn(1834)) - There is a problem with a connection
> from the pool, retrying(rc=7): Timed out waiting for a free available
> connection. (SQLState=08001, ErrorCode=0)
> java.sql.SQLException: Timed out waiting for a free available connection.
> at com.jolbox.bonecp.DefaultConnectionStrategy.getConnectionInternal(
> DefaultConnectionStrategy.java:88)
> at com.jolbox.bonecp.AbstractConnectionStrategy.getConnection(
> AbstractConnectionStrategy.java:90)
> at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:553)
> at com.jolbox.bonecp.BoneCPDataSource.getConnection(
> BoneCPDataSource.java:131)
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.
> getDbConn(TxnHandler.java:1827)
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(
> TxnHandler.java:873)
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(
> TxnHandler.java:814)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$
> HMSHandler.lock(HiveMetaStore.java:5751)
>
>
>
> After looking through code I found out
>
>
>
> 1) TxnHandler class uses connection pool to get db connections and it's
> size is 10.
>
> 2) TxnHandler::lock method requset new connection whe executing this line
> of code:
>
>
>
>
>
>
>
