[
https://issues.apache.org/jira/browse/HIVE-26794?focusedWorklogId=831680&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-831680
]
ASF GitHub Bot logged work on HIVE-26794:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 07/Dec/22 08:18
Start Date: 07/Dec/22 08:18
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on PR #3817:
URL: https://github.com/apache/hive/pull/3817#issuecomment-1340567809
Change the `TxnHandler#connPoolMutex` from a fixed size connection pool to a
dynamic size(2 ~ maxPoolSize) pool, so that HMS won't suffer from starvation of
connection or establish numerous connections to underlying database, Tested
HikariCP, the pool can grow or shrink on demand:
> 2022-12-07T06:45:55,589 DEBUG [mutex housekeeper] pool.HikariPool: mutex -
After cleanup stats (total=3, active=1, idle=2, waiting=0)
2022-12-07T06:46:25,587 DEBUG [mutex housekeeper] pool.HikariPool: mutex -
Before cleanup stats (total=10, active=10, idle=0, waiting=294)
2022-12-07T06:46:25,587 DEBUG [mutex housekeeper] pool.HikariPool: mutex -
After cleanup stats (total=10, active=10, idle=0, waiting=294)
2022-12-07T06:46:55,588 DEBUG [mutex housekeeper] pool.HikariPool: mutex -
Before cleanup stats (total=7, active=1, idle=6, waiting=0)
2022-12-07T06:46:55,588 DEBUG [mutex housekeeper] pool.HikariPool: mutex -
After cleanup stats (total=7, active=1, idle=6, waiting=0)
2022-12-07T06:47:25,591 DEBUG [mutex housekeeper] pool.HikariPool: mutex -
Before cleanup stats (total=3, active=2, idle=1, waiting=0)
2022-12-07T06:47:25,591 DEBUG [mutex housekeeper] pool.HikariPool: mutex -
After cleanup stats (total=3, active=2, idle=1, waiting=0)
2022-12-07T06:47:55,591 DEBUG [mutex housekeeper] pool.HikariPool: mutex -
Before cleanup stats (total=4, active=1, idle=3, waiting=0)
2022-12-07T06:47:55,592 DEBUG [mutex housekeeper] pool.HikariPool: mutex -
After cleanup stats (total=4, active=1, idle=3, waiting=0)
2022-12-07T06:48:25,592 DEBUG [mutex housekeeper] pool.HikariPool: mutex -
Before cleanup stats (total=4, active=1, idle=3, waiting=0)
2022-12-07T06:48:25,592 DEBUG [mutex housekeeper] pool.HikariPool: mutex -
After cleanup stats (total=4, active=1, idle=3, waiting=0)
2022-12-07T06:48:55,593 DEBUG [mutex housekeeper] pool.HikariPool: mutex -
Before cleanup stats (total=3, active=1, idle=2, waiting=0)
The DBCP has a little bit different from `HikariCP`, though I have set
`maxIdle` and `timeBetweenEvictionRunsMillis`
```java
objectPool.setMaxIdle(Math.min(maxPoolSize, 2));
objectPool.setTimeBetweenEvictionRunsMillis(300000);
```
but don't see a drop down of connections when HMS is free.
Issue Time Tracking
-------------------
Worklog Id: (was: 831680)
Time Spent: 2h (was: 1h 50m)
> Explore changing TxnHandler#connPoolMutex to NoPoolConnectionPool
> -----------------------------------------------------------------
>
> Key: HIVE-26794
> URL: https://issues.apache.org/jira/browse/HIVE-26794
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
> Reporter: Zhihua Deng
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Instead of creating a fixed size connection pool for TxnHandler#MutexAPI, the
> pool can be assigned to NoPoolConnectionPool due to:
> * TxnHandler#MutexAPI is primarily designed to provide coarse-grained mutex
> support to maintenance tasks running inside the Metastore, these tasks are
> not user faced;
> * A fixed size connection pool as same as the pool used in ObjectStore is a
> waste for other non leaders in the warehouse;
> The NoPoolConnectionPool provides connection on demand, and
> TxnHandler#MutexAPI only uses getConnection method to fetch a connection from
> the pool, so it's doable to change the pool to NoPoolConnectionPool, this
> would make the HMS more scaleable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
