mszurap opened a new pull request, #5764: URL: https://github.com/apache/hive/pull/5764
### What changes were proposed in this pull request? The secondary connection factory is used for schema generation, and for value generation operations. Under normal circumstances 2 connections should be sufficient, and currently it is hardcoded to that value. We have found in [HIVE-28839](https://issues.apache.org/jira/browse/HIVE-28839) that connection starvation may still happen. (in that case due to a datanucleus bug) We should make the "objectstore-secondary" pool size configurable. This is needed at least until a final fix is not available for [HIVE-28839](https://issues.apache.org/jira/browse/HIVE-28839), but probably it is good to have on the long run if we hit other edge cases. ### Why are the changes needed? The "objectstore-secondary" pool size is currently hardcoded to 2. The "datanucleus.connectionPool.maxPoolSize" controls the "objectstore" poolsize, I propose we introduce a new configuration as "datanucleus.connectionPool.maxSecondaryPoolSize" which is not an existing Datanucleus config, however we could still use that only in Hive to control the number of connections in the "objectstore-secondary" pool. If that config name is not desired, we can use a different name like we have for "metastore.compactor.connectionPool.maxPoolSize". ### Does this PR introduce _any_ user-facing change? No ### Is the change a dependency upgrade? No ### How was this patch tested? As I see there are no automated tests for the pool handling, so I made a manual test: - Set the "datanucleus.connectionPool.maxSecondaryPoolSize" new config item in the HMS hive-site.xml, and observed the new size is now is logged out: `2025-04-11T13:09:25,881 INFO [main] datasource.HikariCPDataSourceProvider: Creating Hikari connection pool for the MetaStore, maxPoolSize: 4, name: objectstore-secondary` - Observed that the number of database connections increased on my MySQL side ("show processlist" output) - During testing found from the logs that with this change the "objectstore-compactor-secondary" pool size is also increased from 2 to the new size. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
