Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/22231 )
Change subject: IMPALA-13620: Refresh compute_table_stats.py script ...................................................................... Patch Set 9: Ok, I did a little bit of digging, and I think our theory isn't quite right. There is a postgres connection limit set to 100, but we aren't hitting it. HMS has a Hikari connection pool that it uses to connect to postgres and that has a size limit. By default, the connection pool has a size limit of 10. There are two connection pools for different things, so that's a max of 20 connections to postgres. The Hikari connection pool has a connection timeout, which is how long it will wait to get a connection from the pool. It defaults to 30 seconds, and that is what we are hitting. The good news is that these Hikari settings are configurable by modifying hive-site.xml. We can change the size of the connection pool by setting datanucleus.connectionPool.maxPoolSize. I think we could try bumping this to 20-30. See https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java#L695-L702 The 30 second connectionTimeout is also configurable by setting hikaricp.connectionTimeout (units are milliseconds) (based on https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java#L43 and https://github.com/brettwooldridge/HikariCP?tab=readme-ov-file#frequently-used ). I don't think there are major issues from doubling or tripling that. -- To view, visit http://gerrit.cloudera.org:8080/22231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1ebf02f95b957e7dda3a30622b87e8fca3197699 Gerrit-Change-Number: 22231 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Mon, 06 Jan 2025 23:35:41 +0000 Gerrit-HasComments: No
