Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22231 )
Change subject: IMPALA-13620: Refresh compute_table_stats.py script ...................................................................... Patch Set 9: > Patch Set 9: > > Ok, I did a little bit of digging, and I think our theory isn't quite right. > There is a postgres connection limit set to 100, but we aren't hitting it. > HMS has a Hikari connection pool that it uses to connect to postgres and that > has a size limit. By default, the connection pool has a size limit of 10. > There are two connection pools for different things, so that's a max of 20 > connections to postgres. The Hikari connection pool has a connection timeout, > which is how long it will wait to get a connection from the pool. It defaults > to 30 seconds, and that is what we are hitting. The good news is that these > Hikari settings are configurable by modifying hive-site.xml. > > We can change the size of the connection pool by setting > datanucleus.connectionPool.maxPoolSize. I think we could try bumping this to > 20-30. See > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java#L695-L702 > > The 30 second connectionTimeout is also configurable by setting > hikaricp.connectionTimeout (units are milliseconds) (based on > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java#L43 > and > https://github.com/brettwooldridge/HikariCP?tab=readme-ov-file#frequently-used > ). I don't think there are major issues from doubling or tripling that. Thanks Joe! I will try adding those to fe/src/test/resources/hive-site.xml.py. -- To view, visit http://gerrit.cloudera.org:8080/22231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1ebf02f95b957e7dda3a30622b87e8fca3197699 Gerrit-Change-Number: 22231 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Tue, 07 Jan 2025 00:17:59 +0000 Gerrit-HasComments: No
