pvary commented on pull request #2180: URL: https://github.com/apache/iceberg/pull/2180#issuecomment-770687222
> @pvary, could you elaborate a bit more? I am not sure I got it. By my understanding the relevant change in #1875 is [this](https://github.com/apache/iceberg/commit/6731211eaf746c6a4abfe71386ef53172a3fc137#diff-e3082ef0c407c6d7644f453cd24a426eedc21fcdcb1fc501beeffb07b3070135L141-R138): ``` - HiveCatalog hiveCatalog = HiveCatalogs.loadCatalog(conf); - TableIdentifier tableIdentifier = TableIdentifier.parse(path.get()); - return hiveCatalog.loadTable(tableIdentifier); + return CustomCatalogs.table(lazySparkSession(), path.get()); ``` Am I right when assuming that this change replaces `HiveCatalogs.loadCatalog(conf)` with `CatalogUtil.buildIcebergCatalog(name, options, conf)`, which ultimately calls `CatalogUtil.loadCatalog(catalogImpl, name, options, conf)`? One important difference between the 2 methods is that `HiveCatalogs.loadCatalog(conf)` comes with a `CATALOG_CACHE`, so if you call loadCatalog with the same thrift uri then we reuse the previously created HiveCatalog instance. This is good as this reuses the HiveClientPool instance in that catalog, and also could be problematic since if there is any difference in the configuration then it will not be applied to the new pool. OTOH `CatalogUtil` always creates a new HiveCatalog instance and with it a HiveClientPool instance. My assumption was that the tests are creating multiple Catalog instances or they are not closing the Catalog instances correctly and this is the problem which causes the exhaustion of the Metastore thread pool. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
