szehon-ho opened a new pull request, #7441: URL: https://github.com/apache/iceberg/pull/7441
As the term Hive-Catalog is overloaded across, I'll use "Iceberg-HiveCatalog" to refer to Iceberg's HiveCatalog class and "HMS Catalog" to refer to Hive specific concept. https://issues.apache.org/jira/browse/HIVE-18685 added support for HMS catalogs, ie different namespaces for dbs/tables in the metastore. However, the Iceberg-HiveCatalog uses a global cache of HMS connections based on Hive-Metastore-URI. This prevents different instances Iceberg-HiveCatalogs existing in same JVM from talking to different HMS Catalogs. See issue: https://github.com/apache/iceberg/pull/5378. This is important in Spark, where user may try to configure different SparkCatalog pointing to separate catalog on same HMS. This change takes advantage of https://github.com/apache/iceberg/pull/6698 to fix this. It does two things: - Add a 'hive-catalog' flag on Iceberg-HiveCatalog (similar to "uri" and "warehouse"), which automatically sets this on HMS client to point to right HMS Catalog. - Adds the HMS client config "metastore.catalog.default" to the list of cache keys. From Spark point of view, user can configure 'spark.sql.catalog.$cat1.hive-catalog=foo', spark.sql.catalog.$cat2.hive-catalog=bar'. Note; @RussellSpitzer pointed to me that setting spark.sql.catalog.$myCatalog.hadoop.metastore.catalog.default=foo seems to work to configure each SparkCatalog. But it is a bit clunky and undocumented, and hence the proposal to add 'hive_catalog' shorthand. In either case, the second change is needed to allow different cached connections on same HMS to go to different HMS-catalogs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
