[
https://issues.apache.org/jira/browse/HIVE-30231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30231:
----------------------------------
Component/s: Iceberg integration
Locking
> Revert the Iceberg part of HIVE-29244
> -------------------------------------
>
> Key: HIVE-30231
> URL: https://issues.apache.org/jira/browse/HIVE-30231
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration, Locking
> Reporter: Denys Kuzmenko
> Assignee: Neeraj Khatri
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> h3. Problem
>
>
> Since HIVE-29244, {{MetastoreLock}} puts the Iceberg catalog name on the
> HMS lock ({{LockComponent.catName}}) and on its {{ShowLocksRequest}}. But the
> Iceberg catalog name is not an HMS
> catalog. {{'iceberg.catalog'}} / {{iceberg.catalog.<name>.*}} is a
> connector-style alias for a set of catalog properties (type, uri, warehouse,
> io), like a Trino catalog. A HiveCatalog under
> any alias reads and writes its tables through two-argument HMS calls
> ({{HiveOperationsBase.java:89}}, {{HiveTableOperations.java:182}}), so always
> in {{metastore.catalog.default}}.
>
>
>
> HMS reports a lock conflict only between locks of the same catalog
> ({{CheckLockFunction}}). The Iceberg {{MetastoreLock}} of every other engine
> (Impala, Spark, Trino, Flink) sets no catalog,
> so its lock lands in the Thrift default {{hive}}. As a result, Hive's
> commit locks don't exclude the other engines:
>
>
>
>
> ||Table||Lives in HMS catalog||Hive's commit lock (HIVE-29244)||Other
> engines' locks||
>
> |no 'iceberg.catalog'|hive|default_iceberg (fallback name)|hive|
>
>
> |'iceberg.catalog'='foo'|hive|foo|hive|
>
>
>
>
>
> With {{engine.hive.lock-enabled=true}}, commits use the lock and no
> {{metadata_location}} compare-and-swap. So concurrent Hive and Impala/Spark
> commits can revert each other.
>
>
>
> Affects master since HIVE-29244. Releases without HIVE-29244 lock in
> {{hive}} and are not affected.
>
>
>
>
> h3. Multi-catalog status (context)
>
>
> * Hive's catalog SQL: CREATE/DROP/DESC CATALOG (HIVE-26227), ALTER CATALOG
> ... SET PROPERTIES (HIVE-29178), SET CATALOG (HIVE-29177). SET CATALOG sets
> the session's current catalog, which
> Hive uses to resolve table names.
>
>
> * The Iceberg integration uses none of it. No Iceberg code reads the
> current catalog, catalog properties are not Iceberg catalog properties, and
> HiveCatalog always works in
> {{metastore.catalog.default}}.
>
>
> * Consequence (from the code, not run): after {{SET CATALOG cat2}}, Hive
> resolves {{t}} as {{cat2.db.t}}, while Iceberg's HiveCatalog commits against
> {{hive.db.t}}. Iceberg tables in a
> non-default HMS catalog are not supported yet. Out of scope here.
>
>
>
>
>
> h3. Fix
>
>
> Revert the Iceberg part of HIVE-29244 ({{MetastoreLock}},
> {{TestHiveCommitLocks}}). The HMS lock and its SHOW LOCKS lookup then carry
> no catalog, as in upstream Iceberg's {{MetastoreLock}}.
> Hive's commit locks conflict with every other engine's in every deployment.
> The rest of HIVE-29244 (the catalog in {{ShowLocksRequest}} / SHOW LOCKS)
> stays.
>
>
>
> Locking in {{MetaStoreUtils.getDefaultCatalog(conf)}} instead was
> considered and rejected: with {{metastore.catalog.default}} other than
> {{hive}}, it would lock in a different catalog than
> the other engines. Catalog-aware Iceberg locking needs every engine to send
> the same catalog. It has to start in Iceberg upstream, together with
> HiveCatalog multi-catalog support.
>
>
>
> h3. Test
>
>
>
> {{TestHiveCommitLocks#testLockConflictsWithOtherEnginesWhateverTheCatalogName}}:
> a {{MetastoreLock}} of an Iceberg catalog named {{default_iceberg}} must
> make another engine's lock request
> (no catalog) on the same table wait. Before the fix the other lock is
> ACQUIRED.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
