[
https://issues.apache.org/jira/browse/HIVE-30231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30231:
----------------------------------
Description:
h3. Problem
Since HIVE-29244, {{MetastoreLock}} puts the Iceberg catalog name on the HMS
lock ({{LockComponent.catName}}) and on its {{ShowLocksRequest}}. But the
Iceberg catalog name is not an HMS
catalog. {{'iceberg.catalog'}} / {{iceberg.catalog.<name>.*}} is a
connector-style alias for a set of catalog properties (type, uri, warehouse,
io), like a Trino catalog. A HiveCatalog under
any alias reads and writes its tables through two-argument HMS calls
({{HiveOperationsBase.java:89}}, {{HiveTableOperations.java:182}}), so always
in {{metastore.catalog.default}}.
HMS reports a lock conflict only between locks of the same catalog
({{CheckLockFunction}}). The Iceberg {{MetastoreLock}} of every other engine
(Impala, Spark, Trino, Flink) sets no catalog,
so its lock lands in the Thrift default {{hive}}. As a result, Hive's commit
locks don't exclude the other engines:
||Table||Lives in HMS catalog||Hive's commit lock (HIVE-29244)||Other
engines' locks||
|no 'iceberg.catalog'|hive|default_iceberg (fallback name)|hive|
|'iceberg.catalog'='foo'|hive|foo|hive|
With {{engine.hive.lock-enabled=true}}, commits use the lock and no
{{metadata_location}} compare-and-swap. So concurrent Hive and Impala/Spark
commits can revert each other.
Affects master since HIVE-29244. Releases without HIVE-29244 lock in {{hive}}
and are not affected.
h3. Multi-catalog status (context)
* Hive's catalog SQL: CREATE/DROP/DESC CATALOG (HIVE-26227), ALTER CATALOG
... SET PROPERTIES (HIVE-29178), SET CATALOG (HIVE-29177). SET CATALOG sets the
session's current catalog, which
Hive uses to resolve table names.
* The Iceberg integration uses none of it. No Iceberg code reads the current
catalog, catalog properties are not Iceberg catalog properties, and HiveCatalog
always works in
{{metastore.catalog.default}}.
* Consequence (from the code, not run): after {{SET CATALOG cat2}}, Hive
resolves {{t}} as {{cat2.db.t}}, while Iceberg's HiveCatalog commits against
{{hive.db.t}}. Iceberg tables in a
non-default HMS catalog are not supported yet. Out of scope here.
h3. Fix
Revert the Iceberg part of HIVE-29244 ({{MetastoreLock}},
{{TestHiveCommitLocks}}). The HMS lock and its SHOW LOCKS lookup then carry no
catalog, as in upstream Iceberg's {{MetastoreLock}}.
Hive's commit locks conflict with every other engine's in every deployment.
The rest of HIVE-29244 (the catalog in {{ShowLocksRequest}} / SHOW LOCKS)
stays.
Locking in {{MetaStoreUtils.getDefaultCatalog(conf)}} instead was considered
and rejected: with {{metastore.catalog.default}} other than {{hive}}, it would
lock in a different catalog than
the other engines. Catalog-aware Iceberg locking needs every engine to send
the same catalog. It has to start in Iceberg upstream, together with
HiveCatalog multi-catalog support.
h3. Test
{{TestHiveCommitLocks#testLockConflictsWithOtherEnginesWhateverTheCatalogName}}:
a {{MetastoreLock}} of an Iceberg catalog named {{default_iceberg}} must make
another engine's lock request
(no catalog) on the same table wait. Before the fix the other lock is
ACQUIRED.
was:
Since HIVE-29244, MetastoreLock puts the Iceberg catalog name on the HMS lock.
Hive names the HiveCatalog of a table without 'iceberg.catalog'
default_iceberg, while the table lives in theHMS default catalog (hive). So
Hive's commit locks don't conflict with other writers of the table: Impala,
Spark, and Hive's own external locks.
Fix: that HiveCatalog takes the name of the HMS default catalog
(metastore.catalog.default).
> Revert the Iceberg part of HIVE-29244
> -------------------------------------
>
> Key: HIVE-30231
> URL: https://issues.apache.org/jira/browse/HIVE-30231
> Project: Hive
> Issue Type: Bug
> Reporter: Denys Kuzmenko
> Priority: Major
>
> h3. Problem
>
>
> Since HIVE-29244, {{MetastoreLock}} puts the Iceberg catalog name on the
> HMS lock ({{LockComponent.catName}}) and on its {{ShowLocksRequest}}. But the
> Iceberg catalog name is not an HMS
> catalog. {{'iceberg.catalog'}} / {{iceberg.catalog.<name>.*}} is a
> connector-style alias for a set of catalog properties (type, uri, warehouse,
> io), like a Trino catalog. A HiveCatalog under
> any alias reads and writes its tables through two-argument HMS calls
> ({{HiveOperationsBase.java:89}}, {{HiveTableOperations.java:182}}), so always
> in {{metastore.catalog.default}}.
>
>
>
> HMS reports a lock conflict only between locks of the same catalog
> ({{CheckLockFunction}}). The Iceberg {{MetastoreLock}} of every other engine
> (Impala, Spark, Trino, Flink) sets no catalog,
> so its lock lands in the Thrift default {{hive}}. As a result, Hive's
> commit locks don't exclude the other engines:
>
>
>
>
> ||Table||Lives in HMS catalog||Hive's commit lock (HIVE-29244)||Other
> engines' locks||
>
> |no 'iceberg.catalog'|hive|default_iceberg (fallback name)|hive|
>
>
> |'iceberg.catalog'='foo'|hive|foo|hive|
>
>
>
>
>
> With {{engine.hive.lock-enabled=true}}, commits use the lock and no
> {{metadata_location}} compare-and-swap. So concurrent Hive and Impala/Spark
> commits can revert each other.
>
>
>
> Affects master since HIVE-29244. Releases without HIVE-29244 lock in
> {{hive}} and are not affected.
>
>
>
>
> h3. Multi-catalog status (context)
>
>
> * Hive's catalog SQL: CREATE/DROP/DESC CATALOG (HIVE-26227), ALTER CATALOG
> ... SET PROPERTIES (HIVE-29178), SET CATALOG (HIVE-29177). SET CATALOG sets
> the session's current catalog, which
> Hive uses to resolve table names.
>
>
> * The Iceberg integration uses none of it. No Iceberg code reads the
> current catalog, catalog properties are not Iceberg catalog properties, and
> HiveCatalog always works in
> {{metastore.catalog.default}}.
>
>
> * Consequence (from the code, not run): after {{SET CATALOG cat2}}, Hive
> resolves {{t}} as {{cat2.db.t}}, while Iceberg's HiveCatalog commits against
> {{hive.db.t}}. Iceberg tables in a
> non-default HMS catalog are not supported yet. Out of scope here.
>
>
>
>
>
> h3. Fix
>
>
> Revert the Iceberg part of HIVE-29244 ({{MetastoreLock}},
> {{TestHiveCommitLocks}}). The HMS lock and its SHOW LOCKS lookup then carry
> no catalog, as in upstream Iceberg's {{MetastoreLock}}.
> Hive's commit locks conflict with every other engine's in every deployment.
> The rest of HIVE-29244 (the catalog in {{ShowLocksRequest}} / SHOW LOCKS)
> stays.
>
>
>
> Locking in {{MetaStoreUtils.getDefaultCatalog(conf)}} instead was
> considered and rejected: with {{metastore.catalog.default}} other than
> {{hive}}, it would lock in a different catalog than
> the other engines. Catalog-aware Iceberg locking needs every engine to send
> the same catalog. It has to start in Iceberg upstream, together with
> HiveCatalog multi-catalog support.
>
>
>
> h3. Test
>
>
>
> {{TestHiveCommitLocks#testLockConflictsWithOtherEnginesWhateverTheCatalogName}}:
> a {{MetastoreLock}} of an Iceberg catalog named {{default_iceberg}} must
> make another engine's lock request
> (no catalog) on the same table wait. Before the fix the other lock is
> ACQUIRED.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
