[
https://issues.apache.org/jira/browse/IMPALA-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith updated IMPALA-12768:
-----------------------------------
Component/s: Catalog
(was: Backend)
> Set iceberg.engine.hive.lock-enabled To false By Default
> ---------------------------------------------------------
>
> Key: IMPALA-12768
> URL: https://issues.apache.org/jira/browse/IMPALA-12768
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Jason Fehr
> Priority: Major
> Labels: backend, iceberg, impala
>
> Iceberg tables are currently prone to leaving a permanent lock in HMS if
> Impala shuts down while the lock is held. This was observed intermittently
> during {{sys.impala_query_log}} testing and noted in
> https://github.com/apache/iceberg/issues/2301. It's a short lock, so it's
> fairly rare, but could happen and the only recourse is to delete rows from
> HIVE_LOCKS in HMS's database.
> https://github.com/apache/iceberg/pull/6570 introduced an alternative update
> mechanism in Iceberg 1.3 that depends on HIVE-26682. [Iceberg
> documentation|https://iceberg.apache.org/docs/latest/configuration/#hadoop-configuration]
> says to set the property "iceberg.engine.hive.lock-enabled
> {noformat}
> Warn: Setting iceberg.engine.hive.lock-enabled=false will cause HiveCatalog
> to commit to tables without using Hive locks. This should only be set to
> false if all following conditions are met:
> * HIVE-26882 is available on the Hive Metastore server
> * All other HiveCatalogs committing to tables that this HiveCatalog commits
> to are also on Iceberg 1.3 or later
> * All other HiveCatalogs committing to tables that this HiveCatalog commits
> to have also disabled Hive locks on commit.
> Failing to ensure these conditions risks corrupting the table.
> {noformat}
> Setting this property to false uses a new Hive atomic operation and avoids
> taking a lock, so it can't get stuck if Impala shuts down at the wrong time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
