[
https://issues.apache.org/jira/browse/HIVE-30229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138736#comment-18138736
]
Denys Kuzmenko edited comment on HIVE-30229 at 11/5/26, 12:31 PM:
------------------------------------------------------------------
Merged to master
Thanks for the review, @Dmitriy Fingerman !
was (Author: dkuzmenko):
Merged to master
Thanks for the review, @difin!
> Iceberg: Statistics updates and ALTER TABLE without the HMS lock can revert a
> concurrent commit
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-30229
> URL: https://issues.apache.org/jira/browse/HIVE-30229
> Project: Hive
> Issue Type: Bug
> Reporter: Denys Kuzmenko
> Assignee: Denys Kuzmenko
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> *Problem*
>
>
>
>
>
> HIVE-28851 made {{HiveIcebergMetaHook}} take the HMS lock only when
> {{engine.hive.lock-enabled=true}}, and never for queries
> ({{HiveOperation.QUERY}}); otherwise it uses {{NoLock}}. Before that,
> {{preAlterTable}} always held the lock from the {{metadata_location}} refresh
> (HIVE-26576) until the HMS {{alter_table}} call.
>
>
>
>
> With {{NoLock}}, {{preAlterTable}} still refreshes {{metadata_location}} to
> the current Iceberg metadata, but {{alter_table}} then replaces the table
> parameters unconditionally. If another
> writer commits to the Iceberg table between the refresh and the HMS call,
> {{metadata_location}} is set back to the refreshed value and that writer's
> snapshot is lost.
>
>
>
> Every INSERT with {{hive.stats.autogather=true}} goes through this window:
> the {{BasicStatsTask}} stats update is a plain {{alter_table}}, and INSERT
> always uses {{NoLock}}.
>
>
>
> *Reproduction*
>
>
> # Create an Iceberg v2 table and insert one row.
>
>
> # Hive: {{insert into t values ('hive', 1)}}; pause HS2 in the stats
> {{alter_table}}, after the {{metadata_location}} refresh (e.g. a breakpoint
> at the end of
> {{HiveIcebergMetaHook.assertNotCrossTableMetadataLocationChange}}).
>
>
> # Another client (Impala, or a second Hive session): {{insert into t values
> ('other', 1)}}; it succeeds and {{metadata_location}} moves from 00003 to
> 00004.
> # Resume HS2: {{metadata_location}} goes back to 00003. The other client's
> row and snapshot are lost.
>
>
>
>
> *Fix*
>
>
>
>
>
> Without a lock, send the refreshed location as the expected
> {{metadata_location}} ({{HiveOperationsBase.hmsEnvContext}}, the same as
> {{HiveTableOperations}} does for unlocked commits). HMS
> applies the alter only if {{metadata_location}} is unchanged, and otherwise
> rejects it ("The table has been modified").
>
>
>
>
> For the stats update, {{BasicStatsTask}} logs the rejection as a warning
> and the INSERT succeeds; only that stats update is skipped. With
> {{hive.stats.reliable=true}} the statement fails at the stats step, as for
> any other stats update failure.
>
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
