[
https://issues.apache.org/jira/browse/IMPALA-14637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095132#comment-18095132
]
ASF subversion and git services commented on IMPALA-14637:
----------------------------------------------------------
Commit e0c3291c1f1734121aaf88edf450268cc2f85ecf in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=e0c3291c1 ]
IMPALA-14637: COMMIT_TXN events should trigger reload for truncate ops
Truncate operations generate ALTER events in HMS. These events trigger
metadata reloads when catalogd processes them. For a transactional
table, the ALTER event could be processed before the transaction is
committed. Then a stale snapshot is loaded. Catalogd should reload the
metadata in processing the corresponding COMMIT_TXN event. However, that
doesn't happen for truncate operations. When processing COMMIT_TXN
events, catalogd fetches the WriteEventInfo list for the transaction.
This doesn't include the truncate operations (HIVE-29677), which causes
COMMIT_TXN events skip the reloads.
This patch fixes the issue by tracking transactional truncate operations
when receiving ALTER_TABLE, ALTER_PARTITION and ALTER_PARTITIONS events.
A map from TableWriteId (db, tbl, writeId) to the truncated partition
list is maintained for this. When processing ABORT_TXN events, the
entries in this map will be cleared and no updates happen.
A new class, TableWriteEvent, is added to represent WriteEventInfo
returned from HMS and the truncation info. When processing a COMMIT_TXN
event, after fetching the WriteEventInfo list, we convert it into a list
of TableWriteEvent and then add all the truncation items of that
transaction. Reloads are triggered based on this list and ValidWriteIds
list of the table is updated accordingly.
Tests
- Added FE tests for ALTER_TABLE and ALTER_PARTITION events.
- Due to the dependent Hive version is missing HIVE-28668, HMS can't
generate a single ALTER_PARTITIONS event when truncating a
partitioned table. So tests for ALTER_PARTITIONS events are missing.
Assisted-by: Opus 4.7 (Claude Code)
Change-Id: I89aac12819f08dd9ed42d5d8b21a96c04b04d75c
Reviewed-on: http://gerrit.cloudera.org:8080/23805
Reviewed-by: Zoltan Borok-Nagy <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> test_event_based_replication is flaky for truncate table
> --------------------------------------------------------
>
> Key: IMPALA-14637
> URL: https://issues.apache.org/jira/browse/IMPALA-14637
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Attachments:
> catalogd.0beeab027c72.impala.log.INFO.20251218-020359.1.gz,
> hadoop-hdfs.tar.gz, hive-metastore.log.gz, hive-server2.log.gz,
> impalad.52d7008a6fb3.impala.log.INFO.20251218-020359.1_part.gz
>
>
> Saw this test failed again like IMPALA-12187:
> https://jenkins.impala.io/job/ubuntu-20.04-dockerised-tests/4377
> {code:python}
> metadata/test_event_processing.py:122: in test_event_based_replication
> self._run_event_based_replication_tests_impl(self,
> metadata/test_event_processing_base.py:305: in
> _run_event_based_replication_tests_impl
> assert rows_in_part_tbl_target == 0
> E assert 100 == 0{code}
> But the implementation is much different now that we have both the
> hms_event_sync feature and hierarchical event processing enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
