Hello [email protected], Zoltan Borok-Nagy, Sai Hemanth Gantasala,
Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23805
to look at the new patch set (#11).
Change subject: IMPALA-14637: COMMIT_TXN events should trigger reload for
truncate ops
......................................................................
IMPALA-14637: COMMIT_TXN events should trigger reload for truncate ops
Truncate operations generate ALTER events in HMS. These events trigger
metadata reloads when catalogd processes them. For a transactional
table, the ALTER event could be processed before the transaction is
committed. Then a stale snapshot is loaded. Catalogd should reload the
metadata in processing the corresponding COMMIT_TXN event. However, that
doesn't happen for truncate operations. When processing COMMIT_TXN
events, catalogd fetches the WriteEventInfo list for the transaction.
This doesn't include the truncate operations (HIVE-29677), which causes
COMMIT_TXN events skip the reloads.
This patch fixes the issue by tracking transactional truncate operations
when receiving ALTER_TABLE, ALTER_PARTITION and ALTER_PARTITIONS events.
A map from TableWriteId (db, tbl, writeId) to the truncated partition
list is maintained for this.
A new class, TableWriteEvent, is added to represent WriteEventInfo
returned from HMS and the truncation info. When processing a COMMIT_TXN
event, after fetching the WriteEventInfo list, we convert it into a list
of TableWriteEvent and then add all the truncation items of that
transaction. Reloads are triggered based on this list and ValidWriteIds
list of the table is updated accordiingly. For ABORT_TXN events, the
entries in this map will be cleared and no updates happen.
Tests
- Added FE tests for ALTER_TABLE and ALTER_PARTITION events.
- Due to the dependent Hive version is missing HIVE-28668, HMS can't
generate a single ALTER_PARTITIONS event when truncating a
partitioned table. So tests for ALTER_PARTITIONS events are missing.
Assisted-by: Opus 4.7 (Claude Code)
Change-Id: I89aac12819f08dd9ed42d5d8b21a96c04b04d75c
---
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
A fe/src/main/java/org/apache/impala/catalog/TableWriteEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
8 files changed, 346 insertions(+), 51 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/23805/11
--
To view, visit http://gerrit.cloudera.org:8080/23805
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I89aac12819f08dd9ed42d5d8b21a96c04b04d75c
Gerrit-Change-Number: 23805
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Anonymous Coward <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
