Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23805 )
Change subject: IMPALA-14637: COMMIT_TXN events should trigger reload for truncate ops ...................................................................... IMPALA-14637: COMMIT_TXN events should trigger reload for truncate ops Truncate operations generate ALTER events in HMS. These events trigger metadata reloads when catalogd processes them. For a transactional table, the ALTER event could be processed before the transaction is committed. Then a stale snapshot is loaded. Catalogd should reload the metadata in processing the corresponding COMMIT_TXN event. However, that doesn't happen for truncate operations. When processing COMMIT_TXN events, catalogd fetches the WriteEventInfo list for the transaction. This doesn't include the truncate operations (HIVE-29677), which causes COMMIT_TXN events skip the reloads. This patch fixes the issue by tracking transactional truncate operations when receiving ALTER_TABLE, ALTER_PARTITION and ALTER_PARTITIONS events. A map from TableWriteId (db, tbl, writeId) to the truncated partition list is maintained for this. When processing ABORT_TXN events, the entries in this map will be cleared and no updates happen. A new class, TableWriteEvent, is added to represent WriteEventInfo returned from HMS and the truncation info. When processing a COMMIT_TXN event, after fetching the WriteEventInfo list, we convert it into a list of TableWriteEvent and then add all the truncation items of that transaction. Reloads are triggered based on this list and ValidWriteIds list of the table is updated accordingly. Tests - Added FE tests for ALTER_TABLE and ALTER_PARTITION events. - Due to the dependent Hive version is missing HIVE-28668, HMS can't generate a single ALTER_PARTITIONS event when truncating a partitioned table. So tests for ALTER_PARTITIONS events are missing. Assisted-by: Opus 4.7 (Claude Code) Change-Id: I89aac12819f08dd9ed42d5d8b21a96c04b04d75c Reviewed-on: http://gerrit.cloudera.org:8080/23805 Reviewed-by: Zoltan Borok-Nagy <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java A fe/src/main/java/org/apache/impala/catalog/TableWriteEvent.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 8 files changed, 381 insertions(+), 51 deletions(-) Approvals: Zoltan Borok-Nagy: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/23805 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I89aac12819f08dd9ed42d5d8b21a96c04b04d75c Gerrit-Change-Number: 23805 Gerrit-PatchSet: 15 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
