Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21175 )
Change subject: IMPALA-12829: Skip processing transaction events if the table is HMS sync disabled. ...................................................................... Patch Set 24: (6 comments) http://gerrit.cloudera.org:8080/#/c/21175/23/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/21175/23/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@4568 PS23, Line 4568: Hms > nit: "Hms" since we use camel case naming convention. Ack http://gerrit.cloudera.org:8080/#/c/21175/23/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/21175/23/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2987 PS23, Line 2987: prevWriteIdChanged = !writeIdList.toString().equals( : validWriteIds_.writeToString()); > This seems too conservative that if there are only some new open write ids, https://issues.apache.org/jira/browse/IMPALA-14208 http://gerrit.cloudera.org:8080/#/c/21175/23/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/21175/23/tests/custom_cluster/test_events_custom_configs.py@1648 PS23, Line 1648: disabled a > nit: "blacklisted" is confused with the blacklisted_dbs, blacklisted_tables Ack http://gerrit.cloudera.org:8080/#/c/21175/23/tests/custom_cluster/test_events_custom_configs.py@1661 PS23, Line 1661: self.run_stmt_in_hive(create_stmt) : EventProcessorUtils.wait_for_event_processing(self) : if disable_level == 'da > Can we add a similar test for setting this in db properties? Ack http://gerrit.cloudera.org:8080/#/c/21175/23/tests/custom_cluster/test_events_custom_configs.py@1696 PS23, Line 1696: class TestEventProcessingWithImpala(TestEventProcessingCustomConfigsBase): > This sleep is added in IMPALA-12543 and it's intended to use an interval lo In case of transactional tables, insert events are not generated, instead COMMIT_TXN event is generated. Now, if we keep the table metadata delay to 2sec and hms event polling interval to 1sec, the following happens: insert query will update the writeId from 0 to 1 but it is not reflected in catalog cache due to table metadata delay. So the commit txn will be processed by event processor and since the cached writeId is 0 and new writeId is 1, we end up reloading the table and hence the test fails because it failed to detect the self-event. Ideally this should be a self-event because incoming writeId should match with the existing writeId. This was always an existing issue until this was rectified: https://gerrit.cloudera.org/c/21175/23/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java#1036 So we should always keep table metadata delay shorter than event processing delay so that commit transaction can be detected as self-event. http://gerrit.cloudera.org:8080/#/c/21175/23/tests/custom_cluster/test_events_custom_configs.py@1713 PS23, Line 1713: e > Why do we change this? This is changed for the same reason as above. -- To view, visit http://gerrit.cloudera.org:8080/21175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5d0ecb3b756755bc04c66a538a9ae6b88011a019 Gerrit-Change-Number: 21175 Gerrit-PatchSet: 24 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Wed, 09 Jul 2025 05:46:06 +0000 Gerrit-HasComments: Yes
