Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19155 )
Change subject: IMPALA-11626: Handle COMMIT_COMPACTION_EVENT from HMS ...................................................................... Patch Set 9: (4 comments) http://gerrit.cloudera.org:8080/#/c/19155/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/19155/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2681 PS9, Line 2681: "Commit compaction event", FileMetadataLoadOpts.FORCE_LOAD); If the event is triggered by a table level compaction, partitionName_ will be DEFAULT_PARTITION_NAME. I think for partitioned tables, partition of this name doesn't exist. Only non-partitioned table has such a partition. To trigger full table refresh for table-level events, I think we need reloadTableFromCatalog() or something similar. http://gerrit.cloudera.org:8080/#/c/19155/9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/19155/9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4456 PS9, Line 4456: if (table == null && !(table instanceof IncompleteTable) && : (table instanceof HdfsTable)) { hmm, if 'table' is null, it can't be an instance of HdfsTable. This condition is always false.. Do you want to use "table != null"? If so, "table instanceof HdfsTable" already implies 'table' is not null and not an instance of IncompleteTable. http://gerrit.cloudera.org:8080/#/c/19155/9/fe/src/test/resources/hive-site.xml.py File fe/src/test/resources/hive-site.xml.py: http://gerrit.cloudera.org:8080/#/c/19155/9/fe/src/test/resources/hive-site.xml.py@167 PS9, Line 167: 'hive.compactor.delta.num.threshold': 1, Could you add a comment about why we need this for all the tests? This will also be used in data loading. Will it impact the data loading performance? http://gerrit.cloudera.org:8080/#/c/19155/9/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/19155/9/tests/custom_cluster/test_events_custom_configs.py@327 PS9, Line 327: assert EventProcessorUtils.get_event_processor_status() == "ACTIVE" We also need tests on unpartitioned tables and table level compaction on partitioned tables. -- To view, visit http://gerrit.cloudera.org:8080/19155 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I464faedb4e3bbcd417bab2e3cb0d57e339d42605 Gerrit-Change-Number: 19155 Gerrit-PatchSet: 9 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Wed, 15 Feb 2023 01:12:45 +0000 Gerrit-HasComments: Yes
