Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20145 )
Change subject: IMPALA-12256: Fix DDLs losing create event ids in reloading partitions ...................................................................... IMPALA-12256: Fix DDLs losing create event ids in reloading partitions When HMS event-processor is enabled, each partition will track the event id of the ADD_PARTITION event that corresponds to its creation. The create event id is used to skip stale DROP_PARTITION events that have lower event ids. However, the partition-level create event ids are not correctly retained during DDLs. DDLs that modify the partition metadata will reload the partitions. Since HdfsPartition objects are immutable, the reload is performed as replacing the old one with the new object created by HdfsPartition.Builder. The HdfsPartition.Builder is built from the old HdfsPartition object. But it doesn't copy the create event id. This patch fixes the issue, also adds logs for comparing partition event ids. Tests: - Add e2e tests to verify DDLs won't loss the partition-level create event ids, so stale ADD_PARTITION events can be skipped as expected. Change-Id: I052faa093bda69fb16db0d424c1478bba103dad9 Reviewed-on: http://gerrit.cloudera.org:8080/20145 Reviewed-by: Csaba Ringhofer <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_events_custom_configs.py 5 files changed, 85 insertions(+), 8 deletions(-) Approvals: Csaba Ringhofer: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/20145 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I052faa093bda69fb16db0d424c1478bba103dad9 Gerrit-Change-Number: 20145 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Fang-Yu Rao <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]>
