Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22956 )
Change subject: IMPALA-14107: Increment table version in fireReloadEvent ...................................................................... IMPALA-14107: Increment table version in fireReloadEvent test_reload_events_with_transient_partitions can get stuck when running in local catalog mode. From the catalod.INFO, it looks like catalog keep looking for non-existent partition id. In local catalog mode, CatalogLookupStatus.PARTITION_NOT_FOUND response from CatalogD will be followed by Coordinator invalidating table metadata cache and table list of database. However, it does not invalidate the partition list of requested table. This cause Coordinator to re-request the same partition id that is not exist anymore in CatalogD. This patch attempt to fix the issue from CatalogD side by incrementing table version in fireReloadEventAndUpdateRefreshEventId if HdfsTable.updatePartition() do detect any partition change. CatalogD will send new topic update that reset Coordinator cache, and Coordinator will retry the same partition request with an updated partition ID. Reworded the InconsistentMetadataFetchException message to clarify the lookup_status failure vs the req.object_desc. This patch also tweak HdfsTable.getPartialInfo() to directly gather all partitions when Coordinator does not request for specific partitions. Testing: Split test_reload_events_with_transient_partitions into two setup: legacy catalog mode and local catalog mode. Confirm that both tests pass. Change-Id: I1bc91023d46a0b57916202d2b64e2cf07b687b74 Reviewed-on: http://gerrit.cloudera.org:8080/22956 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_events_custom_configs.py 4 files changed, 65 insertions(+), 26 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/22956 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1bc91023d46a0b57916202d2b64e2cf07b687b74 Gerrit-Change-Number: 22956 Gerrit-PatchSet: 19 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]>
