Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20148 )
Change subject: IMPALA-12257: Fix NPE in createInsertEvents when partitions only exist in HMS ...................................................................... IMPALA-12257: Fix NPE in createInsertEvents when partitions only exist in HMS When files have been generated for INSERT statements, coordinator invokes the updateCatalog RPC to let catalogd finalize the HMS changes. Catalogd will fire INSERT events for external tables at the end of this work. This helps other systems that consume HMS events (e.g. Hive replication or other Impala clusters) be notified for the changes. It should be best-effort that any errors in it should not fail the INSERT statement. We've seen a NullPointerException thrown in createInsertEvents() in the case that an updated partition doesn't exist in the metadata cache of catalogd but exist in HMS. Note that the partition list cached in catalogd could differ from the actual list in HMS, especially when HMS event processing is disabled. If an INSERT statement modifies a partition that exists in HMS but not yet synced to catalogd, createInsertEvents() will throw a NPE when finding such a partition in catalogd. This fixes the NPE by skipping this finding step and using the partition names directly. This patch also refactors createInsertEvents to make it shorter. Tests - Add e2e tests Change-Id: I7d77844e26283ecb16b3b3aeb9f634bb3113eacd Reviewed-on: http://gerrit.cloudera.org:8080/20148 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_insert_behaviour.py 3 files changed, 167 insertions(+), 73 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7d77844e26283ecb16b3b3aeb9f634bb3113eacd Gerrit-Change-Number: 20148 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]>
