Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20148
to look at the new patch set (#6).
Change subject: IMPALA-12257: Fix NPE in createInsertEvents when partitions
only exist in HMS
......................................................................
IMPALA-12257: Fix NPE in createInsertEvents when partitions only exist in HMS
When files have been generated for INSERT statements, coordinator
invokes the updateCatalog RPC to let catalogd finalize the HMS changes.
Catalogd will fire INSERT events for external tables at the end of this
work. This helps other systems that consume HMS events (e.g. Hive
replication or other Impala clusters) be notified for the changes. It
should be best-effort that any errors in it should not fail the INSERT
statement.
We've seen a NullPointerException thrown in createInsertEvents() in the
case that an updated partition doesn't exist in the metadata cache of
catalogd but exist in HMS. Note that the partition list cached in
catalogd could differ from the actual list in HMS, especially when HMS
event processing is disabled. If an INSERT statement modifies a
partition that exists in HMS but not yet synced to catalogd,
createInsertEvents() will throw a NPE when finding such a partition in
catalogd. This fixes the NPE by skipping this finding step and using the
partition names directly.
This patch also refactors createInsertEvents to make it shorter.
Tests
- Add e2e tests
Change-Id: I7d77844e26283ecb16b3b3aeb9f634bb3113eacd
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_insert_behaviour.py
3 files changed, 167 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/20148/6
--
To view, visit http://gerrit.cloudera.org:8080/20148
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7d77844e26283ecb16b3b3aeb9f634bb3113eacd
Gerrit-Change-Number: 20148
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
