Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20148
to look at the new patch set (#5).
Change subject: IMPALA-12257: Fix NPE in createInsertEvents when partitions
only exist in HMS
......................................................................
IMPALA-12257: Fix NPE in createInsertEvents when partitions only exist in HMS
When files are generated for INSERT statements, coordinator invokes the
updateCatalog RPC to let catalogd finalize the HMS changes. Catalogd
will fire INSERT events for external tables at the end of this work.
This helps other systems that consume HMS events (e.g. Hive replication
or other Impala clusters) be notified for the changes. It should be
best-effort that any errors in it should not fail the INSERT statement.
We've seen a NullPointerException thrown in createInsertEvents() in the
case that an updated partition doesn't exist in the metadata cache of
catalogd. Note that the partition list cached in catalogd could differ
from the actual list in HMS. If an INSERT statement modifies a partition
that exists in HMS but not yet synced to catalogd, createInsertEvents()
will throw a NPE when finding such a partition in the table. This fixes
the NPE by skipping this step and using the partition names directly.
This patch also refactors createInsertEvents to make it shorter.
Tests
- Add e2e tests
Change-Id: I7d77844e26283ecb16b3b3aeb9f634bb3113eacd
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_insert_behaviour.py
3 files changed, 131 insertions(+), 68 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/20148/5
--
To view, visit http://gerrit.cloudera.org:8080/20148
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7d77844e26283ecb16b3b3aeb9f634bb3113eacd
Gerrit-Change-Number: 20148
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
