Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21437
to look at the new patch set (#15).
Change subject: IMPALA-12277: Fix NullPointerException for partitioned inserts
when partition list is stale
......................................................................
IMPALA-12277: Fix NullPointerException for partitioned inserts when
partition list is stale
When event processor is turned off, inserting values into partitioned
table can lead to NullPointerException if the partition is deleted
outside impala (eg: HMS). Since event processor is turned off, impala
is unaware of the metadata changes to the table.
Currently in impala, we always load the partitions from cached table.
This can lead to data inconsistency issue especially in the case
of event processor being turned off or lagged behind. This patch
address this issue by always verify the partitions from HMS without
loading the file metadata from HMS regardless of state ofevent
processor. This approach will ensure that partition metadata is
always consistent with metastore.
The issue can be seen with the following steps:
- Turn off the event processor
- create a partitioned table and add a partition from impala
- drop the same partition from hive
- from impala, insert values into the partition (expectation is that
if the partition didn't exist, it will create a new one).
Testing:
- Verified manually that NullPointerException is avoided with this patch
- Added end-to-end tests to verify the above scenario for external and
manged tables.
Change-Id: Ide8f1f6bf017e9a040b53bb5d5291ff2ea3e0d18
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/catalog/HdfsTableLoadParams.java
A fe/src/main/java/org/apache/impala/catalog/HdfsTableLoadParamsBuilder.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
5 files changed, 534 insertions(+), 191 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/21437/15
--
To view, visit http://gerrit.cloudera.org:8080/21437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ide8f1f6bf017e9a040b53bb5d5291ff2ea3e0d18
Gerrit-Change-Number: 21437
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]>
