Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16649
Change subject: IMPALA-10283: Fix IllegalStateException in applying incremental partition updates ...................................................................... IMPALA-10283: Fix IllegalStateException in applying incremental partition updates When incremental metadata updates are enabled (by default), catalogd sends incremental partition updates based on the last sent table snapshot. Coordinators will apply these partition updates on their existing table snapshots. Each partition update is aka a partition instance. Partition instances are identified by partition ids. Each partition instance is a snapshot of the metadata of a partition. When applying incremental partition updates, ImpaladCatalog#addTable() has a Precondition check assuming that new partition updates should not be duplicated with existing partition ids. The motivation of this check is to detect whether catalogd is sending duplicate partition updates. However, it could be hitted when the coordinator has a newer version of the table than the last sent table snapshot in catalogd. This happens when two coordinators both execute DMLs on the same table (e.g. insert into different partitions), and the DMLs finish within a catalog topic update time window. Note that coordinator will receive a table snapshot from catalogd as a response of the DML request. So one of the coordinator will have a table version that is lower than the latest version in catalogd but larger than the last sent table version in catalogd. When applying incremental partition updates on this coordinator, the Precondition check will be hitted since it has get some partitions in the previous DML response. This patch removes this Precondition check to accept this case. Tests: - Add a test to reproduce the scenario mentioned above. It fails without this patch. Change-Id: I1657684f8853b76b1524475a3b3c35fa22a0e36e --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A tests/custom_cluster/test_incremental_metadata_updates.py 3 files changed, 78 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16649/1 -- To view, visit http://gerrit.cloudera.org:8080/16649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1657684f8853b76b1524475a3b3c35fa22a0e36e Gerrit-Change-Number: 16649 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <[email protected]>
