[
https://issues.apache.org/jira/browse/IMPALA-10283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237313#comment-17237313
]
ASF subversion and git services commented on IMPALA-10283:
----------------------------------------------------------
Commit acc3de40fb6633af21f33fb51631a6b567191887 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=acc3de4 ]
IMPALA-10283: Fix IllegalStateException in applying incremental partition
updates
When incremental metadata updates are enabled (by default), catalogd
sends incremental partition updates based on the last sent table
snapshot. Coordinators will apply these partition updates on their
existing table snapshots.
Each partition update is via a partition instance. Partition instances
are identified by partition ids. Each partition instance is a snapshot
of the metadata of a partition. When applying incremental partition
updates, ImpaladCatalog#addTable() has a Precondition check assuming
that new partition updates should not be duplicated with existing
partition ids.
The motivation of this check is to detect whether catalogd is sending
duplicate partition updates. However, it could be hitted when the
coordinator has a newer version of the table than the last sent table
snapshot in catalogd. This happens when two coordinators both execute
DMLs on the same table (e.g. insert into different partitions), and the
DMLs finish within a catalog topic update time window. Note that
coordinator will receive a table snapshot from catalogd as a response of
the DML request. So one of the coordinator will have a table version
that is lower than the latest version in catalogd but larger than the
last sent table version in catalogd. For an example, let's see the
following sequence of events on a table:
t0: coord1 and coord2 both have the latest version as catalogd
t1: coord1 executes a DML to add a partition p2
t2: coord2 executes a DML to add another partition p3
t3: catalogd sends topic update with {p2, p3}
t1 and t2 happen inside a topic-update window. So catalogd will send the
update of {p2, p3}. The following table shows the table version and
corresponding partition instances in each server.
+----+---------------+--------------+---------------+
| | catalogd | coordinator1 | coordinator2 |
+----+---------------+--------------+---------------+
| t0 | v0:{p1} | v0:{p1} | v0:{p1} |
+----+---------------+--------------+---------------+
| t1 | v1:{p1,p2} | v1:{p1,p2} | v0:{p1} |
+----+---------------+--------------+---------------+
| t2 | v2:{p1,p2,p3} | v1:{p1,p2} | v2:{p1,p2,p3} |
+----+---------------+--------------+---------------+
At t3, coordinator2 will skip the table update since it already has a
version equal to the one in the topic update. However, on coordinator1,
the table version is smaller than v2, so it will apply the incremental
updates of {p2,p3} and then hit the Precondition check complaining that
p2 already exists.
It's legal that a coordinator has got some partition instances in the
DML responses. So we can't assume that all partition updates in a topic
update don't exist in the coordinator. This patch removes this
Precondition check to accept this case.
Tests:
- Add a test to reproduce the scenario mentioned above. It fails
without this patch.
Change-Id: I1657684f8853b76b1524475a3b3c35fa22a0e36e
Reviewed-on: http://gerrit.cloudera.org:8080/16649
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> IllegalStateException in applying incremental partition updates
> ---------------------------------------------------------------
>
> Key: IMPALA-10283
> URL: https://issues.apache.org/jira/browse/IMPALA-10283
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 4.0
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> When incremental metatdata updates are enabled (by default), catalogd sends
> incremental partition updates based on the last sent table snapshot.
> Coordinators will apply these partition updates on their existing table
> snapshots.
> Each partition update is aka a partition instance. Partition instances are
> identified by partition ids. Each partition instance is a snapshot of the
> metadata of a partition. When applying incremental partition updates, there
> is a Precondition check assuming that new partition updates should not be
> duplicated with existing partition ids:
> [https://github.com/apache/impala/blob/3ba8d637cdf38a68e25e573afa8d1d05047df2f6/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java#L515]
> The motivation of this check is to detect whether catalogd is sending
> duplicated partition updates. However, it could be hitted when the
> coordinator has a newer version of the table than the last sent table
> snapshot in catalogd. This happens when two coordinators both execute DMLs on
> the same table, and the DMLs finish within a catalog topic update time
> window. Note that coordinator will receive a table snapshot from catalogd as
> a response of the DML request. So one of the coordinator will have a table
> version that is lower than the latest version in catalogd but larger than the
> last sent table version in catalogd. When applying incremental partition
> updates on this coordinator, the Precondition check will be hitted. We should
> remove this check to accept this case.
> To reproduce the issue, create a partitioned table and warm up its metadata
> cache by running any query on it (e.g. describe)
> {code:sql}
> create table multi_inserts_tbl (id int) partitioned by (p int);
> desc multi_inserts_tbl;
> {code}
> Run two inserts using two different coordinators in one command:
> {code:java}
> bin/impala-shell.sh -q "insert into multi_inserts_tbl partition (p) values
> (0, 0)"; bin/impala-shell.sh -i localhost:21051 -q "insert into
> multi_inserts_tbl partition (p) values (1, 1)"
> {code}
> We may find the following IllegalStateException in logs of the first
> coordinator:
> {code:java}
> I1026 14:54:06.127398 11497 ImpaladCatalog.java:224] Adding:
> TABLE:default.multi_inserts_tbl version: 1495 size: 1464
> I1026 14:54:06.127557 11497 ImpaladCatalog.java:224] Adding:
> CATALOG_SERVICE_ID version: 1495 size: 60
> I1026 14:54:06.127887 11497 ImpaladCatalog.java:249] Adding 2 partition(s):
> HDFS_PARTITION:default.multi_inserts_tbl:(p=0,p=1), version=1495,
> size=(avg=597, min=597, max=597, sum=1194)
> E1026 14:54:06.134311 11497 ImpaladCatalog.java:256] Error adding catalog
> object: null
> Java exception follows:
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:492)
> at
> org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:515)
> at
> org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:325)
> at
> org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:254)
> at
> org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:114)
> at
> org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:378)
> at
> org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:178)
> {code}
> This makes the first coordinator fail to update the table to the latest
> version:
> {code:java}
> $ bin/impala-shell.sh -q "show partitions multi_inserts_tbl"
> Starting Impala Shell with no authentication using Python 2.7.16
> Warning: live_progress only applies to interactive shell sessions, and is
> being skipped for now.
> Opened TCP connection to localhost:21050
> Connected to localhost:21050
> Server version: impalad version 4.0.0-SNAPSHOT DEBUG (build
> 3ba8d637cdf38a68e25e573afa8d1d05047df2f6)
> Query: show partitions multi_inserts_tbl
> +-------+-------+--------+------+--------------+-------------------+--------+-------------------+-------------------------------------------------------------+
> | p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format |
> Incremental stats | Location
> |
> +-------+-------+--------+------+--------------+-------------------+--------+-------------------+-------------------------------------------------------------+
> | 0 | -1 | 1 | 2B | NOT CACHED | NOT CACHED | TEXT |
> false |
> hdfs://localhost:20500/test-warehouse/multi_inserts_tbl/p=0 |
> | Total | -1 | 1 | 2B | 0B | | |
> |
> |
> +-------+-------+--------+------+--------------+-------------------+--------+-------------------+-------------------------------------------------------------+
> Fetched 2 row(s) in 0.01s {code}
> Partition p=1 is missing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
