[
https://issues.apache.org/jira/browse/IGNITE-29088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maksim Davydov updated IGNITE-29088:
------------------------------------
Description:
When a node leaves the cluster with the only copy of a partition, another node
takes the partition over. Under the IGNORE policy it becomes the owner of an
empty partition, and under the safe policies it marks the partition as lost.
The other nodes don't wait for it to report this and write the new state into
their partition maps themselves. If the new owner sent its partition map
shortly before, the coordinator treats that map as newer and sends it to the
other nodes, so until the next partition map exchange they see the partition
without an owner or not marked as lost. Under IGNORE, SQL queries from these
nodes fail with "Failed to map SQL query to topology", while key operations
work.
With the fix, only the node that holds a partition reports its state: it sends
its partition map at the end of the exchange, and the other nodes take the
state from it. Until the map arrives, about 1.5 seconds by default, they see no
owner of the partition, and SQL queries wait instead of failing. The same
applies when a node takes over partitions that no other node holds, or marks
its partitions as lost when it joins. In a rolling upgrade, the nodes switch to
the new behavior only after the whole cluster runs the new version.
was:
Under any partition loss policy, the coordinator's partition map can keep a
stale state of a lost partition until the next exchange.
When a server node leaves with the only copy of a partition, the new primary
creates the partition as MOVING on the coordinator's full message. Then, in
detectLostPartitions, it owns the partition (IGNORE) or marks it LOST (SAFE
policies). The coordinator sets the same state in its own detectLostPartitions.
If the new primary sends its partition map between these two steps, for example
a scheduled resend after an eviction, the map carries MOVING with a newer
update sequence, and the coordinator takes it. Nothing sends the map again
after the partition changes.
Under IGNORE, the coordinator and every node that takes its full map see no
owner of the partition until the next exchange. SQL queries from those nodes
fail with `Failed to map SQL query to topology during timeout: 30000ms`, while
key operations and scans work. `awaitPartitionMapExchange()` times out in
tests:` IgniteTopologyValidatorGridSplitCacheTest` (32 nodes, 50 caches without
backups) hits this in about half of local runs. Under SAFE, the coordinator's
map shows MOVING instead of LOST.
Fix: when detectLostPartitions changes a local partition, a non-coordinator
node resends its single map. GridDhtPartitionTopologyImpl#detectLostPartitions
reports a change of any local partition, not only of the last one it checks.
> Fix stale lost partition states in the coordinator's partition map
> ------------------------------------------------------------------
>
> Key: IGNITE-29088
> URL: https://issues.apache.org/jira/browse/IGNITE-29088
> Project: Ignite
> Issue Type: Bug
> Reporter: Maksim Davydov
> Assignee: Maksim Davydov
> Priority: Major
> Labels: ise
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> When a node leaves the cluster with the only copy of a partition, another
> node takes the partition over. Under the IGNORE policy it becomes the owner
> of an empty partition, and under the safe policies it marks the partition as
> lost. The other nodes don't wait for it to report this and write the new
> state into their partition maps themselves. If the new owner sent its
> partition map shortly before, the coordinator treats that map as newer and
> sends it to the other nodes, so until the next partition map exchange they
> see the partition without an owner or not marked as lost. Under IGNORE, SQL
> queries from these nodes fail with "Failed to map SQL query to topology",
> while key operations work.
> With the fix, only the node that holds a partition reports its state: it
> sends its partition map at the end of the exchange, and the other nodes take
> the state from it. Until the map arrives, about 1.5 seconds by default, they
> see no owner of the partition, and SQL queries wait instead of failing. The
> same applies when a node takes over partitions that no other node holds, or
> marks its partitions as lost when it joins. In a rolling upgrade, the nodes
> switch to the new behavior only after the whole cluster runs the new version.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)