[
https://issues.apache.org/jira/browse/SOLR-18413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhenyuLi updated SOLR-18413:
----------------------------
Description:
h2. Description
During a SolrCloud {{{}MIGRATE{}}}, the source slice receives a temporary
routing rule that forwards matching updates to the target collection.
For example, migrating route key {{a!}} from {{source}} to {{target}} creates
state similar to:
{noformat}
source/shard1.routingRules["a!"] = {
targetCollection: "target",
...
}
{noformat}
The routing rule references the target collection by name. Deleting {{target}}
does not remove routing rules in other collections that still refer to it.
This can leave the cluster in the following state:
{noformat}
source/shard1.routingRules["a!"].targetCollection = "target"
ClusterState:
source -> exists
target -> missing
{noformat}
While this routing rule remains active, an update to {{source}} with a matching
ID such as {{a!2}} is first applied locally on the source shard leader.
The leader then enters {{{}DistributedZkUpdateProcessor.doDistribAdd(){}}},
which resolves routing-rule destinations before distributing the update to the
source shard replicas:
{code:java}
final List<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState,
coll,
cmd.getIndexedIdStr(),
cmd.getSolrInputDocument());
if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
// Forward to the MIGRATE target.
...
}
if (nodes != null) {
// Distribute to replicas of the source shard.
cmdDistrib.distribAdd(...);
}
{code}
For a matching routing rule, {{getNodesByRoutingRules()}} currently resolves
the target with:
{code:java}
DocCollection targetColl =
cstate.getCollection(rule.getTargetCollectionName());
{code}
If the target collection has already been deleted, this throws:
{noformat}
HTTP 400
Could not find collection : target
{noformat}
At this point, however, the source leader has already applied the update
locally.
The exception prevents execution from reaching the subsequent source-replica
distribution block.
The resulting state can therefore be:
{noformat}
source/shard1 leader: a!2 exists
source/shard1 replica: a!2 missing
client: HTTP 400
{noformat}
This is more than a dangling-reference error. The request reports failure after
partially applying the mutation, leaving replicas of the source shard
inconsistent. No update request is sent to the source replica. Therefore,
{{SolrCmdDistributor}} does not observe a replica failure. Consequently, this
request does not lower the replica's shard term or directly trigger recovery.
h2. Execution flow
{noformat}
Client: ADD source / id=a!2
|
v
source shard leader
|
v
versionAdd()
|
v
local index/tlog update
|
v
getNodesByRoutingRules()
|
v
getCollection("target")
|
+---- target missing
|
v
HTTP 400 exception
|
X source replica distribution is never executed
{noformat}
h2. Reproduction
# Start a SolrCloud cluster with at least two nodes.
# Create a {{source}} collection with one shard and two replicas:
{{{}numShards=1{}}}, {{{}replicationFactor=2{}}}.
# Create a {{target}} collection: {{{}numShards=1{}}},
{{{}replicationFactor=1{}}}.
# Add and commit a document with a composite ID {{a!1}} to the {{source}}
collection.
# Run {{MIGRATE}} for route key {{{}a!{}}}:
{noformat}
source collection = source
target collection = target
split.key = a!
forward.timeout = 45
{noformat}
# Confirm that {{source/shard1}} contains an unexpired routing rule for {{a!}}
whose target collection is {{{}target{}}}.
# Delete the {{target}} collection and wait until it is absent from
{{{}ClusterState{}}}.
# Before the routing rule expires, add another matching document ({{{}id =
a!2{}}}) to the {{source}} collection. The request returns:
{noformat}
HTTP 400
Could not find collection : target
{noformat}
# Commit the {{source}} collection and query each source replica directly with
distributed querying disabled ({{{}q=id:"a!2"&distrib=false{}}}). The result is:
{noformat}
source/shard1 leader: a!2 exists
source/shard1 replica: a!2 missing
{noformat}
h2. Expected behavior
A routing rule whose target collection no longer exists should not abort a
source update after the source leader has already applied it.
The missing target should invalidate the dangling routing rule. Solr should:
# Stop forwarding updates through the stale routing rule.
# Remove the dangling rule on a best-effort basis.
# Continue distributing the update to all replicas of the source shard.
After the request completes:
{noformat}
client: update succeeds
source/shard1 leader: a!2 exists
source/shard1 replica: a!2 exists
source routing rule: dangling a! rule removed
{noformat}
was:
h2. Description
During a SolrCloud {{MIGRATE}}, the source slice receives a temporary routing
rule that forwards matching updates to the target collection.
For example, migrating route key {{a!}} from {{source}} to {{target}} creates
state similar to:
{noformat}
source/shard1.routingRules["a!"] = {
targetCollection: "target",
...
}
{noformat}
The routing rule references the target collection by name. Deleting {{target}}
does not remove routing rules in other collections that still refer to it.
This can leave the cluster in the following state:
{noformat}
source/shard1.routingRules["a!"].targetCollection = "target"
ClusterState:
source -> exists
target -> missing
{noformat}
While this routing rule remains active, an update to {{source}} with a matching
ID such as {{a!2}} is first applied locally on the source shard leader.
The leader then enters {{DistributedZkUpdateProcessor.doDistribAdd()}}, which
resolves routing-rule destinations before distributing the update to the source
shard replicas:
{code:java}
final List<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState,
coll,
cmd.getIndexedIdStr(),
cmd.getSolrInputDocument());
if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
// Forward to the MIGRATE target.
...
}
if (nodes != null) {
// Distribute to replicas of the source shard.
cmdDistrib.distribAdd(...);
}
{code}
For a matching routing rule, {{getNodesByRoutingRules()}} currently resolves
the target with:
{code:java}
DocCollection targetColl =
cstate.getCollection(rule.getTargetCollectionName());
{code}
If the target collection has already been deleted, this throws:
{noformat}
HTTP 400
Could not find collection : target
{noformat}
At this point, however, the source leader has already applied the update
locally.
The exception prevents execution from reaching the subsequent source-replica
distribution block.
The resulting state can therefore be:
{noformat}
source/shard1 leader: a!2 exists
source/shard1 replica: a!2 missing
client: HTTP 400
{noformat}
This is more than a dangling-reference error. The request reports failure after
partially applying the mutation, leaving replicas of the source shard
inconsistent.
No update request is sent to the source replica. Therefore,
{{SolrCmdDistributor}} does not observe a replica failure. Consequently, this
request does not lower the replica's shard term or directly trigger recovery.
Until another recovery mechanism repairs the replica:
* Queries may return different results depending on which replica serves the
request.
* A leader change may promote a replica that does not contain the update.
* The client cannot safely determine whether retrying the failed request is
appropriate.
* Retrying non-idempotent atomic updates could apply the operation more than
once.
h2. Execution flow
{noformat}
Client: ADD source / id=a!2
|
v
source shard leader
|
v
versionAdd()
|
v
local index/tlog update
|
v
getNodesByRoutingRules()
|
v
getCollection("target")
|
+---- target missing
|
v
HTTP 400 exception
|
X source replica distribution is never executed
{noformat}
h2. Reproduction
# Start a SolrCloud cluster with at least two nodes.
# Create a {{source}} collection with one shard and two replicas:
{{numShards=1}}, {{replicationFactor=2}}.
# Create a {{target}} collection: {{numShards=1}}, {{replicationFactor=1}}.
# Add and commit a document with a composite ID {{a!1}} to the {{source}}
collection.
# Run {{MIGRATE}} for route key {{a!}}:
{noformat}
source collection = source
target collection = target
split.key = a!
forward.timeout = 45
{noformat}
# Confirm that {{source/shard1}} contains an unexpired routing rule for {{a!}}
whose target collection is {{target}}.
# Delete the {{target}} collection and wait until it is absent from
{{ClusterState}}.
# Before the routing rule expires, add another matching document ({{id = a!2}})
to the {{source}} collection. The request returns:
{noformat}
HTTP 400
Could not find collection : target
{noformat}
# Commit the {{source}} collection and query each source replica directly with
distributed querying disabled ({{q=id:"a!2"&distrib=false}}). The result is:
{noformat}
source/shard1 leader: a!2 exists
source/shard1 replica: a!2 missing
{noformat}
h2. Expected behavior
A routing rule whose target collection no longer exists should not abort a
source update after the source leader has already applied it.
The missing target should invalidate the dangling routing rule. Solr should:
# Stop forwarding updates through the stale routing rule.
# Remove the dangling rule on a best-effort basis.
# Continue distributing the update to all replicas of the source shard.
After the request completes:
{noformat}
client: update succeeds
source/shard1 leader: a!2 exists
source/shard1 replica: a!2 exists
source routing rule: dangling a! rule removed
{noformat}
> Deleting a MIGRATE target collection can leave source shard replicas
> inconsistent
> ---------------------------------------------------------------------------------
>
> Key: SOLR-18413
> URL: https://issues.apache.org/jira/browse/SOLR-18413
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 9.10.1
> Reporter: ZhenyuLi
> Priority: Major
>
> h2. Description
> During a SolrCloud {{{}MIGRATE{}}}, the source slice receives a temporary
> routing rule that forwards matching updates to the target collection.
> For example, migrating route key {{a!}} from {{source}} to {{target}} creates
> state similar to:
> {noformat}
> source/shard1.routingRules["a!"] = {
> targetCollection: "target",
> ...
> }
> {noformat}
> The routing rule references the target collection by name. Deleting
> {{target}} does not remove routing rules in other collections that still
> refer to it.
> This can leave the cluster in the following state:
> {noformat}
> source/shard1.routingRules["a!"].targetCollection = "target"
> ClusterState:
> source -> exists
> target -> missing
> {noformat}
> While this routing rule remains active, an update to {{source}} with a
> matching ID such as {{a!2}} is first applied locally on the source shard
> leader.
> The leader then enters {{{}DistributedZkUpdateProcessor.doDistribAdd(){}}},
> which resolves routing-rule destinations before distributing the update to
> the source shard replicas:
> {code:java}
> final List<SolrCmdDistributor.Node> nodesByRoutingRules =
> getNodesByRoutingRules(
> clusterState,
> coll,
> cmd.getIndexedIdStr(),
> cmd.getSolrInputDocument());
> if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
> // Forward to the MIGRATE target.
> ...
> }
> if (nodes != null) {
> // Distribute to replicas of the source shard.
> cmdDistrib.distribAdd(...);
> }
> {code}
> For a matching routing rule, {{getNodesByRoutingRules()}} currently resolves
> the target with:
> {code:java}
> DocCollection targetColl =
> cstate.getCollection(rule.getTargetCollectionName());
> {code}
> If the target collection has already been deleted, this throws:
> {noformat}
> HTTP 400
> Could not find collection : target
> {noformat}
> At this point, however, the source leader has already applied the update
> locally.
> The exception prevents execution from reaching the subsequent source-replica
> distribution block.
> The resulting state can therefore be:
> {noformat}
> source/shard1 leader: a!2 exists
> source/shard1 replica: a!2 missing
> client: HTTP 400
> {noformat}
> This is more than a dangling-reference error. The request reports failure
> after partially applying the mutation, leaving replicas of the source shard
> inconsistent. No update request is sent to the source replica. Therefore,
> {{SolrCmdDistributor}} does not observe a replica failure. Consequently, this
> request does not lower the replica's shard term or directly trigger recovery.
> h2. Execution flow
> {noformat}
> Client: ADD source / id=a!2
> |
> v
> source shard leader
> |
> v
> versionAdd()
> |
> v
> local index/tlog update
> |
> v
> getNodesByRoutingRules()
> |
> v
> getCollection("target")
> |
> +---- target missing
> |
> v
> HTTP 400 exception
> |
> X source replica distribution is never executed
> {noformat}
> h2. Reproduction
> # Start a SolrCloud cluster with at least two nodes.
> # Create a {{source}} collection with one shard and two replicas:
> {{{}numShards=1{}}}, {{{}replicationFactor=2{}}}.
> # Create a {{target}} collection: {{{}numShards=1{}}},
> {{{}replicationFactor=1{}}}.
> # Add and commit a document with a composite ID {{a!1}} to the {{source}}
> collection.
> # Run {{MIGRATE}} for route key {{{}a!{}}}:
> {noformat}
> source collection = source
> target collection = target
> split.key = a!
> forward.timeout = 45
> {noformat}
> # Confirm that {{source/shard1}} contains an unexpired routing rule for
> {{a!}} whose target collection is {{{}target{}}}.
> # Delete the {{target}} collection and wait until it is absent from
> {{{}ClusterState{}}}.
> # Before the routing rule expires, add another matching document ({{{}id =
> a!2{}}}) to the {{source}} collection. The request returns:
> {noformat}
> HTTP 400
> Could not find collection : target
> {noformat}
> # Commit the {{source}} collection and query each source replica directly
> with distributed querying disabled ({{{}q=id:"a!2"&distrib=false{}}}). The
> result is:
> {noformat}
> source/shard1 leader: a!2 exists
> source/shard1 replica: a!2 missing
> {noformat}
> h2. Expected behavior
> A routing rule whose target collection no longer exists should not abort a
> source update after the source leader has already applied it.
> The missing target should invalidate the dangling routing rule. Solr should:
> # Stop forwarding updates through the stale routing rule.
> # Remove the dangling rule on a best-effort basis.
> # Continue distributing the update to all replicas of the source shard.
> After the request completes:
> {noformat}
> client: update succeeds
> source/shard1 leader: a!2 exists
> source/shard1 replica: a!2 exists
> source routing rule: dangling a! rule removed
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
