[
https://issues.apache.org/jira/browse/SOLR-18413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111983#comment-18111983
]
ASF subversion and git services commented on SOLR-18413:
--------------------------------------------------------
Commit 465859177153921cd0c09fb9f6c4d643d2f30d41 in solr's branch
refs/heads/main from Zhenyu Li
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=46585917715 ]
SOLR-18413: Handle deleted MIGRATE routing targets (#4862)
Problem fixed: Say the target of a MIGRATE collection is deleted, yet the
source collection exists and receives an update request. The request reports
failure after partially applying the mutation, leaving replicas of the source
shard inconsistent. No update request is sent to the source replica. Therefore,
SolrCmdDistributor does not observe a replica failure. Consequently, this
request does not lower the replica's shard term or directly trigger recovery.
> Deleting a MIGRATE target collection can leave source shard replicas
> inconsistent
> ---------------------------------------------------------------------------------
>
> Key: SOLR-18413
> URL: https://issues.apache.org/jira/browse/SOLR-18413
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 9.10.1
> Reporter: ZhenyuLi
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> h2. Description
> During a SolrCloud {{{}MIGRATE{}}}, the source slice receives a temporary
> routing rule that forwards matching updates to the target collection.
> For example, migrating route key {{a!}} from {{source}} to {{target}} creates
> state similar to:
> {noformat}
> source/shard1.routingRules["a!"] = {
> targetCollection: "target",
> ...
> }
> {noformat}
> The routing rule references the target collection by name. Deleting
> {{target}} does not remove routing rules in other collections that still
> refer to it.
> This can leave the cluster in the following state:
> {noformat}
> source/shard1.routingRules["a!"].targetCollection = "target"
> ClusterState:
> source -> exists
> target -> missing
> {noformat}
> While this routing rule remains active, an update to {{source}} with a
> matching ID such as {{a!2}} is first applied locally on the source shard
> leader.
> The leader then enters {{{}DistributedZkUpdateProcessor.doDistribAdd(){}}},
> which resolves routing-rule destinations before distributing the update to
> the source shard replicas:
> {code:java}
> final List<SolrCmdDistributor.Node> nodesByRoutingRules =
> getNodesByRoutingRules(
> clusterState,
> coll,
> cmd.getIndexedIdStr(),
> cmd.getSolrInputDocument());
> if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
> // Forward to the MIGRATE target.
> ...
> }
> if (nodes != null) {
> // Distribute to replicas of the source shard.
> cmdDistrib.distribAdd(...);
> }
> {code}
> For a matching routing rule, {{getNodesByRoutingRules()}} currently resolves
> the target with:
> {code:java}
> DocCollection targetColl =
> cstate.getCollection(rule.getTargetCollectionName());
> {code}
> If the target collection has already been deleted, this throws:
> {noformat}
> HTTP 400
> Could not find collection : target
> {noformat}
> At this point, however, the source leader has already applied the update
> locally.
> The exception prevents execution from reaching the subsequent source-replica
> distribution block.
> The resulting state can therefore be:
> {noformat}
> source/shard1 leader: a!2 exists
> source/shard1 replica: a!2 missing
> client: HTTP 400
> {noformat}
> This is more than a dangling-reference error. The request reports failure
> after partially applying the mutation, leaving replicas of the source shard
> inconsistent. No update request is sent to the source replica. Therefore,
> {{SolrCmdDistributor}} does not observe a replica failure. Consequently, this
> request does not lower the replica's shard term or directly trigger recovery.
> h2. Execution flow
> {noformat}
> Client: ADD source / id=a!2
> |
> v
> source shard leader
> |
> v
> versionAdd()
> |
> v
> local index/tlog update
> |
> v
> getNodesByRoutingRules()
> |
> v
> getCollection("target")
> |
> +---- target missing
> |
> v
> HTTP 400 exception
> |
> X source replica distribution is never executed
> {noformat}
> h2. Reproduction
> # Start a SolrCloud cluster with at least two nodes.
> # Create a {{source}} collection with one shard and two replicas:
> {{{}numShards=1{}}}, {{{}replicationFactor=2{}}}.
> # Create a {{target}} collection: {{{}numShards=1{}}},
> {{{}replicationFactor=1{}}}.
> # Add and commit a document with a composite ID {{a!1}} to the {{source}}
> collection.
> # Run {{MIGRATE}} for route key {{{}a!{}}}:
> {noformat}
> source collection = source
> target collection = target
> split.key = a!
> forward.timeout = 45
> {noformat}
> # Confirm that {{source/shard1}} contains an unexpired routing rule for
> {{a!}} whose target collection is {{{}target{}}}.
> # Delete the {{target}} collection and wait until it is absent from
> {{{}ClusterState{}}}.
> # Before the routing rule expires, add another matching document ({{{}id =
> a!2{}}}) to the {{source}} collection. The request returns:
> {noformat}
> HTTP 400
> Could not find collection : target
> {noformat}
> # Commit the {{source}} collection and query each source replica directly
> with distributed querying disabled ({{{}q=id:"a!2"&distrib=false{}}}). The
> result is:
> {noformat}
> source/shard1 leader: a!2 exists
> source/shard1 replica: a!2 missing
> {noformat}
> h2. Expected behavior
> A routing rule whose target collection no longer exists should not abort a
> source update after the source leader has already applied it.
> The missing target should invalidate the dangling routing rule. Solr should:
> # Stop forwarding updates through the stale routing rule.
> # Remove the dangling rule on a best-effort basis.
> # Continue distributing the update to all replicas of the source shard.
> After the request completes:
> {noformat}
> client: update succeeds
> source/shard1 leader: a!2 exists
> source/shard1 replica: a!2 exists
> source routing rule: dangling a! rule removed
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
