[
https://issues.apache.org/jira/browse/SOLR-18413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18413:
----------------------------------
Labels: pull-request-available (was: )
> Deleting a MIGRATE target collection can leave source shard replicas
> inconsistent
> ---------------------------------------------------------------------------------
>
> Key: SOLR-18413
> URL: https://issues.apache.org/jira/browse/SOLR-18413
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 9.10.1
> Reporter: ZhenyuLi
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h2. Description
> During a SolrCloud {{{}MIGRATE{}}}, the source slice receives a temporary
> routing rule that forwards matching updates to the target collection.
> For example, migrating route key {{a!}} from {{source}} to {{target}} creates
> state similar to:
> {noformat}
> source/shard1.routingRules["a!"] = {
> targetCollection: "target",
> ...
> }
> {noformat}
> The routing rule references the target collection by name. Deleting
> {{target}} does not remove routing rules in other collections that still
> refer to it.
> This can leave the cluster in the following state:
> {noformat}
> source/shard1.routingRules["a!"].targetCollection = "target"
> ClusterState:
> source -> exists
> target -> missing
> {noformat}
> While this routing rule remains active, an update to {{source}} with a
> matching ID such as {{a!2}} is first applied locally on the source shard
> leader.
> The leader then enters {{{}DistributedZkUpdateProcessor.doDistribAdd(){}}},
> which resolves routing-rule destinations before distributing the update to
> the source shard replicas:
> {code:java}
> final List<SolrCmdDistributor.Node> nodesByRoutingRules =
> getNodesByRoutingRules(
> clusterState,
> coll,
> cmd.getIndexedIdStr(),
> cmd.getSolrInputDocument());
> if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
> // Forward to the MIGRATE target.
> ...
> }
> if (nodes != null) {
> // Distribute to replicas of the source shard.
> cmdDistrib.distribAdd(...);
> }
> {code}
> For a matching routing rule, {{getNodesByRoutingRules()}} currently resolves
> the target with:
> {code:java}
> DocCollection targetColl =
> cstate.getCollection(rule.getTargetCollectionName());
> {code}
> If the target collection has already been deleted, this throws:
> {noformat}
> HTTP 400
> Could not find collection : target
> {noformat}
> At this point, however, the source leader has already applied the update
> locally.
> The exception prevents execution from reaching the subsequent source-replica
> distribution block.
> The resulting state can therefore be:
> {noformat}
> source/shard1 leader: a!2 exists
> source/shard1 replica: a!2 missing
> client: HTTP 400
> {noformat}
> This is more than a dangling-reference error. The request reports failure
> after partially applying the mutation, leaving replicas of the source shard
> inconsistent. No update request is sent to the source replica. Therefore,
> {{SolrCmdDistributor}} does not observe a replica failure. Consequently, this
> request does not lower the replica's shard term or directly trigger recovery.
> h2. Execution flow
> {noformat}
> Client: ADD source / id=a!2
> |
> v
> source shard leader
> |
> v
> versionAdd()
> |
> v
> local index/tlog update
> |
> v
> getNodesByRoutingRules()
> |
> v
> getCollection("target")
> |
> +---- target missing
> |
> v
> HTTP 400 exception
> |
> X source replica distribution is never executed
> {noformat}
> h2. Reproduction
> # Start a SolrCloud cluster with at least two nodes.
> # Create a {{source}} collection with one shard and two replicas:
> {{{}numShards=1{}}}, {{{}replicationFactor=2{}}}.
> # Create a {{target}} collection: {{{}numShards=1{}}},
> {{{}replicationFactor=1{}}}.
> # Add and commit a document with a composite ID {{a!1}} to the {{source}}
> collection.
> # Run {{MIGRATE}} for route key {{{}a!{}}}:
> {noformat}
> source collection = source
> target collection = target
> split.key = a!
> forward.timeout = 45
> {noformat}
> # Confirm that {{source/shard1}} contains an unexpired routing rule for
> {{a!}} whose target collection is {{{}target{}}}.
> # Delete the {{target}} collection and wait until it is absent from
> {{{}ClusterState{}}}.
> # Before the routing rule expires, add another matching document ({{{}id =
> a!2{}}}) to the {{source}} collection. The request returns:
> {noformat}
> HTTP 400
> Could not find collection : target
> {noformat}
> # Commit the {{source}} collection and query each source replica directly
> with distributed querying disabled ({{{}q=id:"a!2"&distrib=false{}}}). The
> result is:
> {noformat}
> source/shard1 leader: a!2 exists
> source/shard1 replica: a!2 missing
> {noformat}
> h2. Expected behavior
> A routing rule whose target collection no longer exists should not abort a
> source update after the source leader has already applied it.
> The missing target should invalidate the dangling routing rule. Solr should:
> # Stop forwarding updates through the stale routing rule.
> # Remove the dangling rule on a best-effort basis.
> # Continue distributing the update to all replicas of the source shard.
> After the request completes:
> {noformat}
> client: update succeeds
> source/shard1 leader: a!2 exists
> source/shard1 replica: a!2 exists
> source routing rule: dangling a! rule removed
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
