[
https://issues.apache.org/jira/browse/SOLR-18413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhenyuLi updated SOLR-18413:
----------------------------
Description:
# Description
During a SolrCloud `MIGRATE`, the source slice receives a temporary routing
rule that forwards matching updates to the target collection.
For example, migrating route key `a!` from `source` to `target` creates state
similar to:
```text
source/shard1.routingRules["a!"] = {
targetCollection: "target",
...
}
```
The routing rule references the target collection by name. Deleting `target`
does not remove routing rules in other collections that still refer to it.
This can leave the cluster in the following state:
```text
source/shard1.routingRules["a!"].targetCollection = "target"
ClusterState:
source -> exists
target -> missing
```
While this routing rule remains active, an update to `source` with a matching
ID such as `a!2` is first applied locally on the source shard leader.
The leader then enters `DistributedZkUpdateProcessor.doDistribAdd()`, which
resolves routing-rule destinations before distributing the update to the source
shard replicas:
```java
final List<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState,
coll,
cmd.getIndexedIdStr(),
cmd.getSolrInputDocument());
if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
// Forward to the MIGRATE target.
...
}
if (nodes != null) {
// Distribute to replicas of the source shard.
cmdDistrib.distribAdd(...);
}
```
For a matching routing rule, `getNodesByRoutingRules()` currently resolves the
target with:
```java
DocCollection targetColl =
cstate.getCollection(rule.getTargetCollectionName());
```
If the target collection has already been deleted, this throws:
```text
HTTP 400
Could not find collection : target
```
At this point, however, the source leader has already applied the update
locally.
The exception prevents execution from reaching the subsequent source-replica
distribution block.
The resulting state can therefore be:
```text
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 missing
client:
HTTP 400
```
This is more than a dangling-reference error. The request reports failure after
partially applying the mutation, leaving replicas of the source shard
inconsistent.
No update request is sent to the source replica. Therefore,
`SolrCmdDistributor` does not observe a replica failure. Consequently, this
request does not lower the replica's shard term or directly trigger recovery.
Until another recovery mechanism repairs the replica:
- Queries may return different results depending on which replica serves the
request.
- A leader change may promote a replica that does not contain the update.
- The client cannot safely determine whether retrying the failed request is
appropriate.
- Retrying non-idempotent atomic updates could apply the operation more than
once.
## Execution flow
```text
Client: ADD source / id=a!2
|
v
source shard leader
|
v
versionAdd()
|
v
local index/tlog update
|
v
getNodesByRoutingRules()
|
v
getCollection("target")
|
+---- target missing
|
v
HTTP 400 exception
|
X
source replica distribution
is never executed
```
# Reproduction
1. Start a SolrCloud cluster with at least two nodes.
2. Create a `source` collection with one shard and two replicas:
```text
collection = source
numShards = 1
replicationFactor = 2
```
3. Create a `target` collection:
```text
collection = target
numShards = 1
replicationFactor = 1
```
4. Add and commit a document with a composite ID to the `source` collection:
```text
id = a!1
```
5. Run `MIGRATE` for route key `a!`:
```text
source collection = source
target collection = target
split.key = a!
forward.timeout = 45
```
6. Confirm that `source/shard1` contains an unexpired routing rule for `a!`
whose target collection is `target`.
7. Delete the `target` collection and wait until it is absent from
`ClusterState`.
8. Before the routing rule expires, add another matching document to the
`source` collection:
```text
id = a!2
```
The request returns:
```text
HTTP 400
Could not find collection : target
```
9. Commit the `source` collection and query each source replica directly with
distributed querying disabled:
```text
q=id:"a!2"
distrib=false
```
The result is:
```text
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 missing
```
# Expected behavior
A routing rule whose target collection no longer exists should not abort a
source update after the source leader has already applied it.
The missing target should invalidate the dangling routing rule. Solr should:
1. Stop forwarding updates through the stale routing rule.
2. Remove the dangling rule on a best-effort basis.
3. Continue distributing the update to all replicas of the source shard.
After the request completes:
```text
client:
update succeeds
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 exists
source routing rule:
dangling a! rule removed
```
was:
# Description
During a SolrCloud `MIGRATE`, the source slice receives a temporary routing
rule that forwards matching updates to the target collection.
For example, migrating route key `a!` from `source` to `target` creates state
similar to:
source/shard1.routingRules["a!"] = {
targetCollection: "target",
...
}
The routing rule references the target collection by name. Deleting `target`
does not remove routing rules in other collections that still refer to it.
This can leave the cluster in the following state:
source/shard1.routingRules["a!"].targetCollection = "target"
ClusterState:
source -> exists
target -> missing
While this routing rule remains active, an update to `source` with a matching
ID such as `a!2` is first applied locally on the source shard leader.
The leader then enters `DistributedZkUpdateProcessor.doDistribAdd()`, which
resolves routing-rule destinations before distributing the update to the source
shard replicas:
final List<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState,
coll,
cmd.getIndexedIdStr(),
cmd.getSolrInputDocument());
if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
// Forward to the MIGRATE target.
...
}
if (nodes != null) {
// Distribute to replicas of the source shard.
cmdDistrib.distribAdd(...);
}
For a matching routing rule, `getNodesByRoutingRules()` currently resolves the
target with:
DocCollection targetColl =
cstate.getCollection(rule.getTargetCollectionName());
If the target collection has already been deleted, this throws:
HTTP 400
Could not find collection : target
At this point, however, the source leader has already applied the update
locally.
The exception prevents execution from reaching the subsequent source-replica
distribution block.
The resulting state can therefore be:
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 missing
client:
HTTP 400
This is more than a dangling-reference error. The request reports failure after
partially applying the mutation, leaving replicas of the source shard
inconsistent.
No update request is sent to the source replica. Therefore,
`SolrCmdDistributor` does not observe a replica failure, and this request does
not lower the replica's shard term or directly trigger recovery.
Until another recovery mechanism repairs the replica:
- Queries may return different results depending on which replica serves the
request.
- A leader change may promote a replica that does not contain the update.
- The client cannot safely determine whether the failed request should be
retried.
- Retrying non-idempotent atomic updates could apply the operation more than
once.
## Execution flow
Client: ADD source / id=a!2
|
v
source shard leader
|
v
versionAdd()
|
v
local index/tlog update
|
v
getNodesByRoutingRules()
|
v
getCollection("target")
|
+---- target missing
|
v
HTTP 400 exception
|
X
source replica distribution
is never executed
# Reproduction
1. Start a SolrCloud cluster with at least two nodes.
2. Create a `source` collection with one shard and two replicas:
collection = source
numShards = 1
replicationFactor = 2
3. Create a `target` collection:
collection = target
numShards = 1
replicationFactor = 1
4. Add and commit a document with a composite ID to `source`:
id = a!1
5. Run `MIGRATE` for route key `a!`:
source collection = source
target collection = target
split.key = a!
forward.timeout = 45
6. Confirm that `source/shard1` contains an unexpired routing rule for `a!`
whose target collection is `target`.
7. Delete the `target` collection and wait until it is absent from
`ClusterState`.
8. Before the routing rule expires, add another matching document to `source`:
id = a!2
The request returns:
HTTP 400
Could not find collection : target
9. Commit the `source` collection and query each source replica directly with
distributed querying disabled:
q=id:"a!2"
distrib=false
The result is:
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 missing
# Expected behavior
A routing rule whose target collection no longer exists should not abort a
source update after the source leader has already applied that update locally.
The missing target should invalidate the dangling routing rule.
Solr should:
1. Stop forwarding updates through the stale routing rule.
2. Remove the dangling rule on a best-effort basis.
3. Continue distributing the update to all replicas of the source shard.
After the request completes, the expected state is:
client:
update succeeds
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 exists
source routing rule:
dangling a! rule removed
> Deleting a MIGRATE target collection can leave source shard replicas
> inconsistent
> ---------------------------------------------------------------------------------
>
> Key: SOLR-18413
> URL: https://issues.apache.org/jira/browse/SOLR-18413
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 9.10.1
> Reporter: ZhenyuLi
> Priority: Major
>
> # Description
> During a SolrCloud `MIGRATE`, the source slice receives a temporary routing
> rule that forwards matching updates to the target collection.
> For example, migrating route key `a!` from `source` to `target` creates state
> similar to:
> ```text
> source/shard1.routingRules["a!"] = {
> targetCollection: "target",
> ...
> }
> ```
> The routing rule references the target collection by name. Deleting `target`
> does not remove routing rules in other collections that still refer to it.
> This can leave the cluster in the following state:
> ```text
> source/shard1.routingRules["a!"].targetCollection = "target"
> ClusterState:
> source -> exists
> target -> missing
> ```
> While this routing rule remains active, an update to `source` with a matching
> ID such as `a!2` is first applied locally on the source shard leader.
> The leader then enters `DistributedZkUpdateProcessor.doDistribAdd()`, which
> resolves routing-rule destinations before distributing the update to the
> source shard replicas:
> ```java
> final List<SolrCmdDistributor.Node> nodesByRoutingRules =
> getNodesByRoutingRules(
> clusterState,
> coll,
> cmd.getIndexedIdStr(),
> cmd.getSolrInputDocument());
> if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
> // Forward to the MIGRATE target.
> ...
> }
> if (nodes != null) {
> // Distribute to replicas of the source shard.
> cmdDistrib.distribAdd(...);
> }
> ```
> For a matching routing rule, `getNodesByRoutingRules()` currently resolves
> the target with:
> ```java
> DocCollection targetColl =
> cstate.getCollection(rule.getTargetCollectionName());
> ```
> If the target collection has already been deleted, this throws:
> ```text
> HTTP 400
> Could not find collection : target
> ```
> At this point, however, the source leader has already applied the update
> locally.
> The exception prevents execution from reaching the subsequent source-replica
> distribution block.
> The resulting state can therefore be:
> ```text
> source/shard1 leader:
> a!2 exists
> source/shard1 replica:
> a!2 missing
> client:
> HTTP 400
> ```
> This is more than a dangling-reference error. The request reports failure
> after partially applying the mutation, leaving replicas of the source shard
> inconsistent.
> No update request is sent to the source replica. Therefore,
> `SolrCmdDistributor` does not observe a replica failure. Consequently, this
> request does not lower the replica's shard term or directly trigger recovery.
> Until another recovery mechanism repairs the replica:
> - Queries may return different results depending on which replica serves the
> request.
> - A leader change may promote a replica that does not contain the update.
> - The client cannot safely determine whether retrying the failed request is
> appropriate.
> - Retrying non-idempotent atomic updates could apply the operation more than
> once.
> ## Execution flow
> ```text
> Client: ADD source / id=a!2
> |
> v
> source shard leader
> |
> v
> versionAdd()
> |
> v
> local index/tlog update
> |
> v
> getNodesByRoutingRules()
> |
> v
> getCollection("target")
> |
> +---- target missing
> |
> v
> HTTP 400 exception
> |
> X
> source replica distribution
> is never executed
> ```
> # Reproduction
> 1. Start a SolrCloud cluster with at least two nodes.
> 2. Create a `source` collection with one shard and two replicas:
> ```text
> collection = source
> numShards = 1
> replicationFactor = 2
> ```
> 3. Create a `target` collection:
> ```text
> collection = target
> numShards = 1
> replicationFactor = 1
> ```
> 4. Add and commit a document with a composite ID to the `source` collection:
> ```text
> id = a!1
> ```
> 5. Run `MIGRATE` for route key `a!`:
> ```text
> source collection = source
> target collection = target
> split.key = a!
> forward.timeout = 45
> ```
> 6. Confirm that `source/shard1` contains an unexpired routing rule for `a!`
> whose target collection is `target`.
> 7. Delete the `target` collection and wait until it is absent from
> `ClusterState`.
> 8. Before the routing rule expires, add another matching document to the
> `source` collection:
> ```text
> id = a!2
> ```
> The request returns:
> ```text
> HTTP 400
> Could not find collection : target
> ```
> 9. Commit the `source` collection and query each source replica directly with
> distributed querying disabled:
> ```text
> q=id:"a!2"
> distrib=false
> ```
> The result is:
> ```text
> source/shard1 leader:
> a!2 exists
> source/shard1 replica:
> a!2 missing
> ```
> # Expected behavior
> A routing rule whose target collection no longer exists should not abort a
> source update after the source leader has already applied it.
> The missing target should invalidate the dangling routing rule. Solr should:
> 1. Stop forwarding updates through the stale routing rule.
> 2. Remove the dangling rule on a best-effort basis.
> 3. Continue distributing the update to all replicas of the source shard.
> After the request completes:
> ```text
> client:
> update succeeds
> source/shard1 leader:
> a!2 exists
> source/shard1 replica:
> a!2 exists
> source routing rule:
> dangling a! rule removed
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
