Hi all,
We are running SolrCloud on Kubernetes with the Solr Operator and ran
into a problem when changing the number of
Solr pods. When the operator rebalances replicas after adding or
removing a pod, Solr regularly fails part of the
operation and leaves extra replicas behind. We traced it to the
Collections API code (details below) and would like
to know whether this is a known issue and whether we should open a JIRA
ticket.
## Environment
- Solr 9.6.1 (official `solr:9.6.1` image); also reproduced on 9.11.0,
see the note at the end
- Solr Operator v0.9.1, SolrCloud with:
```yaml
scaling:
vacatePodsOnScaleDown: true
populatePodsOnScaleUp: true
```
- Overseer mode (default), embedded ZooKeeper provided by the operator
- 6 collections, each `numShards=3`, `replicationFactor=2` (36 replicas
in total)
## Steps to reproduce
1. Deploy a SolrCloud with `spec.replicas: 2` and the `scaling` settings
above.
2. Create 6 collections with `numShards=3&replicationFactor=2`. The
collections can be empty. The two nodes now hold
18 replicas each.
3. Scale the SolrCloud up by one pod:
```bash
kubectl -n <namespace> patch solrcloud <name> --type merge -p
'{"spec":{"replicas":3}}'
```
(or change `replicas` in the manifest / Helm values and apply it).
The operator creates the third pod, waits until
it is ready and then calls `BALANCE_REPLICAS`. This moves about 12
replicas to the new node.
4. Wait until the operator has finished (the
`solr.apache.org/clusterOpsLock` annotation on the StatefulSet is gone),
then check the replica layout with `CLUSTERSTATUS`.
5. Optionally scale back down to 2 pods. The operator empties the third
pod with `REPLACENODE` /
`MIGRATE_REPLICAS` (about 12 replicas again) before removing it.
## Expected result
All replicas are moved: every shard still has exactly 2 replicas, on 2
different nodes, and the replicas are spread
evenly over the 3 nodes.
## Actual result
- The rebalance fails. The operator logs:
```
INFO Found async status {"requestId": "balance-replicas-ScaleUp",
"state": "failed"}
INFO Replica Balancing command failed. Will try again {"message":
"found [balance-replicas-ScaleUp] in failed tasks"}
```
- The Overseer logs:
```
WARN o.a.s.c.a.c.ReplicaMigrationUtils Error deleting =>
java.util.concurrent.RejectedExecutionException: Task ... rejected from
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@7b091450[Running,
pool size = 10, active threads = 10, queued tasks = 0, completed tasks =
...]
ERROR o.a.s.c.a.c.CollectionHandlingUtils Operation balance_replicas
failed => java.util.concurrent.RejectedExecutionException: ...
```
- The new replicas on the third node exist and are active, but part of
the source replicas were never deleted.
Affected shards end up with 3 replicas.
- The operator's retry (another `BALANCE_REPLICAS`) reports success but
does not remove the extra replicas; they stay
until they are deleted by hand.
- Scaling down from 3 to 2 pods fails in the same way (`Operation
replacenode failed`, same exception).
We tried multiple times and we saw this in 7 of 8 scaling steps that
moved about 12 replicas (5 of 6 scale-ups from 2 to 3 pods, 2 of 2
scale-downs
from 3 to 2 pods), with the `simple`, `minimizecores` and `affinity`
placement plugins. Between 1 and 9 shards were
affected per run, a different set each time. Steps that moved 9 or fewer
replicas usually completed.
## Stack trace
```
java.util.concurrent.RejectedExecutionException: Task
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$337/...
rejected from
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@7b091450[Running,
pool size = 10, active threads = 10, queued tasks = 0, completed tasks =
...]
at
java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(Unknown
Source)
at
java.base/java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:344)
at
java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown
Source)
at
org.apache.solr.cloud.api.collections.DeleteReplicaCmd.deleteCore(DeleteReplicaCmd.java:359)
at
org.apache.solr.cloud.api.collections.DeleteReplicaCmd.deleteReplica(DeleteReplicaCmd.java:107)
at
org.apache.solr.cloud.api.collections.ReplicaMigrationUtils.cleanupReplicas(ReplicaMigrationUtils.java:275)
at
org.apache.solr.cloud.api.collections.ReplicaMigrationUtils.migrateReplicas(ReplicaMigrationUtils.java:246)
at
org.apache.solr.cloud.api.collections.BalanceReplicasCmd.call(BalanceReplicasCmd.java:81)
at
org.apache.solr.cloud.api.collections.CollApiCmds$TraceAwareCommand.call(CollApiCmds.java:225)
at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:130)
at
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:564)
...
```
## What we think happens
(References to
`solr/core/src/java/org/apache/solr/cloud/api/collections/` at 9.6.1.)
1. After the new replicas are active,
`ReplicaMigrationUtils.cleanupReplicas` loops over all source replicas and
calls `DeleteReplicaCmd.deleteReplica` with `parallel=true` for each
of them, without limiting how many deletes
are in flight.
2. With `parallel=true`, `DeleteReplicaCmd.deleteCore` submits the
delete to `ccc.getExecutorService()`, which in
Overseer mode is `OverseerCollectionMessageHandler.tpe`: an
`MDCAwareThreadPoolExecutor(5, 10, 0L, ...,
new SynchronousQueue<>(), ...)` without a rejection handler. With a
`SynchronousQueue` it accepts at most 10
concurrent tasks and rejects the 11th (`AbortPolicy`).
3. `cleanupReplicas` catches `KeeperException` and continues, but
rethrows any other exception. The
`RejectedExecutionException` therefore ends the loop, and the
rejected delete and all deletes after it are never
submitted.
4. The command fails without rolling back the added replicas or
completing the deletes, and since Solr 9 does not
enforce `replicationFactor`, nothing removes the extra replicas later.
So every operation that moves more than 10 replicas at once is likely to
leave replicas behind, depending on how fast
the first deletes finish. The same logic is still on `main` and
`branch_9x`, and the pool used in the distributed
Collection API mode (`DistributedCollectionConfigSetCommandRunner`) has
the same size and queue type.
Possible fixes would be to bound the number of in-flight deletes (or
delete sequentially), to handle the rejection
instead of rethrowing, or to give the pool back-pressure (e.g.
`CallerRunsPolicy`).
## Note: reproducible without the Solr Operator
The operator only triggers the Collections API calls; the failure is in
Solr. We reproduced it without any operator
rebalancing on a fresh SolrCloud with the official `solr:9.11.0` image
(`populatePodsOnScaleUp: false`, so the
operator only provided the pods):
1. 2 nodes, 6 collections with `numShards=3&replicationFactor=2` (36
replicas).
2. Add a third node
3. A manual call to the balance API:
```bash
curl -X POST -H 'Content-Type: application/json' -d '{}'
http://<host>:8983/api/cluster/replicas/balance
```
4. The call returned HTTP 500 with
`rejected from ...MDCAwareThreadPoolExecutor@6e9e33fc[Running, pool
size = 10, active threads = 10, queued tasks = 0, completed tasks = 0]`,
the Overseer logged `ReplicaMigrationUtils Error deleting =>
RejectedExecutionException` and
`Operation balance_replicas failed`, and the cluster was left with
38 replicas instead of 36 (2 shards with
3 replicas each).
Is this a known problem? If not, we are happy to open a JIRA ticket with
these details.
Thank you in advance
