thinker0 opened a new issue, #395:
URL: https://github.com/apache/kvrocks-controller/issues/395
## Summary
`kvctl-server` propagates cluster topology via a push to each data node, but
if the push fails (network blip, target restarting, etc.) it is **not
retried**.
After a controller restart or any partial failure the data nodes end up with
different views of the cluster, and the cluster does not self-heal.
## Observed
In an 18-shard cluster (`replicas=3`, 54 nodes) we ran `CLUSTER NODES` on
four representative masters at the same time:
| Master | Visible nodes (master + slave) |
|--------|--------------------------------|
| shard 0 | **3** (a residual view from a 3-shard cluster created earlier) |
| shard 1 | 17 |
| shard 9 | 33 (a residual view from a 27-shard cluster created in between)
|
| shard 17 | 8 |
| Expected | **54** |
`cluster_state:ok` and `cluster_slots_ok:16384` on every node, so routing
formally works, but every node holds a different `cluster_nodes` snapshot
and clients that bootstrap from a "small" master keep redirecting traffic
to a few hot masters.
The state never converges back. Manager restart (rolling, all 3 instances)
only partially refreshed the views. There is no operator-facing endpoint
(`/sync`, `/refresh`, `/reload`, …) to force a re-push — all returned 404.
## Expected
* Controller retries the per-node push with backoff until each node
acknowledges the new topology version, OR
* Exposes a `POST /clusters/{name}/sync` (or similar) endpoint that
re-pushes the current `cluster_nodes` to every member.
## Repro (vanilla setup)
1. Start `kvctl-server` v1.15.0 and 6 Apache Kvrocks 2.15.0 data nodes.
2. Create a cluster with `replicas=2` (2 shards × 3 nodes).
3. `iptables -A INPUT -p tcp --dport <node-3-port> -j DROP` for ~10 s.
4. Remove the rule, wait, then run `CLUSTER NODES` on every node.
Node 3 has a stale `cluster_nodes` view; nothing converges back even
after several minutes.
## Versions
- `kvctl-server` v1.15.0
- Apache Kvrocks 2.15.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
