almostintuitive opened a new pull request, #4474: URL: https://github.com/apache/iggy/pull/4474
## Which issue does this PR address? Relates to #4438 Relates to #4436 Also reproduces the data loss reported in [this #4433 comment](https://github.com/apache/iggy/pull/4433#issuecomment-6044243711). The failing tests for #4438 and #4436 are in #4439 and #4437. ## Rationale #4438, #4436 and the #4433 regression only showed up with real server processes crashing, restarting from disk or losing the network while producers write, and checking afterwards that every acknowledged message can be read back. Each one was found with a throwaway script. This puts that tooling in the repository, so a maintainer can rerun a failure case on one machine with one command, and check a fix against it. ## What changed? `scripts/cluster-chaos` starts a 3-node cluster in Docker from a published `apache/iggy` image (default `edge`). Each node has a fixed IP and its data in a named volume, so `kill` then `start` is a real crash and a recovery from disk. `run-scenario.sh <scenario>` creates a topic, produces while it injects the faults of that scenario, heals every node, reads the whole topic back and writes a verdict and a result JSON. Nothing builds the server. The checker is a new binary in `core/tools` (`cluster-chaos-checker`, next to `data-seeder-tool`) on the in-repo SDK, so it speaks the same protocol as a server built from the same commit. `produce` logs every event before its first send and again once a send carrying it is acknowledged, and retries a failed send with the same events. `verify` reads every partition from offset 0 in rounds, because right after a recovery an empty poll or the end that `GetTopic` lists does not prove the end of the log. A scenario fails on acknowledged events lost, offset gaps, per-key reordering, a key in two partitions, events nobody sent, a listed tail that no read returns, reads that never settle, producers giving up, or a panic or fatal shard error in a node log. Upstream has no server-side deduplication, so duplicates from retries are reported, not failed. Also reported but not failed: polls that came back empty at an offset that held data, and `GetTopic` as each node answers it next to what wa s read. | Scenario | Faults | Reproduces | | --- | --- | --- | | `baseline`, `control` | none (`control` resends 20% of events and must see every copy) | checks the checker | | `crash-restart` | SIGKILL each node in turn, restart from disk | data loss with #4433 applied | | `rolling-restart`, `isolate`, `pause`, `kill-storm` | graceful restarts, network cut, freeze, rapid kills | | | `cascading-crash` | a second node dies during the view change of the first | `replicated` loss below | | `full-outage`, `full-outage-replicated` | all three nodes killed at once | `replicated` loss is by design | | `rolling-crash` | 1 partition, 30k msg/s, `persisted`, one kill every 80 s, 40 s down | #4438 | | `split-primaries` | `crash-restart`, then a clean produce phase with a p99 limit | #4436 | `k8s/kind-cluster.sh` implements the same backend commands on a local kind cluster, with one release of `helm/charts/iggy` per node in cluster mode (the chart's `examples/cluster-3-node.yaml`). A kill there is `kubectl delete pod --force` with the Deployment scaled to zero. The README covers every scenario, the expected and the buggy outcome, and how to read the results. ## Local Execution - Passed. All against `apache/iggy:edge` (revision `2c1f70b53`, two commits behind this branch's base, neither touching the server, SDK or protocol), with the checker built from this branch. Nodes at 1 CPU and 2 GiB, 2000 msg/s from 6 producers on 6 partitions for 120 s unless noted: - pass, 0 lost and 0 gaps: `baseline`, `control` (30,517 resent events, 30,517 duplicates seen), `crash-restart`, `rolling-restart`, `isolate`, `pause`, `kill-storm`, `full-outage` (`persisted`), `cascading-crash` (`persisted`), `split-primaries`, and `rolling-crash` at reduced load (5000 msg/s, kills 40 s apart, 20 s down; 866,300 events, 56 `needs state transfer` log lines, no loss). - `full-outage-replicated`: passes as designed, with 38,980 of 239,480 acknowledged events lost (`replicated` has no stable-storage barrier). - `cascading-crash` with `DURABILITY=replicated` fails: 66,120 of 143,520 acknowledged events lost, and a restarted replica stopped with `The replica is divergent and was fenced` and stayed down. An earlier run lost 51,860 of 196,460 the same way. I read this as the same `replicated` trade-off, since two of three nodes crashed within the flush window, but the remaining two nodes then refused sends to some partitions until the producers gave up after 180 s. I can open an issue for that if it is not expected. - `split-primaries` passed: in Docker at this load every partition group followed each kill to the same new primary, so the primaries never split. #4437 is the deterministic reproduction of #4436. - Not run here: `rolling-crash` at the full 30k msg/s (about 11 million events per run), and `crash-restart` against a #4433 build (that needs a server build). The numbers in the README for those come from #4438 and the #4433 comment. - Kubernetes: not run against a cluster. The kind backend was checked with `helm template` for all three releases (with `examples/cluster-3-node.yaml`) and `kubeconform -strict` against Kubernetes 1.35 (12 resources valid), plus `shellcheck`. - The checker: `cargo fmt` and `cargo clippy -p tools --all-targets --all-features -- -D warnings` clean. `docker build -f scripts/cluster-chaos/Dockerfile .` built the image; the final image was built with the same `cargo build --release --locked -p tools --bin cluster-chaos-checker` in a container capped at 2 CPUs. - Pre-commit hooks ran (`pre-commit run --files <changed>`): markdownlint, shellcheck, license headers, version consistency, trailing whitespace and newline, binary artifacts, typos, skills and taplo passed. cargo fmt and clippy ran separately as above. `cargo sort` was not installed, so I checked the `core/tools/Cargo.toml` order by hand. ## AI Usage 1. Claude (Claude Code). 2. Most of the code and text, adapted from scripts I used to find #4438, #4436 and the #4433 regression. 3. By running the scenarios above against the published image and checking that the checker fails when it must (`control` must see every resent copy; the `replicated` outage scenarios lost acknowledged events and were reported). Before opening, I ran a review in the spirit of `team-review-slim` with two reviewers (checker correctness and data-loss logic; shell, kind and docs) and fixed what applied: failed polls no longer count as a quiet read round, a non-advancing poll cannot loop forever, the partition check also covers duplicate copies, the content check compares every field, logs of an earlier run are never overwritten, fault steps no longer abort a run and node logs are saved before teardown, the kind script is executable and maps memory units, and `split-primaries` fails when its measure phase did not run. Not taken: checking each acknowledged event at its acknowledged offset, and reading every replica separately. 4. Yes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
