almostintuitive opened a new issue, #4438:
URL: https://github.com/apache/iggy/issues/4438
### Bug description
After rolling crash-restarts under produce load, a restarted partition
replica can stay behind its group for good. It keeps serving the partition, so
readers that poll it stop at its stale prefix while the other two replicas hold
the full log.
On restart, the replica's durable state ends below the repair window that
its peers still retain. It refuses the commit floor and asks for a state
transfer:
```
refusing commit floor: repaired window does not connect to recovered durable
state (needs state transfer)
```
The state transfer never completes, and the replica does not escalate or
step aside. It keeps answering polls up to its own `commit_min`.
No acknowledged data is lost with `persisted` durability: restarting only
the lagging node made every acknowledged event readable again. Before that
restart, about half of the acknowledged events could not be read through the
stuck replica.
**Expected:** the restarted replica installs a state transfer (or repairs)
and catches up with its group.
**Actual:** it stays at its stale prefix until an operator restarts it.
### Affected area / component
Clustering / replication (partition plane: journal repair and state transfer)
### Deployment
Compiled from source, 3 `iggy-server` processes in Docker containers on one
host.
### Versions
`master` @ `f6dfbdbb3`. The code paths described below are unchanged on
current `master` (`43fb44364`); see the failing tests in the linked PR.
### Hardware / environment
Linux x86_64, local disk. Each node is a container limited to 2 CPU and 4
GiB of memory.
### Reproduction
1. Start a 3-node cluster.
2. Create one stream and one topic with 1 partition and `--durability
persisted`.
3. Produce at a combined 30,000 msg/s for 360 s: 12 producers, batches of
100.
4. During the load, SIGKILL the nodes one at a time, 80 s apart. Keep each
node down for 40 s, then restart it from its data directory. That is: kill node
0 at 80 s, start it at 120 s, kill node 1 at 200 s, start it at 240 s, kill
node 2 at 320 s, start it at 360 s.
5. After the load stops, read the partition from offset 0 and compare
against every acknowledged message.
Results:
- Plain `f6dfbdbb3`: 1 of 2 runs failed. 2,283,600 of 4,570,200 acknowledged
events were unreadable. Two nodes held an identical 1.62 GB on disk, the
lagging node 0.80 GB. After a restart of only the lagging node, 0 of 4,570,200
events were missing.
- A build with additional, unrelated changes: 4 of 4 runs failed with the
same signature.
The failure is timing dependent. It needs a restarted replica whose
recovered segments end below the oldest op that its peers still retain in their
repair ring, which is likely at this write rate.
### Logs
On the lagging node, repeated for minutes (messages verbatim, structured
fields abbreviated):
```
refusing commit floor: repaired window does not connect to recovered durable
state (needs state transfer)
partition state transfer install failed; falling back to journal repair
error="transfer frontier 34274 is below the local commit frontier 36787"
refusing a partition transfer offer from a replica behind this one
serving_view=20 local_view=58
```
In one run the third line appeared 117 times, while the serving primary at
view 20 already had a higher `commit_max` than the lagging replica. In another
run, after the first two causes below were removed, the first line repeated 239
times over 3.5 minutes with no state transfer request at all.
### Analysis
This section is my analysis. I found three separate livelocks in the
state-transfer path. Each one alone keeps the replica behind. The linked PR
adds one failing test per livelock, marked `#[ignore = "#<this issue>: ..."]`.
1. **The install refuses an offer once `commit_max` moves past it during the
pull** (`core/partitions/src/state_transfer.rs`, `install_state_transfer`). The
serving primary builds the offer at its `commit_min`. The pull takes 10 to 20 s
under load, because the active segment alone is hundreds of MiB. Meanwhile, the
primary's commit heartbeats raise the receiver's `commit_max`. The install
refuses any `commit_op < commit_max`, so every round fails with `transfer
frontier X is below the local commit frontier Y`, and the next round is just as
far behind.
The guard exists so that the rewind to `commit_op` never erases a
journaled op that this replica knows is committed. Those ops are `(commit_op,
min(commit_max, sequencer)]`. A gap-stopped receiver has its sequencer below
the offer, so the install erases nothing. The committed tail above the offer is
what journal repair fetches after the install, as the install's doc comment
already describes.
Suggested direction: refuse only when `commit_op < min(commit_max,
sequencer)`. A primary's sequencer is at least its `commit_max`, so primaries
keep the current behavior.
2. **The descriptor gate compares the offer with a view that the replica
raised alone** (`core/shard/src/lib.rs`, `on_partition_state_transfer_target`).
A replica that needs state transfer cannot finish a view change. With
`persisted` durability it cannot vote in one either: its WAL cannot certify a
holed log, so every view-scoped send is withheld. Its elections time out with
nobody hearing them, and its `view` keeps rising (58 against the primary's 20
in the run above). The gate refuses an offer with `serving_view < local view`,
so it refuses every offer from the group's real primary, even when that offer
holds more committed state.
Suggested direction: compare the serving view with the local `log_view`
(the last view whose log this replica adopted). That still refuses a phantom
view-0 primary, an offer from before the adopted view, and any offer below the
local `commit_max`.
3. **A refused commit floor never becomes definitive when the repair window
is longer than one chunk** (`core/partitions/src/iggy_partition.rs`,
`complete_repair`). A floor refusal converts to a state transfer only when the
whole window `(floor, to_op]` is resident, because a later frame could still
lower `first_batch_offset` into connection. But the next 128-op repair chunk is
requested only when the commit walk advances, and a refused floor never lets it
advance. So every stall retry fetches the same first chunk, and the window
never completes. Under load the serving peer eventually evicts the whole
window, which empties it and ends the wait. After load stops, the replica
refuses the floor forever and never asks for a transfer.
Suggested direction: record the op whose batch set `first_batch_offset`.
Offsets rise with ops, so once every op in `(floor, that op)` is resident, no
later frame can lower the first batch offset, and the refusal is final.
None of these changes serves or acknowledges anything new. Directions 1 and
2 decide which committed state a receiver may install. Direction 3 decides when
to start a transfer.
With all three changes applied on top of `f6dfbdbb3` plus unrelated changes,
the same scenario passed 3 of 3 runs with 0 lost events, and all three replicas
ended with identical segment bytes. In each run the lagging replica needed a
state transfer 2 to 26 times and installed it within 10 to 20 s while the load
continued.
A separate question: should a replica that knows it needs state transfer
keep serving reads for the partition? Its data is committed, so it is stale but
never wrong. But a consumer that polls only that replica sees the partition
stop.
### Contribution
- [x] I'm willing to submit a pull request to fix this bug
I have the fix for all three, with tests, and can open it once the direction
is agreed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
