almostintuitive commented on issue #4266: URL: https://github.com/apache/iggy/issues/4266#issuecomment-6074731238
Data points for this contract from a real-process test harness. It runs a 3-node cluster in Docker; nodes keep their data on volumes and are crashed with SIGKILL. Producers retry every send until it is acknowledged, then a verifier reads every partition back from offset 0 and checks the acknowledged set. The server was built from `f6dfbdbb3` with unrelated patches, and the upstream image showed the same recovery behaviour. **1. Complete cluster process crash, `Replicated`, under load (decision 1)** - Setup: 6 partitions, about 2k msg/s, all three nodes SIGKILLed at the same time, restarted 15 s later. - `durability=replicated`: **2,829 of 179,400 acknowledged messages missing** after recovery. - `durability=persisted`: 0 missing, in two runs. This matches the documented acknowledgement contract (quorum commit with no stable-storage barrier). But it is exactly the "acknowledged data that existed only in volatile state" case, and an operator learns it only by losing data. An executable scenario plus a sentence in the durability docs would make it explicit. **2. A SIGKILLed replica recovers its segment as empty and relies on peers (decisions 1 and 2)** On every build we tried (including master), a node restarted after SIGKILL during writes logs: - `partitions::segment_recovery: truncating a segment file to its recovered bounds; discarding torn tail bytes ... target_size=0` - and on some builds: `no whole batch decodes ... recovering the segment as empty`. It then repairs or transfers everything from its peers. With `persisted` we never lost acknowledged data this way. But the restarted node's own durable copy contributes nothing to recovery, which makes evidence from peers the only path. It would help to state whether that is intended. **3. Rolling crash-restarts under load can leave a recovered replica unable to resume while it still serves reads (decisions 2 and 3)** Filed as #4438, with failing tests in #4439. - Setup: 1 partition, 30k msg/s, `persisted`, nodes SIGKILLed one at a time 80 s apart, each down 40 s. - What happens: a restarted replica's durable state ends before the repair window it is offered, and it logs `refusing commit floor: ... (needs state transfer)`. - The state transfer never completes, yet the replica keeps serving its stale prefix. Restarting it recovers every acknowledged message. - On upstream this happened in 1 of 2 runs. It answers decision 2 in practice: a replica without sufficient evidence still served reads. **4. Related:** the crash-restart regression reported on #4433 (https://github.com/apache/iggy/pull/4433#issuecomment-6044243711). There, a restarted node holds back its whole history and recovers only the first `repair_chunk_max` ops. We're happy to contribute these as executable scenarios. A single-machine Docker harness that runs exactly these failure sequences is being prepared as a separate PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
