diegomrsantos opened a new issue, #4266: URL: https://github.com/apache/iggy/issues/4266
### Question What recovery behavior should Iggy guarantee for a partition with both `durability` and `consumer_offset_durability` set to `replicated`, particularly when every replica has lost its volatile prepare history? The acknowledgment contract is documented, but we should make the supported recovery cases and the conditions for resuming service equally explicit, then exercise them through actual cluster startup. This question arose while investigating #4128 and #4130, but applies to partitions that have never been purged too. ### Context Source links below refer to #4130 at `691ad0f`. #4092 introduced two independent topic policies. `Replicated` completes after VSR quorum commit without requiring recoverable copies on stable storage before acknowledging the operation. `Persisted` requires those copies on the quorum. Both policies normally write data to disk; `Replicated` does not mean that the topic is intentionally stored only in memory. See the [documented contract](https://github.com/apache/iggy/blob/691ad0f54e7eb83f85fd45303b0e7a7ba088f42a/README.md#L285-L295). For a group with multiple replicas, the [prepare WAL is opened only when at least one of those policies is `Persisted`](https://github.com/apache/iggy/blob/691ad0f54e7eb83f85fd45303b0e7a7ba088f42a/core/partitions/src/iggy_partition.rs#L897-L908). A persisted policy for consumer offsets can therefore enable the WAL even when messages use `Replicated`; [each operation still uses its own durability policy](https://github.com/apache/iggy/blob/691ad0f54e7eb83f85fd45303b0e7a7ba088f42a/core/partitions/src/iggy_partition.rs#L1279-L1286). Message segments and the prepare journal serve different purposes. Segments hold stored message data; the journal records ordered replication operations. Recovering one does not automatically establish the other. #4033 already fixed a concrete crash of the entire cluster path where a group with no journal elected on empty logs while flushed data survived on the previous primary but was never served. Omitting synchronous disk writes can be valid in VSR. It depends on recovery preventing a replica that forgot its history from participating until it has recovered sufficiently recent state. Loss of all volatile copies can exceed the durability guarantee. [Viewstamped Replication Revisited, section 4.3](https://www.cs.princeton.edu/courses/archive/fall19/cos418/papers/vr-revisited.pdf#page=6) explains these assumptions. The absence of a WAL alone does not demonstrate an implementation defect, and this issue does not prescribe adding one. ### Why purge exposed the question The [investigation on #4130](https://github.com/apache/iggy/pull/4130#issuecomment-5779531855) found a mismatch in a local partition regression: a durable purge cutoff survives, while the fixture reconstructs a partition with an empty operation sequence and no prepare WAL. A fresh operation numbered `1` can then be locally committed but hidden by the old cutoff. That fixture configures three replicas but directly drives one partition. It explicitly constructs the empty sequence; it does not run a real election, full server startup, or client acknowledgment flow. It therefore shows the effect of the mismatch, not that production recovery can legitimately reach that state. Restored replicas already have a [`ProbeAsBackup` path that excludes them from quorum participation while recovering](https://github.com/apache/iggy/blob/691ad0f54e7eb83f85fd45303b0e7a7ba088f42a/core/consensus/src/impls.rs#L1223-L1236). This leaves a concrete contract question: must recovery preserve or reconstruct the old operation history, or can some supported procedure establish a new one? If a new history is allowed, persisted cutoffs and other metadata must be interpreted consistently with that history. ### Decisions and validation 1. **Supported failures.** Define the expected result of one replica restarting, a rolling restart with completed recovery between failures, a complete cluster process restart, and power loss. Distinguish acknowledged data that existed only in volatile state from data whose persistence completed. Process death and power loss must not be treated as equivalent storage faults. 2. **Authority to resume.** Define what evidence a recovered replica needs before voting, acknowledging writes, or serving reads. Include how surviving segments, peer state, and durable metadata establish the chosen history. 3. **Insufficient history.** Define whether recovery must remain unavailable, require an explicit operator action, or use another recovery procedure when the required evidence is gone. If resetting history is supported, define how old replicas and metadata are prevented from being mistaken for the new history. The outcome should be a documented contract and executable scenarios for those decisions. Tests should reconstruct state through production startup and recovery, check quorum participation and message visibility, and attempt new writes after recovery. Storage fault tests should discard unsynchronized state explicitly. Any case intentionally unsupported should have a clear refusal or operator recovery procedure. The concrete purge cleanup and boundary handling remain in #4128 and #4130; this issue tracks the broader recovery contract and its validation. ### Related work - #4092 introduced the durability policies and conditional prepare WAL. - #4033 fixed recovery of existing partition directories after a crash of the entire cluster. - #4075 reports a specific restart repair failure that prevents later operations from committing. - #3793 reports journal repair resurrecting purged data. - #4139 tracks missing fault coverage for `Persisted` durability. - #4239 tracks connecting production purge and offset recovery to `SimStorage`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
