krishvishal opened a new pull request, #4073: URL: https://github.com/apache/iggy/pull/4073
## The defect A replica could be promoted to primary while missing operations that the cluster had already committed. The promotion checks started at the merged commit point, so they did not check for missing operations between the replica’s `commit_min` and that point. For example, a replica might have executed operations 1–6 and received operations 8–10, but never received operation 7. If the cluster had committed through operation 10, the promotion scan started at 10 and missed the gap. After promotion, the replica could serve reads and compute replies from incomplete state. It could also attempt to advance `commit_min` past operations it had not executed, triggering an assertion. ## Fixes **Promotion coverage (`shard`).** The coverage scan, repair requests, and repair retries now start at the lower of the merged commit point and `commit_min + 1`. This ensures that promotion checks include missing committed operations and that repair requests cover the same range. Replicas with a contiguous committed prefix keep the existing scan range. **Repair progress (`shard`).** Repair sessions could remain active after their requested range was satisfied, or stop making progress when the selected peer crashed. Restarting repair required another commit, which the incomplete repair could itself prevent. Sessions now clear when their range is satisfied, switch peers after a full retry interval without a response, and restart whenever repair is still needed. **Recovery barrier (`consensus`).** At startup, a replica records the recovered journal head as the point it must commit before accepting client requests as primary. A later view change could discard part of that journal without updating the barrier. If the replica then became primary, it would wait for discarded operations to commit and reject the requests needed to make progress. The barrier is now reassessed when the merged log is finalized at view start or a `StartView` log is adopted. It is lowered if the suffix was truncated and retained if the suffix survived. **Commit ordering (`consensus`, both planes).** The commit pipeline now processes only consecutive operations starting at `commit_min + 1`. Previously, it could pass a later operation to `advance_commit_min` even though an earlier prepare was missing. A debug assertion detects this in simulation and CI. Release builds report the problem and pause commit advancement until repair fills the gap. This avoids terminating the shard task while the process continues to report itself as healthy. **Consumer offset deletion (`partitions`).** Applying a committed `DeleteConsumerOffset` now logs and succeeds when the offset is already absent. An absent offset is valid: `AckLevel::NoAck` stores apply only on the primary, so a follower may never have received the offset, and a restart can also lose it. The previous check depended on the replica’s role when the delete was committed, which did not establish whether it had received the earlier store. Treating absence as an error could therefore fence a partition during a valid committed delete. The apply function no longer returns `Result`. ## Simulator The quiescence checks previously compared partition commit positions but did not compare the committed operations themselves. Content comparisons covered only metadata, where partition-focused runs often committed one operation or none. The simulator now compares partition journal contents per namespace over the range retained by both replicas. Partition journals evict committed entries as they flush, so the check allows entries that have already been removed. It compares `identity_checksum`, because retransmission can change the prepare’s view and therefore its sealed checksum. The recovery-barrier failure did not reproduce in 340 seeds across two fault profiles. A new check fails a run if a metadata primary remains in `Normal` status below its recovery barrier for 2000 ticks. The simulator also reports a fenced partition task explicitly. Previously, queued frames left by an exited task could be reported as a missed wake. The missed-wake check remains in place for actual scheduling failures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
