numinnex opened a new pull request, #4500: URL: https://github.com/apache/iggy/pull/4500
## Which issue does this PR address? Closes #4266 Closes #4476 Stacked on #4485, which adds the failing test this PR turns green. ## Rationale Under `replicated` durability a replica acknowledges a prepare once it is queued to the WAL writer, before it is written, so a replica that restarts can be missing ops it acknowledged. Restarted partition replicas came up `Recovering` and probed for the current view, but after 5 unanswered probes they elected on their on-disk log as if it were complete, and their DoViewChange looked like any other. After two crashes in quick succession the restarted replicas formed the view-change quorum without the only replica holding the committed ops, dropped them, and that replica later failed the WAL parent check and shut its whole node down (#4476). A single crash plus a partition could do the same through a trusted, lagging candidate. ## What changed? A partition replica that rejoins on a log recovered from disk under `replicated` durability marks its log untrusted and sets a new `FLAG_UNTRUSTED_LOG` bit in every DoViewChange it sends. Every candidate merges only trusted logs once it has a view-change quorum of them; otherwise it waits until every replica has reported and does not treat a shorter log as a nack. Any quorum of trusted replicas overlaps every commit quorum in a replica that never lost state, so a committed op always survives. An untrusted replica never becomes Normal, acknowledges nothing and accepts no writes until it adopts a settled view. A replica that cannot use its log (its partition is awaiting a state transfer) answers a StartViewChange with an empty, flagged DoViewChange. It counts toward every replica reporting but can never be chosen as the log or drop an op, so a full restart no longer halts on it. When a replica adopts a view that contradicts history it already applied, only that partition is quarantined to `.fenced.N` and rebuilt through state transfer at the offset frontier both histories agree on, instead of the node stopping on the WAL parent check. Partitions where both `durability` and `consumer_offset_durability` are `persisted`, and the metadata plane, keep the previous merge. The README now states what each durability policy survives and when a `replicated` partition halts. Observability: `untrusted_log` in replica log context, a rate-limited warning naming the replicas whose DoViewChange is awaited, a `partitions_untrusted_log` gauge and a `partition_applied_divergence_total` counter. Wire change: `DoViewChangeHeader` carves a `flags` byte out of `reserved` at offset 156. Every other offset is unchanged and unknown flag bits are rejected. An older replica can read a newer replica's empty vote as a short log, so every replica must be upgraded together, with the cluster down. Known limitations, documented in the README: - After a full-cluster restart under `replicated`, a replica that never returns keeps the partition unavailable; there is no operator override yet. - A replica that already joined a view while its state transfer is pending stays out of later elections until the transfer completes. - A divergence outside the headers carried by the view, or a WAL append parent-check failure, still stops the node. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
