krishvishal opened a new pull request, #3767:
URL: https://github.com/apache/iggy/pull/3767
A metadata replica kept its consensus numbers only in RAM. After a crash it
rebuilt at `view = 0` and inferred a view from the last WAL prepare, so it
could
re-enter a superseded view and re-vote, splitting the log. Sessions and the
at-most-once dedup cache were volatile too, so a returning client hit
`NoSession` and a retried request re-executed. And once a checkpoint
drained the
WAL prefix, nothing durably identified the snapshot it folded into.
## What
**`journal::superblock`** (new). `SuperblockStore` trait plus
`PingPongSuperblock`:
two files written alternately, each replaced `temp, fsync, rename,
dir-fsync`,
highest valid `sequence` wins. Rename is atomic, so an update can never
destroy
the last good record. `read_latest` returns three outcomes, kept distinct
because
collapsing them into an `Option` is exactly what lets a lost superblock
masquerade as a fresh deployment:
| Outcome | Meaning | Response |
| --- | --- | --- |
| `Empty` | never written | fresh boot |
| `Present(payload)` | newest checksum-clean record | recover from it |
| `Unreadable { version }` | written, now torn / foreign / unknown version
| refuse boot |
**`consensus::VsrState`** (new). The 58-byte little-endian payload:
cluster,
replica id and count, `view`, `log_view`, `commit_max`, and the paired
`checkpoint_op` / `checkpoint_checksum`. Offsets pinned by test.
**Durable-before-send gate.** `VsrConsensus` tracks `view_durable` /
`log_view_durable`, exposing `needs_superblock_persist()` and
`mark_superblock_durable(view, log_view)`, which takes the values written
rather
than re-reading `self.view` (the in-memory view can advance across the
write's
await while a checkpoint holds the lock). Every metadata dispatch site
(`SVC` /
`DVC` / `StartView` / `RequestStartView`, the tick, the stale-heartbeat
reply,
`send_prepare_ok`) persists first and withholds on failure. A
`debug_assertions`
tripwire in `dispatch_vsr_actions` asserts callers did so.
`RequestStartView` is
exempt: it is a probe asking to *learn* the view.
**Three-phase checkpoint.** Persist snapshot, record the pairing, then
drain the
WAL. A crash before the pairing write recovers the prior checkpoint with
the WAL
intact; after it, the new one. One `LocalGate` serializes checkpoints
against
view-change persists, since both drive the single superblock, whose
`write` picks
its slot before it awaits, and in-process submits each run on their own
task.
**Client-table durability.** `ClientTableSnapshot` folds into
`MetadataSnapshot`
under `#[serde(default)]`, preserving slot positions so eviction order
survives.
Boot restores it from the snapshot and replays the committed suffix
through the
same `apply_committed_prepare` the live commit walk uses.
**Verify-then-decode recovery.** Recovery cross-checks the superblock
against the
on-disk snapshot before trusting its contents. New typed refusals:
superblock
ahead of snapshot, checksum mismatch at the same op, undecodable payload,
unreadable on every slot. A superblock *lagging* a newer
atomically-complete
snapshot is accepted (`commit_max` is only a recovery lower bound). An
absent
superblock stays a fresh boot; refusing there would brick healthy nodes.
**WAL body integrity.** The primary seals `checksum_body` once and
replicates it
verbatim. The scan truncates a genuinely torn tail but refuses boot on
interior
bit-rot rather than discarding the committed entries behind it.
**Incarnation nonce.** `RequestStartView` carries a per-boot `u128`; the
answering
`StartView` echoes it. While `Recovering`, a replica only adopts a
`StartView`
provably post-restart.
## Wire compatibility
`StartViewHeader` and `RequestStartViewHeader` gain `incarnation: u128`,
carved
from the tail of their existing `reserved` regions. Placed **last** and
16-aligned, so `op` / `commit` / `namespace` keep their offsets and the
structs
stay padding-free (static asserts enforce both). A peer that predates the
field
sends zeros, decoding as `incarnation == 0`, inert per the adoption guard.
Header
size unchanged, so a mixed-version rolling upgrade is safe.
## Determinism invariant
Recovery trusts a checkpoint by recomputing `hash(encode(decode(disk)))`
against
the persist-time `hash(encode(state))`. That holds only while every
collection in
the serialized form keeps a deterministic order, so an unordered map
anywhere in
`MetadataSnapshot` would make a *healthy* node refuse boot after a restart.
Documented on the three snapshot types and guarded by two byte-stability
tests.
## Not in scope
- Partition-plane consensus is not superblock-backed and is exempt from
the gate.
- No metadata state transfer, so a corrupt checkpoint refuses boot rather
than
repairing from a peer. Marked `TODO(state-transfer)` where the fetch
belongs.
- Header `checksum` and its `parent` chain stay unsealed: activating them
needs
the retransmit path to re-seal a re-stamped header.
- `checksum_body` has no format-version gate, so a WAL written before this
change
refuses boot. Fail-safe, but a hard upgrade break; `TODO(wal-integrity)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
