GitHub user chiradip added a comment to the discussion: Metadata layer in cluster.
Greg — I pushed through the tradeoffs and I think we should prefer segmented append-only logs + snapshot-triggered truncation over a bounded circular buffer for metadata. Reason: the circular ring requires strict timing guarantees (snapshot must finish before wrap) and a bunch of brittle guardrails (preemptive snapshot triggers, throttling, overflow spill, complex crash recovery). Those are doable but add a lot of correctness surface area. Segmented logs tolerate follower lag and crashes far more naturally, simplify recovery, and are easier to test/operate. If we accept slightly more IO/manifest work, we get drastically simpler correctness. We can still keep logical domain namespaces and modular snapshots (MuxStateMachine inside one VSR group) and can optimize later if real measurements demand it. Happy to sketch segment format + compaction/truncation rules if that helps. GitHub link: https://github.com/apache/iggy/discussions/2346#discussioncomment-14978734 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
