This is an automated email from the ASF dual-hosted git repository. hubcio pushed a commit to branch feat/durability-docs in repository https://gitbox.apache.org/repos/asf/iggy-website.git
commit cb136ad7c4d0e9bbdd922e19b4e7e9f37c896759 Author: hubcio <[email protected]> AuthorDate: Thu Sep 3 21:44:19 2026 +0200 docs: document what an acknowledgement actually guarantees Nothing in the docs said what it means when the server accepts a write. Readers had to infer it from the flush knobs, and the obvious inference is wrong: the remainder that has not been flushed yet lives in the server process's heap rather than in the kernel's page cache, so a SIGKILL loses it where a page-cache design would not. Two pages rather than one, because the guarantee has a different source in each deployment. On a single node it comes from the filesystem, so that page carries the write path, the per-topic knobs and what they cost. In a cluster it comes from replication, so that page carries the quorum rules and the independence assumption the trade rests on. Both record the parts that are easy to get wrong. Flushes trigger on message count and bytes with no periodic timer, so a quiet topic can hold acknowledged data indefinitely. A graceful stop does force a final flush, so an orderly restart loses nothing. --- content/docs/clustering/durability.mdx | 103 ++++++++++++++++++++++ content/docs/clustering/meta.json | 2 +- content/docs/server/durability.mdx | 156 +++++++++++++++++++++++++++++++++ content/docs/server/meta.json | 2 +- 4 files changed, 261 insertions(+), 2 deletions(-) diff --git a/content/docs/clustering/durability.mdx b/content/docs/clustering/durability.mdx new file mode 100644 index 00000000..b737d53b --- /dev/null +++ b/content/docs/clustering/durability.mdx @@ -0,0 +1,103 @@ +--- +title: Cluster Durability +description: "What an acknowledgement means in a cluster, and when replication is a sound substitute for fsync." +--- + +> **This page is about durability in a clustered deployment**: what an +> acknowledgement actually guarantees, and when replication is a sound +> substitute for fsync. +> +> **For the single-node picture** -- the write path, the topic options that +> control flushing, and what they cost -- start with +> [Durability](/docs/server/durability). + +## What an acknowledgement means + +Iggy replicates with [Viewstamped Replication](/docs/clustering/vsr). An +operation is committed once a replication quorum has acknowledged it: + +| Replicas | Must acknowledge before commit | +| --- | --- | +| 2 | 2 | +| 3 | 2 | +| 5 | 3 | + +The rule is `ceil(n / 2)`, with two replicas special-cased to 2 so that a +two-replica cluster stays durable, and a cap on very wide clusters. The +view-change quorum grows to compensate, so a replication quorum and a +view-change quorum always intersect. + +The important consequence: **with default topic options, an acknowledgement +means the operation is in a quorum's memory, not on any node's disk.** +Replication is providing the durability, not the filesystem. + +## Relying on replication instead of fsync + +For a cluster this is a reasonable trade, and it is what most replicated log +systems do. Leaving `enforce_fsync = false` and `messages_required_to_save` at +its default keeps the throughput that fsync costs, and accepts that durability +comes from holding the data in several places rather than from having it on one +node's platters. + +The reasoning holds as long as failures are **independent**. A single node +losing power is fully covered. The surviving replicas hold the data, and the +failed node rebuilds from them when it returns. + +## Where that reasoning breaks + +The assumption that carries this trade is independence, not node count. Two +properties of Iggy are worth knowing before relying on it. + +**The partition journal is in memory and is not replayed at boot.** A restarting +replica does not recover its own log, so all of its recovery is peer-driven. + +This is orderly rather than dangerous: it is what Viewstamped Replication +prescribes. The state a replica must not forget across a restart, its view and +log view, is kept in a durable superblock, while the log itself is volatile and +refetched. A booting replica sits in a recovering status where it acknowledges +nothing, votes in no election, and initiates nothing, until it has probed for +the current view and had its log refilled by state transfer. An empty replica +does not quietly rejoin as a voting member and cannot win an election on an +empty log. + +What it does mean is that the guarantee rests entirely on peers being available +and holding the data. That is fine when they are. It is not fine when they are +not, and it is the reason simultaneous failure of a quorum sits outside the +protocol's model rather than inside it. + +**Segment rotation is offset-keyed, so every replica seals at the same offset.** +This is deliberate: it keeps offset-keyed segment cleanup converging across the +cluster. The side effect is that under a simultaneous power loss every replica is +sitting at a similar point in its segment lifecycle rather than at independent +ones. + +Correlated failure is not exotic. A rack losing power, a shared storage fault, a +bad rollout, an out-of-memory cascade, or a hypervisor failure all take several +nodes at once, and none of them are covered by the observation that two +*independent* machines rarely fail at the same moment. + +Note also that the exposure is not limited to machine-level events. Because the +unflushed remainder lives in the server process's memory rather than in the +kernel's page cache, a correlated *process* failure is enough. An out-of-memory +cascade, an eviction storm, or a panic that every replica hits on the same input +will take the unflushed data with it while every machine stays up. See +[Durability](/docs/server/durability) for what does and does not survive a +process-level crash. + +## Guidance + +**Cluster, throughput-oriented.** Leave `enforce_fsync = false` and the default +`messages_required_to_save`, and rely on replication. Spend the effort on making +failures genuinely independent instead: replicas should not share a power +domain, a hypervisor, or a storage backend, because that independence is what +the trade actually rests on. See [Deployment](/docs/clustering/deploy). + +**Cluster, cannot lose acknowledged writes even to a site-wide event.** Set +`enforce_fsync = true` and `messages_required_to_save = 1` on the topics that +matter. These are per-topic options, so this can be scoped to the few topics +that need it rather than paid across every topic on the cluster. + +**Single node.** Replication is not available as a fallback and the in-memory +journal is not replayed at boot, so any data worth keeping needs +`enforce_fsync = true` and `messages_required_to_save = 1`. See +[Durability](/docs/server/durability). diff --git a/content/docs/clustering/meta.json b/content/docs/clustering/meta.json index 9d6ae8e6..fd86ff6a 100644 --- a/content/docs/clustering/meta.json +++ b/content/docs/clustering/meta.json @@ -1,4 +1,4 @@ { "title": "Clustering", - "pages": ["vsr", "deploy", "configuration", "security", "client-failover"] + "pages": ["vsr", "durability", "deploy", "configuration", "security", "client-failover"] } diff --git a/content/docs/server/durability.mdx b/content/docs/server/durability.mdx new file mode 100644 index 00000000..1c8ca8c0 --- /dev/null +++ b/content/docs/server/durability.mdx @@ -0,0 +1,156 @@ +--- +title: Durability +description: "What survives a crash on a single node, which topic options control it, and what they cost." +--- + +> **This page is about single-node durability**: what survives a crash on one +> machine, which topic options control it, and what they cost. It assumes +> Linux, since Iggy's storage path is built around `io_uring` and the +> behaviour described here is specific to Linux filesystems. +> +> **In a cluster the guarantee comes from replication rather than from the +> filesystem**, and the trade-offs are different. See +> [Cluster Durability](/docs/clustering/durability). + +As of now Iggy does not support `O_DIRECT`. Durability is guaranteed by the +combination of `messages_required_to_save = 1` and `enforce_fsync = true`. +Direct I/O plus `O_DSYNC` is on the roadmap; that pairing would make a write +durable on completion of the write itself, removing the need for a separate +`fdatasync` on the hot path. + +## The write path + +```text +produce -> consensus commit -> ACK to client -> flush to file -> fdatasync + | | | + returns here page cache durable + (kernel) +``` + +Three facts follow from this ordering, and they are the whole story: + +1. **A write is acknowledged when it commits, and the commit precedes the + flush** unless that commit trips a flush threshold. +2. **The partition journal holding the unflushed remainder is in memory and is + not replayed at boot.** A restarting node cannot replay its own journal to + recover the unflushed tail. +3. Therefore **acks are durability-gated only at + `messages_required_to_save = 1`**, and segment `.log` files are the sole + durable substrate for message payloads on a node. + +## Where the knobs live + +The durability knobs are **per topic, set at `CreateTopic`**. They are not server +configuration keys, and placing them in `config.toml` makes the server refuse to +start with `Config(InvalidConfigurationValue)`. See +[Topic Options](/docs/server/topic-options). + +| Option | Default | Effect | +| --- | --- | --- | +| `enforce_fsync` | `false` | `fdatasync` the `.log` and `.index` on each flush | +| `messages_required_to_save` | `1024` | Flush after this many messages | +| `size_of_messages_required_to_save` | 1 MiB | Flush after this many bytes | +| `segment_size` | 1 GiB | Soft limit; a segment may close one batch past it | +| `preallocate_segments` | `false` | Reserve the whole segment up front | + +`enforce_fsync` decides whether a flush *syncs*, never whether a flush +*happens*. The two `required_to_save` thresholds decide when flushes happen, and +whichever trips first wins. + +The `.log` and `.index` fdatasyncs are issued concurrently, so the two round +trips overlap rather than running one after the other. + +One related server-wide key: `system.partition.validate_checksum` (default +`true`) re-hashes each batch on read. Turning it off serves whatever decodes, +which can hand a consumer bytes that are provably not the ones that were +written. See [Configuration](/docs/server/configuration). + +## What survives what + +Acknowledged messages sit in one of two places, and they behave differently, so +the answer depends on which. Data that a flush has already written to the file +is in the kernel's page cache. Data acknowledged since the last flush is in the +server process's own memory. + +| Failure | Already flushed, no fsync | Not yet flushed | `mrts = 1` and `enforce_fsync` | +| --- | --- | --- | --- | +| Graceful shutdown (`SIGTERM`, Ctrl-C) | Survives | Survives | Survives | +| Process crash, `SIGKILL`, panic, OOM kill | Survives | **Lost** | Survives | +| Machine crash, power loss, kernel panic | **Lost** | **Lost** | Survives | + +For flushed data the distinction is the page cache. A process-level crash leaves +the kernel running and the page cache intact, so anything already written to the +file survives even without an fsync. Only a machine-level failure loses page +cache contents, and that is the case fsync exists for. + +The unflushed remainder behaves differently. It has not reached a file +descriptor at all, so the page cache argument does not apply to it. It lives in +the server process's heap and dies with the process. Systems that write every +record into the file immediately, and let the kernel decide when to write back, +pay nothing for a `SIGKILL`. Here it costs everything acknowledged since the +last flush. + +Graceful shutdown is handled. On `SIGTERM` or Ctrl-C the server stops accepting +work, answers in-flight requests, and then forces a flush of every partition +past the thresholds before exiting. If that final flush fails the server exits +non-zero rather than reporting a clean stop. So an orderly restart, a +`docker stop`, or a Kubernetes pod termination loses nothing. + +If the drive has a volatile write cache, `enforce_fsync` issues a real device +flush rather than merely a kernel writeback. + +### Flushes are triggered by volume, never by time + +There is no periodic flush. Both thresholds count messages and bytes, and +nothing flushes a partition simply because time has passed. A busy topic +therefore has a short exposure window, while a quiet one can hold acknowledged +messages in memory indefinitely, waiting for a 1024th message that may be hours +away. + +This is worth planning around for low-traffic topics that still matter. Sizing +`messages_required_to_save` and `size_of_messages_required_to_save` to the +topic's actual rate bounds the window. Setting `messages_required_to_save` to 1 +removes it. + +## The durable configuration + +For genuine single-node durability you need both options together: + +``` +enforce_fsync = true +messages_required_to_save = 1 +``` + +`enforce_fsync` on its own is not enough. Without `messages_required_to_save = 1` +the ack still precedes the flush, so a crash loses everything acknowledged since +the last threshold trip, up to 1024 messages or 1 MiB, whichever came first. +Note that this bound is on the *volume* of unflushed data, not on its age: with +no periodic flush, the data can be arbitrarily old. + +### What it costs + +Measured on ext4 with an NVMe drive, 1000-byte messages, one producer: + +| Configuration | Throughput | +| --- | --- | +| `enforce_fsync = false`, `messages_required_to_save = 1024` | 3544 MB/s | +| `enforce_fsync = true`, `messages_required_to_save = 1` | 511 MB/s | + +Roughly a sevenfold reduction. This measures the two options together; the +individual contribution of each has not been isolated. Treat it as an order of +magnitude for planning, not as a number to quote, since it depends heavily on +the drive and filesystem. See [Benchmarking](/docs/server/benchmarking) for how +to measure it on your own hardware. + +Because these are per-topic options, the cost can be scoped to the topics that +actually need durability rather than paid across the whole server. + +## Recovery + +On restart, recovery walks each segment's log, stops at the first batch that +fails to decode, and truncates the file at that point. Every batch a walk +accepts passes its own checksum. + +A torn or short `.index` is not a problem. It is rebuilt from the log and +installed by an atomic rename, so only the `.log` file determines what is +recoverable. diff --git a/content/docs/server/meta.json b/content/docs/server/meta.json index 4c9c3bd6..08e1f2f0 100644 --- a/content/docs/server/meta.json +++ b/content/docs/server/meta.json @@ -1,4 +1,4 @@ { "title": "Server", - "pages": ["introduction", "configuration", "topic-options", "storage-engine", "networking", "security", "docker", "benchmarking"] + "pages": ["introduction", "configuration", "topic-options", "storage-engine", "durability", "networking", "security", "docker", "benchmarking"] }
