This is an automated email from the ASF dual-hosted git repository. hubcio pushed a commit to branch docs/align-with-topic-durability in repository https://gitbox.apache.org/repos/asf/iggy-website.git
commit 4f05859cb81939ed969543f3d74638728207bd7d Author: hubcio <[email protected]> AuthorDate: Fri Sep 11 00:16:49 2026 +0200 fix(docs): align clustering with server 0.9.0 --- content/docs/clustering/client-failover.mdx | 18 ++++++++++-------- content/docs/clustering/configuration.mdx | 28 +++++++++++++++++++++------- content/docs/clustering/deploy.mdx | 21 +++++++++++++++------ content/docs/clustering/durability.mdx | 29 ++++++++++++++++------------- content/docs/clustering/security.mdx | 6 ++++-- content/docs/clustering/vsr.mdx | 8 ++++---- 6 files changed, 70 insertions(+), 40 deletions(-) diff --git a/content/docs/clustering/client-failover.mdx b/content/docs/clustering/client-failover.mdx index ebe9de48..ea86024e 100644 --- a/content/docs/clustering/client-failover.mdx +++ b/content/docs/clustering/client-failover.mdx @@ -7,28 +7,30 @@ A clustered deployment changes what a client can expect from any single node: re ## Where requests are served -**Reads** are served from the local replicated state on whichever node owns the namespace locally, followers included. A client connected to a follower can poll messages and read metadata without touching the primary. +**Reads** use the local replicated state by default on whichever node owns the namespace locally, followers included. A client connected to a follower can poll messages and read metadata without touching the primary. -**Writes** go through consensus and are admitted on the plane's primary *only*: metadata operations on the metadata primary, message and offset writes on the primary of that partition's consensus group. A write returns after the required VSR commit, or returns a retryable error when the replica is changing view or catching up. +**Writes** go through consensus and are admitted on the plane's primary *only*: metadata operations on the metadata primary, message and offset writes on the primary of that partition's consensus group. An acknowledged write returns after the required VSR commit, or returns a retryable error when the replica is changing view or catching up. HTTP `ack=none` returns after dispatch instead. -One caveat on follower reads: server-managed offset auto-commit replicates the polled offset through the partition consensus, which only the partition primary may do. A poll served by a follower **does not advance the durable consumer offset**. Auto-commit is best-effort *at-least-once* delivery either way, so this widens the redelivery window after failover rather than losing data. +One caveat on follower reads: server-managed offset auto-commit replicates the polled offset through the partition consensus, which only the partition primary may do. A poll served by a follower **does not advance the durable consumer offset**. Auto-commit is best-effort and the poll response does not await offset persistence. On a primary, the offset submission precedes delivery of the poll reply, so it does not guarantee that the application received or processed the batch. Store offse [...] ## Leader redirection -Leader-aware SDKs find the primary instead of requiring you to point them at it: +Leader-aware SDKs discover the metadata primary. Rust binary clients follow this sequence: -1. The client connects to any node from its connection string and fetches cluster metadata, which lists every node with its name, client endpoints, role, and status. +1. The client connects and authenticates to its configured node, then fetches cluster metadata, which lists every node with its name, client endpoints, role, and status. 2. When the connected node isn't the leader, the client reconnects to the leader's advertised endpoint for the transport in use. -3. When the cluster is transiently leaderless (for roughly one heartbeat timeout after a primary fails, while the election completes), the client polls the metadata until a leader appears instead of failing. +3. When the cluster is transiently leaderless, the client polls metadata every 250ms for up to 5s. If no healthy leader appears within that window, it continues on the contacted node and lets subsequent requests report their outcome. 4. Redirects are **capped** (three in the Rust SDK) so a flapping roster can't bounce the client forever. The address a client is redirected to is the node's `advertised_address` (or the matching `advertised_addresses` selector), falling back to the roster `ip`. Getting those right matters: the client redials **exactly what the metadata advertises**. See [Configuration](/docs/clustering/configuration) for the selector rules. +Metadata and partition primaries can differ. After a `TransientNotAccepted` response, Rust binary clients check the metadata leader and can then try the remaining advertised endpoints in a bounded roster walk. This fallback is separate from the three-redirect limit. + The Rust implementation lives in [`core/sdk/src/leader_aware.rs`](https://github.com/apache/iggy/blob/master/core/sdk/src/leader_aware.rs). ## Retryable errors -During a view change or while a replica catches up, writes fail with transient errors rather than definitive ones. The SDKs replay those transparently, so a primary failover surfaces as **added latency rather than an error**, as long as it completes within the client's retry budget. Definitive errors (validation failures, permission denials) are **never replayed**. +During a view change or while a replica catches up, writes can return transient errors. Rust binary clients retry within their request budgets. `TransientNotAccepted` proves that the request was never admitted and permits retry on another node. `TransientNotCommitted` has an uncertain outcome and is replayed with the same session and request identity, then returned if the budget expires. Disconnects and exhausted routing attempts can still reach the caller. Definitive validation and perm [...] ## SDK support @@ -50,4 +52,4 @@ Run the suite for one SDK with: ./scripts/run-bdd-tests.sh rust leader_redirection ``` -For SDKs without redirection coverage, connect clients to the current primary for write-heavy workloads, or front the cluster's HTTP transport with a load balancer and rely on [follower HTTP forwarding](/docs/clustering/security) for control-plane operations. +For SDKs without redirection coverage, connect clients to the current primary for write-heavy workloads, or front the cluster's HTTP transport with a load balancer and rely on [follower HTTP forwarding](/docs/clustering/security) for control-plane operations and acknowledged partition writes. diff --git a/content/docs/clustering/configuration.mdx b/content/docs/clustering/configuration.mdx index 1bb5acb0..aea59257 100644 --- a/content/docs/clustering/configuration.mdx +++ b/content/docs/clustering/configuration.mdx @@ -5,7 +5,7 @@ description: Reference for the [cluster] settings, consensus timing, and node ro All clustering settings live in the `[cluster]` section of the server configuration. The authoritative defaults with detailed comments are in [`core/server/config.toml`](https://github.com/apache/iggy/blob/master/core/server/config.toml). Every value can also be set through an `IGGY_`-prefixed environment variable, for example `IGGY_CLUSTER_ENABLED=true` or `IGGY_CLUSTER_NODES_0_IP=10.0.1.5`. -The server validates the whole section at boot and **refuses to start** on an invalid value, printing the exact rule that failed. +With clustering enabled, the server validates the section at boot and **refuses to start** on an invalid value, printing the rule that failed. The repair chunk limit and superblock failure timeout are validated even when clustering is disabled. ## Top-level settings @@ -18,7 +18,7 @@ The server validates the whole section at boot and **refuses to start** on an in ## Consensus timing -Consensus runs on a **fixed 10ms tick**. Every duration below is converted to whole ticks: values are **rounded down** to a multiple of 10ms, and anything under 10ms is raised to a single tick rather than firing sooner. +Consensus runs on a **fixed 10ms tick**. Every duration below is converted to whole ticks: values are **rounded down** to a multiple of 10ms, and anything under 10ms is raised to a single tick rather than firing sooner. `repair_gap_debounce_interval` has an additional 500ms floor. | Setting | Default | Purpose | | --- | --- | --- | @@ -30,6 +30,7 @@ Consensus runs on a **fixed 10ms tick**. Every duration below is converted to wh | `request_start_view_retransmit_interval` | `1s` | How often a recovering replica re-requests the current view's `StartView` | | `view_probe_attempts_max` | `5` | Unanswered `RequestStartView` probes a recovering replica tolerates before falling back to an election | | `repair_retry_interval` | `1s` | How long a stalled journal-repair stream waits before re-requesting its remaining window | +| `repair_gap_debounce_interval` | `1s` | How long a committed-history gap waits before opening a repair session; floored at 500ms | | `repair_chunk_max` | `128` | Prepares a peer serves per repair round | Raise `heartbeat_timeout` on oversubscribed hosts where scheduling stalls fake primary death. Otherwise keep the defaults unless tests show a specific scheduling or network problem. @@ -44,9 +45,22 @@ Boot-time validation enforces these constraints: - `view_probe_attempts_max` must be between 1 and 100. - `repair_chunk_max` must be between 1 and 1024, and strictly below `message_bus.peer_queue_capacity`: each repair frame rides the per-peer message-bus queue, and a full round above that capacity would overrun the queue and drop frames. +## Superblock failure handling + +`superblock_wedged_fatal_timeout` defaults to `2m`. It controls how long repeated superblock persistence failures can continue before the process exits. The affected consensus group stays fenced while persistence is failing. `0`, `disabled`, or `unlimited` disables the process exit, leaving the group fenced; a nonzero value must be at least `30s`. + +## Coordinator placement + +These settings live under `[cluster.coordinator]` and apply when the server has more than one shard: + +| Setting | Default | Purpose | +| --- | --- | --- | +| `skip_shard_zero_for_replicas` | `true` | Places replica connections on peer shards instead of shard 0 | +| `skip_shard_zero_for_clients` | `false` | When enabled, excludes shard 0 from client connection placement | + ## Node roster -`cluster.nodes` is the full roster of cluster members and is intended to be **byte-identical on every node**. The running node finds its own entry through the `--replica-id` CLI flag. +`cluster.nodes` is the full roster of cluster members and is intended to be **byte-identical on every node**. The running node finds its own entry through the `--replica-id` CLI flag. Environment variables replace the entire `cluster.nodes` array rather than merging individual fields into TOML entries. If you configure any roster entry through environment variables, provide the complete roster that way. ```toml [[cluster.nodes]] @@ -63,7 +77,7 @@ Node name, unique within the roster and non-empty. ### `ip` -The node's roster address. Replica-to-replica consensus traffic and follower-to-primary HTTP forwarding dial it, so it must be a **literal IPv4 or IPv6 address**. Hostnames are rejected when the server resolves the roster at startup. +The node's roster address. Replica-to-replica consensus traffic and follower-to-primary HTTP forwarding dial it, so it must be a **literal IPv4 or IPv6 address**. Hostnames and unspecified addresses (`0.0.0.0` or `::`) are rejected when the server resolves the roster at startup. `ip` is **not the bind interface** for the client transports. `tcp`, `quic`, `http`, and `websocket` bind whatever their own `address` settings say. The roster supplies only their ports. The defaults bind loopback, so a cluster spread across hosts needs each transport's `address` set to `0.0.0.0` or the routable interface. The server warns at startup when a bind cannot serve the advertised `ip`. @@ -73,7 +87,7 @@ Numeric replica id for VSR consensus, `0`-based. Ids must be unique and strictly ### `ports` -Per-node listener ports: `tcp`, `quic`, `http`, `websocket`, and `tcp_replica`. In cluster mode `ports` is the **single source of listener ports**: every enabled transport needs an explicit per-node port, otherwise the server refuses to start. `tcp_replica` carries consensus traffic and is always required. Port `0` is rejected, and no two roster entries may claim the same `ip:port`. +Per-node listener ports: `tcp`, `quic`, `http`, `websocket`, and `tcp_replica`. In cluster mode `ports` is the **single source of listener ports**: every enabled transport needs an explicit per-node port, otherwise the server refuses to start. `tcp` and `tcp_replica` are always required, even when client TCP is disabled. `tcp_replica` carries consensus traffic. Port `0` is rejected, and no two roster entries may claim the same `ip:port`. ### `advertised_address` @@ -89,7 +103,7 @@ name = "iggy-node-1" ip = "10.0.1.5" # replica plane + last-resort fallback advertised_address = "203.0.113.10" # catch-all for unmatched clients replica_id = 0 -ports = { tcp = 8090, http = 3000, tcp_replica = 9090 } +ports = { tcp = 8090, quic = 8080, http = 3000, websocket = 8093, tcp_replica = 9090 } [[cluster.nodes.advertised_addresses]] client_cidr = "10.0.0.0/16" # in-VPC clients stay private @@ -105,7 +119,7 @@ Rules and caveats: - Matching sees the transport-level peer address, so clients behind a proxy or load balancer match the proxy's network, not their own. - Every `address` must be routable from inside its own `client_cidr`: leader-aware clients redial whatever address metadata advertises, and a selector pointing at an unreachable host strands them mid-redirect. - Prefer literal IPs over hostnames. SDKs differ in how they compare an advertised hostname against the address they dialed, and a mismatch costs a reconnect on every fresh connect. -- Rolling upgrades: older server binaries reject a TOML config containing `advertised_addresses` but **silently ignore** the equivalent `IGGY_CLUSTER_NODES_*_ADVERTISED_ADDRESSES_*` env vars. Upgrade every binary first, then add selectors. +- Rolling upgrades: VSR builds predating selector support reject a TOML config containing `advertised_addresses`. Their release builds warn and ignore the equivalent `IGGY_CLUSTER_NODES_*_ADVERTISED_ADDRESSES_*` env vars; debug builds can panic on them. Upgrade every binary first, then add selectors. ## Authentication and TLS diff --git a/content/docs/clustering/deploy.mdx b/content/docs/clustering/deploy.mdx index 72d0d8c1..2d7afeb7 100644 --- a/content/docs/clustering/deploy.mdx +++ b/content/docs/clustering/deploy.mdx @@ -30,7 +30,7 @@ The default `core/server/config.toml` has clustering disabled, so a plain run st cargo run --bin iggy-server -- --with-default-root-credentials ``` -`--with-default-root-credentials` sets the root user to `iggy`/`iggy` unless `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` are already exported. **Only the first boot** on an empty data directory reads these values. +`--with-default-root-credentials` sets the root user to `iggy`/`iggy` unless `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` are already exported. These values initialize the root user **only on its first creation**. Later starts recover the stored user; supplied environment values are still validated at boot. When clustering is disabled, `--replica-id 0` is still accepted. Any other id is rejected because it would have to match a `cluster.nodes` entry. @@ -79,7 +79,7 @@ export IGGY_ROOT_PASSWORD=iggy export IGGY_CLUSTER_AUTH_SHARED_SECRET="replace-with-at-least-32-random-bytes" ``` -Root credentials are **mandatory** once the cluster is enabled. The shared secret must be **at least 32 bytes**. See [Security](/docs/clustering/security) for how it's used and rotated. +Root credentials are **mandatory on the first boot of each replica** when clustering is enabled. Restarts recover the stored root user. The shared secret must be **at least 32 bytes**. See [Security](/docs/clustering/security) for how it's used and rotated. Start each replica in a separate terminal. Use a different data path for every process: @@ -101,7 +101,7 @@ In cluster mode: - `--replica-id` is required and must match exactly one `cluster.nodes` entry - `replica_id` values must be unique and contiguous from `0` - `ports` is the single source of listener ports: every enabled transport needs an explicit per-node port, otherwise the server **refuses to start** -- `tcp_replica` carries replica-to-replica consensus traffic and is **always required** +- `tcp` and `tcp_replica` ports are **always required**, even when client TCP is disabled; `tcp_replica` carries replica-to-replica consensus traffic - `ip` must be a **literal IP address**. Use `advertised_address` when clients can't reach it (see [Configuration](/docs/clustering/configuration)) - use a different root `path` for each process on the same host @@ -126,7 +126,16 @@ IGGY_CLUSTER_NODES_0_PORTS_WEBSOCKET=8093 IGGY_CLUSTER_NODES_0_PORTS_TCP_REPLICA=9090 IGGY_CLUSTER_NODES_1_NAME=node-2 IGGY_CLUSTER_NODES_1_IP=172.28.0.102 -# ... and so on for every node and every enabled transport +IGGY_CLUSTER_NODES_1_REPLICA_ID=1 +IGGY_CLUSTER_NODES_1_PORTS_TCP=8090 +IGGY_CLUSTER_NODES_1_PORTS_QUIC=8080 +IGGY_CLUSTER_NODES_1_PORTS_HTTP=3000 +IGGY_CLUSTER_NODES_1_PORTS_WEBSOCKET=8093 +IGGY_CLUSTER_NODES_1_PORTS_TCP_REPLICA=9090 +IGGY_TCP_ADDRESS=0.0.0.0:8090 +IGGY_QUIC_ADDRESS=0.0.0.0:8080 +IGGY_HTTP_ADDRESS=0.0.0.0:3000 +IGGY_WEBSOCKET_ADDRESS=0.0.0.0:8093 IGGY_CLUSTER_AUTH_ENABLED=true IGGY_CLUSTER_AUTH_SHARED_SECRET=replace-with-at-least-32-random-bytes IGGY_ROOT_USERNAME=iggy @@ -155,7 +164,7 @@ The response lists the cluster name, nodes, client endpoints, and current roles. ## Run the cluster test suites -The cross-SDK BDD suites run against a real cluster started from `bdd/docker-compose.cluster.yml`: +The cross-SDK BDD suites use a standalone server for basic features and add the two-node cluster from `bdd/docker-compose.cluster.yml` for `leader_redirection` or `all`: ```bash ./scripts/run-bdd-tests.sh rust @@ -163,4 +172,4 @@ The cross-SDK BDD suites run against a real cluster started from `bdd/docker-com ./scripts/run-bdd-tests.sh rust leader_redirection ``` -The first argument selects the SDK (`rust`, `python`, `php`, `go`, `go-race`, `node`, `csharp`, `java`, `cpp`, or `all`) and the optional second argument selects one feature (`basic_messaging`, `leader_redirection`, `raw_command`, or `all`). `leader_redirection` is the clustered feature. The Rust, Go, C#, and Java suites support it. +The first argument selects the SDK (`rust`, `python`, `php`, `go`, `go-race`, `node`, `csharp`, `java`, `cpp`, or `all`) and the optional second argument selects one feature (`basic_messaging`, `leader_redirection`, `raw_command`, `stream_crud`, or `all`). `leader_redirection` is the clustered feature. The Rust, Go, C#, and Java suites support it. diff --git a/content/docs/clustering/durability.mdx b/content/docs/clustering/durability.mdx index ec1c261e..479df2ed 100644 --- a/content/docs/clustering/durability.mdx +++ b/content/docs/clustering/durability.mdx @@ -39,7 +39,7 @@ Iggy's replication quorum is not a strict majority for every group size: Except for two replicas, the replication quorum is `min(ceil(n / 2), 3)` and the view-change quorum is `n - replication_quorum + 1`. Two replicas require both for either quorum. The quorums therefore intersect. These are -the [implemented quorum rules](https://github.com/apache/iggy/blob/97f7b0c0335f81691bef34f923f21d783016a6a4/core/consensus/src/impls.rs), +the [implemented quorum rules](https://github.com/apache/iggy/blob/4a38798a506c9231afdfc83a34ebed394d41dac2/core/consensus/src/impls.rs), not a configurable acknowledgement count. ## Replicated completion @@ -63,15 +63,17 @@ Replication count alone does not establish a bound on that loss. ## Persisted completion When either policy is `persisted`, each multi-replica partition uses a -bounded on-disk prepare WAL. The WAL records full prepares, including message -payloads. A prepare requiring persistence cannot release its `PrepareOk` -until its history and durable frontier are recoverable. +bounded on-disk prepare WAL. Message prepares contain headers and segment +references; their payloads live in segment files retained by hard links until +the WAL history can be reclaimed. Other operations are stored inline. A prepare +requiring persistence cannot release its `PrepareOk` until its message bodies, +WAL history, and durable frontier are recoverable. Prepares can be forwarded while local persistence is pending. Once enough replicas have met the required barrier, the operation can commit and the -primary can apply it and reply. An acknowledged message may still be in -the prepare WAL rather than in a segment file; its recovery does not depend -on first reaching a segment flush threshold. +primary can apply it and reply. An acknowledged message can be recovered +through the WAL before ordinary segment materialization and index updates +finish; recovery does not depend on first reaching a segment flush threshold. The WAL also retains predecessors across message and offset operations. Consequently: @@ -79,7 +81,7 @@ Consequently: - `durability=persisted` does not make an offset response persisted when `consumer_offset_durability=replicated`. - `consumer_offset_durability=persisted` with replicated messages still - journals message payloads. A durable offset's predecessor history must + retains message payloads through WAL segment references. A durable offset's predecessor history must remain recoverable. - A shared barrier can also persist co-batched operations with the weaker policy. That incidental persistence does not strengthen what their earlier @@ -89,8 +91,9 @@ Consequently: WAL history is reclaimed only after the materialized segment and offset state needed to replace it has been synchronized. The server's -`[partition] wal_bytes_max` setting, default `256 MiB`, bounds active WAL -and queued/in-flight prepare bytes per partition. Capacity pressure causes +`[partition] wal_bytes_max` setting, default `256 MiB`, bounds retained +and queued/in-flight prepare bytes per partition. The accounting includes +message bodies even when the WAL stores only references to them. Capacity pressure causes checkpointing and backpressure; it does not downgrade a persisted operation. Temporary rewrites need additional disk space. @@ -130,8 +133,8 @@ Neither policy turns HTTP `ack=none` or a poll's auto-commit into an awaited durable result. See [Which responses prove completion](/docs/server/durability#which-responses-prove-completion). The storage mechanisms are implemented in the -[partition persistence worker](https://github.com/apache/iggy/blob/97f7b0c0335f81691bef34f923f21d783016a6a4/core/partitions/src/persistence.rs) -and [prepare journal](https://github.com/apache/iggy/blob/97f7b0c0335f81691bef34f923f21d783016a6a4/core/journal/src/partition_journal.rs). -The [crash-recovery tests](https://github.com/apache/iggy/blob/97f7b0c0335f81691bef34f923f21d783016a6a4/core/integration/tests/cluster/crash_durability.rs) +[partition persistence worker](https://github.com/apache/iggy/blob/4a38798a506c9231afdfc83a34ebed394d41dac2/core/partitions/src/persistence.rs) +and [prepare journal](https://github.com/apache/iggy/blob/4a38798a506c9231afdfc83a34ebed394d41dac2/core/journal/src/partition_journal.rs). +The [crash-recovery tests](https://github.com/apache/iggy/blob/4a38798a506c9231afdfc83a34ebed394d41dac2/core/integration/tests/cluster/crash_durability.rs) exercise acknowledged persisted messages and offsets below ordinary flush thresholds, on both a singleton and a three-replica cluster. diff --git a/content/docs/clustering/security.mdx b/content/docs/clustering/security.mdx index 7391a382..d66882e8 100644 --- a/content/docs/clustering/security.mdx +++ b/content/docs/clustering/security.mdx @@ -62,6 +62,8 @@ Follower nodes forward control-plane HTTP requests (stream, topic, user, and sim - explicitly configured `http.jwt` signing secrets, identical on every node, or - the cluster PSK: when HTTP is enabled, cluster auth is enabled, and no `http.jwt` secrets are configured, the signing key is derived from `shared_secret`. -Either way, a bearer token minted on any node verifies on every node, which is the invariant the forwarding depends on. Forwarded requests dial the primary at its roster `ip` and `ports.http`. The follower verifies the caller's bearer locally before forwarding, and the primary re-authenticates the request through its normal stack. +Either way, matching JWT key material lets a bearer token minted on one node verify on another. If JWT keys are derived from the PSK, rotating the PSK also changes the JWT key: `previous_shared_secret` applies only to replica authentication. Use independently configured, shared JWT keys when HTTP tokens must remain valid across PSK rotation. Forwarded requests dial the primary at its roster `ip` and `ports.http`. The follower verifies the caller's bearer locally before forwarding, and th [...] -The forwarding covers the control plane only. Partition-plane writes over HTTP (producing messages, storing consumer offsets) are **not forwarded**: each partition is its own consensus group whose primary can diverge from the metadata primary, so those requests must land on the right node. See [Client failover](/docs/clustering/client-failover) for how clients find it. +Acknowledged partition-plane writes over HTTP (producing messages, storing or deleting consumer offsets) first run on the contacted node. If it returns `TransientNotAccepted`, the server tries the other configured HTTP nodes at most once each, within a bounded retry window. That response proves the operation was never admitted. Ambiguous outcomes are returned without replay, and `ack=none` does not provide an acknowledgement for this fallback. Each partition is its own consensus group, s [...] + +When HTTP TLS is enabled, forwarding uses HTTPS and pins the destination certificate to the forwarding node's own HTTP leaf certificate. All nodes therefore need the same HTTP certificate for forwarding; distinct per-node HTTP certificates fail verification. Replica TLS is configured separately and does not encrypt the HTTP forwarding hop. diff --git a/content/docs/clustering/vsr.mdx b/content/docs/clustering/vsr.mdx index bfc89051..24ec8359 100644 --- a/content/docs/clustering/vsr.mdx +++ b/content/docs/clustering/vsr.mdx @@ -56,23 +56,23 @@ VSR uses three main flows: 2. **View change**: replicas exchange `StartViewChange` and `DoViewChange`, then the new primary sends `StartView`. 3. **Recovery**: a restarted or lagging replica requests the current view and repairs missing WAL ranges before serving current state. -A cluster of `2f + 1` replicas tolerates `f` unavailable replicas. Use **at least three replicas** for one-node fault tolerance. A two-node cluster is useful for development, but it **cannot make progress** after either node fails. +Three replicas tolerate one unavailable replica; five tolerate two. Larger groups use a capped replication quorum and a growing view-change quorum, so the usual `2f + 1` rule does not apply to every group size. See [Cluster durability](/docs/clustering/durability#what-an-acknowledgement-means) for the quorum rules. Use **at least three replicas** for one-node fault tolerance. A two-node cluster is useful for development, but it **cannot make progress** after either node fails. ## Wire protocol and SDKs -VSR framing is the *only* wire protocol. Every SDK (Rust, Go, Java, C#, C++, Node.js, Python) speaks it, and every BDD suite runs against the clustered-capable `iggy-server`. Clients built before the VSR migration **cannot talk to current servers**. +VSR framing is the *only binary* wire protocol, used by TCP, QUIC, and WebSocket connections. HTTP uses the REST API. Every SDK (Rust, Go, Java, C#, C++, Node.js, Python) speaks it, and every BDD suite runs against the clustered-capable `iggy-server`. Binary clients built before the VSR migration **cannot talk to current servers**. The Rust SDK needs no feature flags. Depend on the published crate: ```toml [dependencies] -iggy = "0.11.0-edge.4" +iggy = "0.11.0-edge.7" tokio = { version = "1", features = ["full"] } ``` or track the repository directly with `iggy = { git = "https://github.com/apache/iggy"; }`. -The client API is the same in single-node and cluster mode: +The client API is the same in single-node and cluster mode. This example uses the `iggy` / `iggy` credentials explicitly provisioned in [Getting started](/docs/introduction/getting-started): ```rust use iggy::prelude::*;
