This is an automated email from the ASF dual-hosted git repository. hubcio pushed a commit to branch docs/align-with-topic-durability in repository https://gitbox.apache.org/repos/asf/iggy-website.git
commit d660c5761977cedb8c08684ed473880c4fc177b9 Author: hubcio <[email protected]> AuthorDate: Fri Sep 11 03:45:53 2026 +0200 fix(docs): align binary protocol with server 0.9.0 --- content/docs/binary-protocol/cluster.mdx | 36 +++++++++--------- content/docs/binary-protocol/commands.mdx | 27 +++++++++----- .../docs/binary-protocol/connection-lifecycle.mdx | 7 ++-- content/docs/binary-protocol/encodings.mdx | 10 ++--- content/docs/binary-protocol/framing.mdx | 43 ++++++++++++---------- content/docs/binary-protocol/index.mdx | 4 +- 6 files changed, 70 insertions(+), 57 deletions(-) diff --git a/content/docs/binary-protocol/cluster.mdx b/content/docs/binary-protocol/cluster.mdx index 954af26a..6b814be6 100644 --- a/content/docs/binary-protocol/cluster.mdx +++ b/content/docs/binary-protocol/cluster.mdx @@ -3,15 +3,15 @@ title: Server-to-server description: "The replica-to-replica plane: its dedicated TCP port, command discriminants, and traffic that never reaches a client." --- -Replicas talk to each other with the same 256-byte [framing](/docs/binary-protocol/framing) the client protocol uses, on a **dedicated TCP port**, with their own set of `command` discriminants. None of these frames ever appears on a client connection, and a client cannot reach this plane: it is a separate listener, gated by a handshake. +Replicas talk to each other with the same 256-byte [framing](/docs/binary-protocol/framing) the client protocol uses, on a **dedicated TCP port**, with their own set of `command` discriminants. Client listeners reject replica control commands. The replica listener requires a handshake; peer authentication is optional and must be configured separately. -This page documents the replica plane as of binary protocol 0.11.0. It is internal protocol: only Iggy servers speak it, and it can change between server releases without a client-facing version bump. +This page documents the replica plane used with binary protocol 0.11.0. These internal frames do not negotiate a replica protocol version. Use compatible server builds and follow the [cluster upgrade requirements](/docs/clustering/configuration). ## Transport - Every node in `[[cluster.nodes]]` exposes a `tcp_replica` port next to the client ports: `ports = { tcp = 8090, quic = 8080, http = 3000, websocket = 8092, tcp_replica = 9090 }`. The node's `ip` is the roster address for the replica plane (clients fall back to it when `advertised_address` is unset). - The replica plane is **TCP only**, by design: the prepare hash chain, cross-shard fd delegation, and view-change timing all assume an ordered byte stream. -- **Directional dialing:** a replica dials only peers with a strictly greater `replica_id` and accepts inbound connections only from strictly lower ids. Exactly one connection exists per pair, with no tiebreaker races. +- **Directional dialing:** a replica dials only peers with a strictly greater `replica_id` and accepts inbound connections only from strictly lower ids. This gives each connected pair one dialing direction, avoiding simultaneous dial races. - Frames are the standard `[256-byte header][optional body]` with `size` at offset 48. The maximum frame is **64 MiB** by default (`message_bus.max_message_size`). Headers are `#[repr(C)]`, decoded zero-copy, and require 16-byte alignment. - Nothing inside a frame marks it as replica traffic. The separation is structural: client listeners parse every inbound frame as a `RequestHeader` (command `5` only), the replica listener requires the first frame to be `ReplicaHello`, and the client-bound commands `Reply` (8) and `Eviction` (13) are rejected if they arrive on the replica plane. @@ -23,7 +23,7 @@ The full `command` byte registry, client values included. Values above 29 are re |-------|---------|-----------|---------| | 0 | `Reserved` | - | Invalid sentinel | | 1-4 | `Ping`, `Pong`, `PingClient`, `PongClient` | - | Reserved; no production traffic today | -| 5 | `Request` | client to server | Client command ([framing](/docs/binary-protocol/framing)) | +| 5 | `Request` | client to server; accepted on replica ingress too | Client command; replica ingress decodes the internal routed-request shape | | 6 | `Prepare` | primary to backup, backup to next backup | Replicate one operation | | 7 | `PrepareOk` | backup to primary | Acknowledge a prepare | | 8 | `Reply` | server to client | Client reply; rejected on the replica plane | @@ -55,16 +55,16 @@ There is no dedicated replica heartbeat: liveness is the `Commit` broadcast, sen The `checksum` and `checksum_body` fields that client frames leave zero are live on the replica plane: -- **Frame seal** (`checksum`, bytes 0..16): XxHash3-64 over header bytes 16..256, widened to u128. Sealed on every replica message **except** `Prepare`, `RepairPrepare`, and the three handshake frames (whose integrity comes from the keyed MAC). Verified once, at typed decode, before any field validation; a mismatch drops the frame. -- **Prepare identity** (`Prepare` / `RepairPrepare` only): those two spend `checksum` on an identity hash instead - XxHash3-64 over the whole 256-byte header computed with `checksum = 0` and `view = 0`, so a retransmit that re-stamps `view` keeps the same identity. `PrepareOk` echoes it in `prepare_checksum`, and each prepare's `parent` field carries the previous prepare's identity, forming a hash chain. -- **Body seal** (`checksum_body`, bytes 16..32): three regimes. Metadata-plane prepares seal their body with XxHash3-64. Partition-plane prepares leave it zero: the message batch inside already carries `batch_checksum`, verified at network ingress. `DoViewChange` and `StartView` seal their suffix bodies. Every other message leaves it zero (state-transfer bodies are verified per artifact, not per frame). +- **Frame seal** (`checksum`, bytes 0..16): XxHash3-64 over header bytes 16..256, widened to u128. Sealed on replica control messages. `Request` is unsealed; `Prepare` and `RepairPrepare` use an identity hash instead. Handshake frames use a keyed MAC when authentication is enabled. Verified at typed decode before field validation; a mismatch drops the frame. +- **Prepare identity** (`Prepare` / `RepairPrepare` only): those two spend `checksum` on an identity hash instead - XxHash3-64 over the whole 256-byte header computed with `checksum = 0` and `view = 0`, so a retransmit that re-stamps `view` keeps the same identity. `RepairPrepare` retains the original identity; the receiver restores command `Prepare` before integrity validation. `PrepareOk` echoes it in `prepare_checksum`, and each prepare's `parent` field carries the previous prepare's [...] +- **Body seal** (`checksum_body`, bytes 16..32): three regimes. Metadata-plane prepares seal their body with XxHash3-64. Partition-plane prepares leave it zero. `SendMessages` carries its own `batch_checksum`, verified at network ingress; consumer-offset writes have no message batch checksum. `DoViewChange` and `StartView` seal their suffix bodies. Every other message leaves it zero (state-transfer bodies are verified per artifact, not per frame). - The seals are **unkeyed integrity checks, not authentication**. Peer authentication comes from the handshake below; without `cluster.auth` and TLS enabled, the replica port trusts any peer that can reach it. Do not expose `tcp_replica` beyond the cluster network. -`Prepare`, `RepairPrepare`, and the session-forwarding headers validate their reserved regions as zero; the other replica headers leave reserved bytes covered by the frame seal only. +`Prepare` and `RepairPrepare` validate their common and per-command reserved regions as zero. Session-forwarding headers validate their per-command reserved tails. Other reserved bytes in sealed control headers are covered by the frame seal. ## Handshake -Commands 14-16, exchanged before any consensus traffic. These are raw `GenericHeader` frames (`size = 256`) using the per-command area at bytes 128..256: +With `cluster.auth.enabled = true`, commands 14-16 are exchanged before any consensus traffic. With authentication disabled, the dialer sends only `ReplicaHello`. These are raw `GenericHeader` frames (`size = 256`) using the per-command area at bytes 128..256: | Offset | Size | Content | |--------|------|---------| @@ -72,7 +72,7 @@ Commands 14-16, exchanged before any consensus traffic. These are raw `GenericHe | 160 | 32 | BLAKE3 keyed MAC (acceptor's in `Challenge`, dialer's in `Finish`) | | 192 | 1 | `Challenge` only: handshake status | -Statuses: `0` Ok, `1` UnknownCommand, `2` ClusterMismatch, `3` DirectionalRule; values `4` and `5` are reserved (never sent), and a dialer treats any unknown status byte as a rejection. The MAC key derives from `cluster.auth.shared_secret` (32+ bytes; `previous_shared_secret` gives a rotation window), and with cluster TLS enabled (TLS 1.3 only, ALPN `iggy-replica`) the TLS exporter is folded into every MAC as channel binding. The handshake authenticates **cluster membership**, not per-re [...] +Statuses: `0` Ok, `1` UnknownCommand, `2` ClusterMismatch, `3` DirectionalRule; `4` AuthRequired and `5` MacMismatch are log-only labels (never sent), and a dialer treats any unknown status byte as a rejection. The MAC key derives from `cluster.auth.shared_secret` (32+ bytes; `previous_shared_secret` gives a rotation window), and with cluster TLS enabled (TLS 1.3 only, ALPN `iggy-replica`) the TLS exporter is folded into every MAC as channel binding. The handshake authenticates **cluster [...] The `cluster` header field (bytes 32..48) is the first 16 bytes of `blake3(cluster_name)` as a little-endian u128 on every replica frame, checked during the handshake, so nodes from a differently named cluster cannot connect even with auth disabled. @@ -96,7 +96,7 @@ Per-command fields (bytes 128..256): | 224 | 4 | `user_id` | Authenticated user | | 228 | 28 | reserved | Zero | -For metadata operations the body is the admitted command's payload **verbatim**. For `SendMessages` it is the [message batch](/docs/binary-protocol/messages) as stamped by the primary at journal append (`base_offset`, `base_timestamp`, and the recomputed `batch_checksum` are final); each backup re-derives the expected stamp from its own position and refuses a mismatch. Replication is **chain-form**: the primary sends to `(replica + 1) % replica_count`, each backup forwards to its own suc [...] +Metadata bodies contain the server-prepared payload, which may differ from the client request. For example, topic creation adds partition assignments before replication. For `SendMessages` it is the [message batch](/docs/binary-protocol/messages) as stamped by the primary at journal append (`base_offset`, `base_timestamp`, and the recomputed `batch_checksum` are final); each backup re-derives the expected stamp from its own position and refuses a mismatch. Replication is **chain-form**: [...] ### PrepareOk (7) @@ -146,17 +146,17 @@ Artifact kinds: | Kind | Artifact | Plane | Content | |------|----------|-------|---------| -| 0 | `METADATA_SNAPSHOT` | metadata | `snapshot.bin` verbatim (MessagePack, snapshot format version 3); `frontier` = sequence number | -| 1 | `CLIENT_TABLE` | metadata | Client table encoding (magic `ICT2`); `frontier` = mutation frontier | +| 0 | `METADATA_SNAPSHOT` | metadata | `snapshot.bin` verbatim (MessagePack, snapshot format version 5; readers accept versions 3 through 5); `frontier` = sequence number | +| 1 | `CLIENT_TABLE` | metadata | Client table encoding (magic `ICT3`, including dedup fences); readers also accept `ICT2` without fences; `frontier` = mutation frontier | | 2 | `SEGMENT_LOG` | partition | One retained segment's `.log` verbatim; `frontier` = segment base offset | -| 3 | `CONSUMER_OFFSETS` | partition | Consumer + group offset tables and applied purge generation; `frontier` = offer's `commit_op` | +| 3 | `CONSUMER_OFFSETS` | partition | `ICO1` version 1: consumer/group offsets, purge generation, next message offset, dedup window, prepare-chain checksum and checkpoint prepare; `frontier` = offer's `commit_op` | ## Session forwarding -A client may dial a backup; authentication happens there, and only the verified identity travels to the primary (credentials never cross the replica plane): +A client may dial a backup; authentication happens there, and only the verified identity travels to the primary (login credentials never cross the replica plane): - **ForwardRegister (26):** `client` 128, `nonce` 144, `user_id` 160. -- **ForwardRegisterResult (27):** `nonce` 128, `client` 144, `epoch` 160, `watermark` 168, outcome byte at **255**: `0` Ok, `1` NotPrimary, `2` NotCaughtUp, `3` PipelineFull, `4` InProgress, `5` Canceled, `6` ClientIdOwnedByAnotherUser. `epoch` and `watermark` must be zero on any non-Ok outcome. +- **ForwardRegisterResult (27):** `nonce` 128, `client` 144, `epoch` 160, `watermark` 168, outcome byte at **255**: `0` Ok, `1` NotPrimary, `2` NotCaughtUp (reserved), `3` PipelineFull, `4` InProgress, `5` Canceled, `6` ClientIdOwnedByAnotherUser. `epoch` and `watermark` must be zero on any non-Ok outcome. - **ForwardLogout (28):** `client` 128, `nonce` 144, `session` 160, `request` 168. - **ForwardLogoutResult (29):** `nonce` 128, `client` 144, `commit` 160, outcome at 255: `0` Ok, `1` NotPrimary, `2` PipelineFull, `3` InProgress, `4` Canceled. @@ -165,6 +165,6 @@ A client may dial a backup; authentication happens there, and only the verified Two consensus planes share this one message set; there are no plane-specific commands. Routing is by the `group: u64` field carried on every consensus header except the handshake and session-forwarding frames (those are implicitly metadata-plane): - `group = 1 << 63` is the **metadata plane** (streams, topics, users, consumer groups; durable on-disk journal). -- Any other value is a packed stream/topic/partition key: a **partition plane** group (message batches, consumer offsets; in-memory journal). +- Any other value is a packed stream/topic/partition key: a **partition plane** group (message batches, consumer offsets). Completion and recovery depend on the independent message and offset [durability policies](/docs/server/durability). -Same wire, a few divergent semantics: partition-plane prepares leave `checksum_body` zero and rely on the batch's own `batch_checksum`, their identity checksum covers the header alone, and `StateTransferTarget.unavailable_transient` / `commit_max` are read only by the partition arm. +Same wire, a few divergent semantics: partition-plane prepares leave `checksum_body` zero, with message integrity supplied by the batch's own checksums; their prepare identity covers the header alone, and `StateTransferTarget.unavailable_transient` / `commit_max` are read only by the partition arm. diff --git a/content/docs/binary-protocol/commands.mdx b/content/docs/binary-protocol/commands.mdx index c97ec88c..e70b696f 100644 --- a/content/docs/binary-protocol/commands.mdx +++ b/content/docs/binary-protocol/commands.mdx @@ -7,11 +7,11 @@ description: "The catalog of command codes, their operation bytes, and which com The u32 command codes. Replicated commands are identified on the wire by their `operation` byte (see [Operation discriminants](/docs/binary-protocol/framing#operation-discriminants)). The code column below is the protocol-level registry and, for non-replicated commands, the value carried in header bytes 196..200. -```bash +```text # System PING = 1 # non-replicated; works without login GET_STATS = 10 # non-replicated -GET_SNAPSHOT = 11 # non-replicated +GET_SNAPSHOT_FILE = 11 # non-replicated GET_CLUSTER_METADATA = 12 # non-replicated DESCRIBE_OPTIONS = 13 # non-replicated GET_ME = 20 # non-replicated @@ -136,7 +136,7 @@ Scope: 1 = topic, 2 = stream, 3 = user. Returns the option catalog for the scope [name_len: u8][name: N][options block to end] ``` -The stream option catalog is empty today, so the block is normally empty (zero bytes). +The stream option catalog is empty, so the block is normally empty (zero bytes). **Delete stream. Code: 203.** @@ -222,7 +222,7 @@ Only `compression_algorithm`, `message_expiry`, and `max_topic_size` are updatab [stream_id: Identifier][topic_id: Identifier][partition_id: u32][segments_count: u32] ``` -Deletes the `segments_count` oldest sealed segments of the partition. +Deletes up to `segments_count` oldest sealed segments of the partition. The active segment is retained. ### Messages @@ -234,7 +234,7 @@ Deletes the `segments_count` oldest sealed segments of the partition. [strategy: 9 bytes][count: u32][auto_commit: u8] ``` -`strategy` is a [polling strategy](/docs/binary-protocol/encodings#polling-strategy). `count` is the requested number of messages. `auto_commit = 1` stores the consumer offset server-side as part of the poll. +`strategy` is a [polling strategy](/docs/binary-protocol/encodings#polling-strategy). `count` is the requested number of messages. `auto_commit = 1` asks the server to advance the consumer offset. A primary submits a quorum offset write before replying, but the poll does not wait for that write to commit. A follower-served poll does not advance the durable offset. See [Client failover](/docs/clustering/client-failover#where-requests-are-served). The response body: @@ -243,7 +243,7 @@ The response body: [batch records to end] ``` -The 16-byte prefix is followed by a stream of [batch records](/docs/binary-protocol/messages) served as stored: each record's header carries the stamped `base_offset` and `base_timestamp`, and each frame's deltas resolve against them. A record may be a server-sliced view of a larger stored batch, so the first polled offset is `base_offset + offset_delta` of the first frame, not necessarily `base_offset` itself. `current_offset` is the partition's newest offset at poll time. +The 16-byte prefix is followed by a stream of [batch records](/docs/binary-protocol/messages): each record's header carries the stamped `base_offset` and `base_timestamp`, and each frame's deltas resolve against them. A record may be a server-sliced view of a larger stored batch, so the first polled offset is `base_offset + offset_delta` of the first frame, not necessarily `base_offset` itself. `current_offset` is the partition's newest offset at poll time. The server rebuilds records wh [...] **Send messages. Code: 101.** @@ -256,6 +256,15 @@ The 16-byte prefix is followed by a stream of [batch records](/docs/binary-proto `metadata_length` counts the bytes from `stream_id` through `messages_count` inclusive, so a reader can skip straight to the batch. The producer leaves `partition_id`, `base_offset`, and `base_timestamp` zero in the [batch header](/docs/binary-protocol/messages), and the server stamps them. Every checksum is producer-computed and verified at admission. The reserved regions must be zero. +A committed append returns offset confirmations without a metadata result section: + +```text +[confirmations_count: u32] +[stream_id: u32][topic_id: u32][partition_id: u32][base_offset: u64] x confirmations_count +``` + +Each confirmation is 20 bytes. The server reports one partition per append. A zero count or an empty successful body supplies no offset information; deduplicated requests receive an empty body. Retry coverage is [bounded](/docs/binary-protocol/framing#request-numbering), and crash durability depends on the topic's [durability policy](/docs/server/durability). + **Flush unsaved buffer. Code: 102.** ```text @@ -273,7 +282,7 @@ Parses, but the server always answers `FeatureUnavailable`: there is no on-deman [partition_flag: u8][partition_id: u32] ``` -Response body (20 bytes): +When an offset is stored, the response body is 20 bytes. When none is stored, a successful reply has an empty body. ```text [partition_id: u32][current_offset: u64][stored_offset: u64] @@ -297,7 +306,7 @@ The trailing `ack` byte is mandatory on both write commands. A payload without i | Value | Level | Meaning | |-------|-------|---------| -| 0 | `NoAck` | Leader-local write; respond as soon as local state is updated (the fast path `PollMessages` auto-commit uses) | +| 0 | `NoAck` | Local write on a single-replica partition; replicated partitions use the same consensus path as `Quorum` | | 1 | `Quorum` | Replicate through partition consensus; respond after a quorum commit (default for explicit writes) | ### Consumer groups @@ -364,7 +373,7 @@ Read-only: a member asks for its current partition assignment and the group gene [options block to end] ``` -`permissions_len` and the permissions bytes are present **only** when `has_permissions = 1`. When it is 0 the options block follows immediately. The user option catalog is empty today, so the block is normally empty. `status`: 1 = active, 2 = inactive. +`permissions_len` and the permissions bytes are present **only** when `has_permissions = 1`. When it is 0 the options block follows immediately. The user option catalog is empty, so the block is normally empty. `status`: 1 = active, 2 = inactive. **Delete user. Code: 34.** diff --git a/content/docs/binary-protocol/connection-lifecycle.mdx b/content/docs/binary-protocol/connection-lifecycle.mdx index 8add77d8..dff74543 100644 --- a/content/docs/binary-protocol/connection-lifecycle.mdx +++ b/content/docs/binary-protocol/connection-lifecycle.mdx @@ -15,11 +15,11 @@ bits 9..0 patch value = major << 20 | minor << 10 | patch ``` -Integer order equals semver order. The value tracks the `iggy_binary_protocol` crate release. Under 0.x the compatibility gate is minor-scoped: the server accepts a client whose packed version is at least the server's minimum and whose `major.minor` is at most the server's. Patch releases never change the wire, so the upper bound ignores patch. Past 1.0.0 the gate follows strict semver (major bump = incompatible). +Integer order equals semver order. The value tracks the `iggy_binary_protocol` crate release. Under 0.x the compatibility gate is minor-scoped: the server accepts a client whose packed version is at least the server's minimum and whose `major.minor` is at most the server's. Patch releases never change the wire, so the upper bound ignores patch. The implemented minimum is the current version with its patch component zeroed. Compatibility across minor versions after 1.0.0 requires a future [...] ## Login-register -The only way to authenticate is the register handshake: command code **40** (`LOGIN_REGISTER`, username and password) or **45** (`LOGIN_REGISTER_WITH_PAT`, personal access token). Both ride `operation = 1` (`Register`) with `session = 0` and a freshly minted non-zero `client` id. +The only way to authenticate is the register handshake: command code **40** (`LOGIN_REGISTER`, username and password) or **45** (`LOGIN_REGISTER_WITH_PAT`, personal access token). Both ride `operation = 1` (`Register`) with `session = 0`, `request = 0`, and a non-zero `client` id. Mint a fresh id for a new logical client. A client resuming an existing identity must retain its request counter and use the new session epoch returned by registration. Both request bodies begin with the `ClientVersionInfo` prefix, so the server can gate on the version before touching credentials: @@ -52,9 +52,10 @@ After the prefix: An incompatible protocol version is answered with a 256-byte `Eviction` frame, reason `14` (`IncompatibleProtocol`), carrying the accepted window at bytes 144 (max) and 148 (min) as packed u32 versions. A body without a decodable `ClientVersionInfo` prefix gets reason `15` (`MalformedLogin`) with a zero window. Bad credentials get reasons 9-11. See [EvictionHeader](/docs/binary-protocol/framing#evictionheader) for the frame layout. -A successful login is a normal `Reply` whose body is: +A successful login is a normal `Reply` with `status = 0`. Its body includes an empty [result section](/docs/binary-protocol/framing#result-section) before the login payload: ```text +[result_count: u32 = 0] [user_id: u32] [session: u64] [server_protocol_version: u32] diff --git a/content/docs/binary-protocol/encodings.mdx b/content/docs/binary-protocol/encodings.mdx index 515e46f2..4fce394d 100644 --- a/content/docs/binary-protocol/encodings.mdx +++ b/content/docs/binary-protocol/encodings.mdx @@ -43,7 +43,7 @@ Several commands take an optional partition id encoded as 5 fixed bytes: [flag: u8][partition_id: u32] ``` -`flag = 1` means the id is set. `flag = 0` means none (the value bytes are then zero). For consumer-group consumers the partition id is left unset and the server resolves it. +`flag = 1` means the id is set. `flag = 0` means none (the value bytes are then zero). Group polls and group offset writes require an explicit assigned partition. Plain-consumer polls and consumer-offset reads use partition 0 when the id is omitted. ## Partitioning @@ -57,7 +57,7 @@ How `SendMessages` picks the target partition: - `PartitionId`: `kind = 2`, `length = 4`, value is a u32 partition id. - `MessagesKey`: `kind = 3`, `length = 1..255`, value is a routing key hashed to a partition. -First-party binary SDKs pre-resolve balanced and key routing client-side and normally send `kind = 2`. +The Rust binary SDK resolves balanced and key routing client-side and sends `kind = 2`. SDKs can also send the other wire strategies for server-side resolution. ## Polling strategy @@ -100,7 +100,7 @@ A user headers block (inside a [message frame](/docs/binary-protocol/messages)) [kind: u8][length: u32][data: length bytes] ``` -Fields pair up: first the key, then the value. Keys are `String` kind. Every `length` must be 1..=255, kind 0 is rejected, and the block must consume its byte range exactly. Unknown value kind codes are preserved and forwarded, so headers survive mixed-version clusters. +Fields pair up: first the key, then the value. Keys and values can both use the header kinds above. Every `length` must be 1..=255, kind 0 is rejected, and the block must consume its byte range exactly. The wire validator preserves unknown non-zero kind codes. A typed SDK accessor may still reject a kind it does not recognize. ## Options block @@ -110,7 +110,7 @@ On top of the TLV walk, options enforce: string keys only (kind 2, valid UTF-8), Semantics: -- **Create** requests resolve absent keys to server defaults and persist the effective values, so `GetTopic` always shows what is in force. +- **Create** requests resolve absent keys to server defaults and persist the effective values. An update using a zero retention sentinel can make `GetTopic` option values disagree with its fixed expiry/size fields; see [update semantics](/docs/server/topic-options). - **Update** requests are patches: keys absent from the block are left alone, never reset. A client built before a key existed cannot erase it. - Unknown keys are rejected at the wire edge, never silently skipped. @@ -125,4 +125,4 @@ The `compression_algorithm` value used by topic options: | 1, `none` | No compression (default) | | 2, `gzip` | Gzip | -Any other value is rejected. The option is a **placeholder today**: the value is validated, persisted, and echoed back, but neither the server nor the SDKs compress or decompress payloads yet. To compress today, do it client-side and tag messages via user headers - see the [message headers examples](https://github.com/apache/iggy/tree/master/examples/rust/src/message-headers) in the Iggy repo. +Any other value is rejected. The option is a **placeholder today**: the value is validated, persisted, and echoed back, but it does not enable automatic payload compression or decompression. To compress today, do it client-side and tag messages via user headers - see the [message headers examples](https://github.com/apache/iggy/tree/master/examples/rust/src/message-headers) in the Iggy repo. diff --git a/content/docs/binary-protocol/framing.mdx b/content/docs/binary-protocol/framing.mdx index e3bf5148..896b11d3 100644 --- a/content/docs/binary-protocol/framing.mdx +++ b/content/docs/binary-protocol/framing.mdx @@ -36,9 +36,9 @@ Client to server. 256 bytes. | 60 | 1 | `command` | u8 | `5` (`Request`). | | 61 | 1 | `replica` | u8 | Send zero. | | 62 | 66 | reserved | bytes | Zero. | -| 128 | 16 | `client` | u128 | Client-chosen session identity, non-zero. Minted fresh for each registration. | -| 144 | 16 | `request_checksum` | u128 | Optional integrity stamp over the request body (the Rust SDK uses XxHash3-64 widened to u128). Lets the server's client table catch a `request` number reused for different arguments. Zero disables the comparison. Stamped only for metadata-plane operations; zero for partition-plane and non-replicated ones. | -| 160 | 8 | `timestamp` | u64 | Informational; the server echoes it into `ReplyHeader.timestamp`. May be zero. | +| 128 | 16 | `client` | u128 | Client-chosen identity, non-zero. Mint a fresh id for a new logical client; retain it when resuming that client. | +| 144 | 16 | `request_checksum` | u128 | Optional integrity stamp over the request body (the Rust SDK uses XxHash3-64 widened to u128). Lets the server's client table catch a `request` number reused for different arguments. Zero disables the comparison. The Rust SDK stamps metadata/session operations and `DeleteSegments`; it sends zero for partition operations and non-replicated ones. | +| 160 | 8 | `timestamp` | u64 | May be zero. Direct replies can echo it, but committed prepares use a server-assigned timestamp. | | 168 | 8 | `request` | u64 | Request number, per client. See [request numbering](#request-numbering). | | 176 | 1 | `operation` | u8 | The [`Operation`](#operation-discriminants) discriminant. | | 177 | 7 | padding | bytes | Zero. | @@ -87,19 +87,22 @@ Value 0 is reserved and rejected. The 64..127 range is reserved for server-inter The planes matter for delivery semantics: -- **Metadata operations** replicate through the metadata consensus group. The server deduplicates them by `(client, request)` and caches replies, so a retried request gets the cached answer instead of a double apply (exactly-once). -- **Partition operations** replicate through their partition's consensus group. They are at-least-once: no reply cache, a replay may apply again. -- **Non-replicated operations** are reads. They bypass consensus and deduplication entirely. +- **Metadata operations** replicate through the metadata consensus group. The server deduplicates them by `(client, request)` within its retained client history. Recent replies are cached and replayed; an older request can remain recognized as already applied after its reply has left the cache. This is not an unlimited exactly-once guarantee. +- **Partition operations** normally replicate through their partition's consensus group. Each partition retains client request watermarks and a 128-id committed window to suppress duplicates. There is no reply cache: a recognized duplicate gets an empty success, including no offset confirmations for a retried send. Capacity eviction or a new client identity loses this coverage. Single-replica consumer-offset writes with `ack = 0` bypass consensus and this deduplication. +- **Non-replicated operations** bypass consensus and deduplication. A poll can still submit a separate consumer-offset write when auto-commit is enabled. ## Request numbering `request` is a per-client counter the server's client table tracks for metadata-plane operations: -- Metadata operations must send a strictly increasing `request` (the SDK advances the counter per metadata request). -- Partition operations and non-replicated operations send the current counter value without advancing it. The server doesn't track theirs. +- New metadata operations must use a `request` above the retained watermark; a retry reuses its original number and arguments. +- The Rust SDK advances one shared counter for metadata operations, partition operations, logout, and `DeleteSegments`. Non-replicated operations use the current value without advancing it. +- Register uses `request = 0`; other operations except `NonReplicated` require a non-zero request number. - The counter is a watermark, not a contiguous sequence: any value above the last accepted one is admissible. -`session` is the fence epoch: the value handed back by the login reply. Every request after login must echo it. When the same `client` id registers again, the new registration mints a higher epoch and requests carrying the old one are fenced (rejected as zombies). +`session` is the fence epoch handed back by the login reply, derived from the committed Register log position. Echo it after login. Registering the same `client` again mints a higher epoch; the metadata client table rejects an older epoch. Partition dispatch uses the identity bound to the authenticated connection. + +The partition window is measured in the shared client request-id space. An unseen request arriving 128 or more ids below that partition's watermark is treated as already committed and receives success without executing. Avoid allowing an unresolved write to fall outside that window; the protocol does not provide an unlimited reordered-retry guarantee. ## ReplyHeader @@ -111,21 +114,21 @@ Server to client. 256 bytes, followed by `size - 256` bytes of body. | 16 | 16 | `checksum_body` | u128 | Zero on client-facing frames. | | 32 | 16 | `cluster` | u128 | Cluster id. | | 48 | 4 | `size` | u32 | Total frame length: 256 + body length. | -| 52 | 4 | `view` | u32 | Consensus view the reply was produced in. | +| 52 | 4 | `view` | u32 | View associated with the reply. A cached committed reply retains its original view; direct replies can echo the request value. | | 56 | 4 | `release` | u32 | Zero. | | 60 | 1 | `command` | u8 | `8` (`Reply`). | -| 61 | 1 | `replica` | u8 | Answering replica index. | +| 61 | 1 | `replica` | u8 | Replica index stamped by the reply builder. Cached committed replies retain the original primary index. | | 62 | 66 | reserved | bytes | Zero. | | 128 | 16 | `request_checksum` | u128 | Echoed from the request. | | 144 | 16 | `context` | u128 | Server context. | | 160 | 16 | `client` | u128 | Echoed client id. | -| 176 | 8 | `op` | u64 | Log position of the committed operation. | -| 184 | 8 | `commit` | u64 | Commit point at reply time. | -| 192 | 8 | `timestamp` | u64 | Echo of the request `timestamp`. | -| 200 | 8 | `request` | u64 | Echoed request number; correlate replies by this. | -| 208 | 1 | `operation` | u8 | Echoed operation discriminant. | +| 176 | 8 | `op` | u64 | Committed log position on consensus replies. Direct replies can instead carry the session epoch or zero; this field alone does not prove a commit. | +| 184 | 8 | `commit` | u64 | Commit position associated with the reply. A cached committed reply retains its original position. | +| 192 | 8 | `timestamp` | u64 | Server-assigned prepare timestamp on committed replies, or the request timestamp on direct replies. | +| 200 | 8 | `request` | u64 | Echoed request number. Non-replicated requests can reuse a number, so it is not a unique correlation id for concurrent reads. | +| 208 | 1 | `operation` | u8 | Reply operation. Server rewrites can return `64` for CreateTopic, `65` for CreatePartitions, or `68` for DeleteSegments instead of the original client operation. | | 209 | 7 | padding | bytes | Zero. | -| 216 | 4 | `status` | u32 | `0` = accepted. Nonzero = an `IggyError` code for a failure decided **before** commit (authorization denial, admission reject). A nonzero status always comes with an empty body. | +| 216 | 4 | `status` | u32 | `0` = no header-level failure. Nonzero = an `IggyError` code for a failure decided **before** commit (authorization denial, admission reject). A nonzero status always comes with an empty body. | | 220 | 36 | reserved | bytes | Zero. | Decode order for a client: @@ -137,7 +140,7 @@ Decode order for a client: ## Result section -Replies for all metadata operations and for the partition-plane consumer-offset writes (`StoreConsumerOffset`, `DeleteConsumerOffset`) are **result-framed**: the body starts with a committed-result section ahead of the typed payload. +Replies for all metadata operations, including their server-internal rewrites, and for the partition-plane consumer-offset writes (`StoreConsumerOffset`, `DeleteConsumerOffset`) are **result-framed**. Non-empty Register replies use the same framing. The body starts with a result section ahead of the typed payload. ```text [count: u32] @@ -145,9 +148,9 @@ count x { index: u32, result: u32 } ``` - Success: `count = 0`, and the typed response payload (if any) follows the 4 count bytes. -- Committed business rejection: one entry `{ index: 0, result: error_code }` and no payload. +- Business or transient rejection: one entry `{ index: 0, result: error_code }` and no payload. -The header `status` channel and the result section are mutually exclusive by construction: a reply either failed pre-commit (`status` nonzero, empty body) or committed (`status` zero, result section present). A login-register reply carries the result section only when non-empty. On success its body starts directly with the [login response payload](/docs/binary-protocol/connection-lifecycle#login-register). +A nonzero header `status` has an empty body. With `status = 0`, decode the result section for the operations above: it can report either a committed outcome or a pre-commit transient rejection. Its presence alone does not prove commitment. A successful login-register reply starts with `count = 0`, followed by the [login response payload](/docs/binary-protocol/connection-lifecycle#login-register). An empty Register body is a terminal failure, not a successful login. Replies to non-replicated commands aren't result-framed: after `status = 0` the body is the response payload directly. diff --git a/content/docs/binary-protocol/index.mdx b/content/docs/binary-protocol/index.mdx index a97d18a3..0040cfff 100644 --- a/content/docs/binary-protocol/index.mdx +++ b/content/docs/binary-protocol/index.mdx @@ -3,9 +3,9 @@ title: Binary Protocol description: "Binary protocol 0.11.0: one wire format across TCP, QUIC and WebSocket, covering both the client and replica planes." --- -Iggy speaks one binary protocol over its three binary transports: TCP, QUIC, and WebSocket. Commands, responses, data models, and status codes are the same on all of them. The HTTP transport is separate: it exposes the same operations as JSON REST endpoints, listed in [server.http](https://github.com/apache/iggy/blob/master/core/server/server.http). +Iggy speaks one binary protocol over its three binary transports: TCP, QUIC, and WebSocket. Commands, responses, data models, and status codes are the same on all of them. The HTTP transport is separate: it exposes JSON REST endpoints for a subset of these operations, listed in [server.http](https://github.com/apache/iggy/blob/master/core/server/server.http). -This section describes **binary protocol version 0.11.0**. The protocol version is the semver of the `iggy_binary_protocol` crate, and it's exchanged and checked during login (see [Connection lifecycle](/docs/binary-protocol/connection-lifecycle)). The crate itself is the source of truth: every request and response module carries a `Wire format:` doc comment, and these pages are written against those. +This section describes **binary protocol version 0.11.0**. The protocol version is the `major.minor.patch` of the `iggy_binary_protocol` crate, ignoring prerelease tags, and it's exchanged and checked during login (see [Connection lifecycle](/docs/binary-protocol/connection-lifecycle)). The crate itself is the source of truth: request and response types document their payload layouts, and the server dispatch defines which operations are supported. All multi-byte integers are **little-endian** unless stated otherwise.
