This is an automated email from the ASF dual-hosted git repository. hubcio pushed a commit to branch docs/align-with-topic-durability in repository https://gitbox.apache.org/repos/asf/iggy-website.git
commit dfed33d87b08bf16a831d3e8acbf89e9c5604a3a Author: hubcio <[email protected]> AuthorDate: Fri Sep 11 01:52:31 2026 +0200 fix(docs): align server docs with 0.9.0 Server guidance drifted from configuration, authentication, storage and deployment behavior. Correct the verified claims and examples, qualify historical benchmarks, and document safe result metadata. Validate source startup, all benchmark kinds, Docker and Helm examples, and built pages in both themes. --- content/docs/server/benchmarking.mdx | 14 +++++-- content/docs/server/configuration.mdx | 66 ++++++++++++++++++-------------- content/docs/server/docker.mdx | 23 +++++------ content/docs/server/introduction.mdx | 21 ++++++---- content/docs/server/networking.mdx | 22 +++++------ content/docs/server/security.mdx | 26 ++++++------- content/docs/server/storage-engine.mdx | 16 ++++---- content/docs/server/topic-options.mdx | 28 ++++++++------ src/components/architecture-diagrams.tsx | 12 +++--- 9 files changed, 127 insertions(+), 101 deletions(-) diff --git a/content/docs/server/benchmarking.mdx b/content/docs/server/benchmarking.mdx index 65b4fe0f..59f9770c 100644 --- a/content/docs/server/benchmarking.mdx +++ b/content/docs/server/benchmarking.mdx @@ -5,14 +5,18 @@ description: "How Iggy is benchmarked, the tooling that ships with it, and the p **Benchmarks should be the first-class citizens**. We believe that performance is crucial for any system, and we strive to provide the best possible performance for our users. Please check, why we believe that the **[transparent benchmarking](https://iggy.apache.org/blogs/2025/02/17/transparent-benchmarks)** is so important. -We've also built the **[benchmarking platform](https://benchmarks.iggy.apache.org)** where anyone can upload the benchmarks and compare the results with others. This is the another open-source project available [here](https://github.com/apache/iggy/tree/master/core/bench/dashboard). +We've also built the **[benchmarking platform](https://benchmarks.iggy.apache.org)** where you can browse published benchmarks and compare results. This is the another open-source project available [here](https://github.com/apache/iggy/tree/master/core/bench/dashboard).  +*Historical dashboard screenshot showing a 0.5.0 run, not 0.9.0 measurements.* + Iggy comes with a built-in benchmarking tool, `iggy-bench`. It's written in Rust and uses the `tokio` runtime for asynchronous I/O, mimicking the example client applications, so you can use it to estimate the performance of the server in your environment. It is part of the [core repository](https://github.com/apache/iggy/tree/master/core/bench) and lives in the `core/bench` directory.  +*Historical CLI screenshot. Server-start and cleanup flags shown there have been removed; use the commands below and the current `iggy-bench --help`.* + ## Running benchmarks First build the project in release mode: @@ -79,7 +83,7 @@ Producer and consumer counts default to six. Pinned workloads also default to si cargo r --bin iggy-bench -r -- balanced-producer-and-consumer-group tcp ``` -7. End-to-end producing consumer (`e2e`): each task produces and then consumes its own messages, measuring the full round trip: +7. End-to-end producing consumer (`e2e`): each task alternates producing and consuming, measuring latency from the messages' producer timestamps: ```bash cargo r --bin iggy-bench -r -- end-to-end-producing-consumer tcp @@ -109,11 +113,13 @@ Each transport subcommand accepts a trailing `output` subcommand that persists t cargo r --bin iggy-bench -r -- pinned-producer tcp output -o performance_results --identifier my-host ``` -`-o/--output-dir` defaults to `performance_results`. Optional flags (`--remark`, `--gitref`, extra info) annotate the run. You can inspect persisted results with the in-repo report and runner crates (`core/bench/report`, `core/bench/runner`), and browse or compare them in the dashboard (`core/bench/dashboard`), which also powers the public [benchmarking platform](https://benchmarks.iggy.apache.org). A prebuilt dashboard image is available: `docker pull apache/iggy-bench-dashboard`. +When the server address uses `localhost` or `127.0.0.1`, the saved command includes non-secret environment settings from the server configuration catalog. Credentials and unknown `IGGY_` variables are omitted. + +`-o/--output-dir` defaults to `performance_results`. Optional flags (`--remark`, `--gitref`, extra info) annotate the run. Generated charts use the in-repo report library (`core/bench/report`). The runner (`core/bench/runner`) executes benchmarks across Git revisions. You can browse or compare results in the dashboard (`core/bench/dashboard`), which also powers the public [benchmarking platform](https://benchmarks.iggy.apache.org). A prebuilt dashboard image is available: `docker pull ap [...] ## Performance -The server is thread-per-core and shared-nothing, built on `io_uring` (via `compio`), with shard and CPU pinning configurable under `[sharding]`. Throughput and latency depend heavily on hardware, transport, and payload shape (`messages-per-batch * message-size`). Run the benchmarks on your own hardware, or browse current, dated results on the [benchmarking platform](https://benchmarks.iggy.apache.org). +The server is thread-per-core and shared-nothing, built on `io_uring` on Linux (via `compio`), with shard and CPU pinning configurable under `[sharding]`. Throughput and latency depend heavily on hardware, transport, and payload shape (`messages-per-batch * message-size`). Run the benchmarks on your own hardware, or browse current, dated results on the [benchmarking platform](https://benchmarks.iggy.apache.org). ## Prepare the host and topic policies diff --git a/content/docs/server/configuration.mdx b/content/docs/server/configuration.mdx index a5982da1..a550393b 100644 --- a/content/docs/server/configuration.mdx +++ b/content/docs/server/configuration.mdx @@ -32,7 +32,7 @@ Configuration is resolved in three layers. Later layers win: 2. **Config file**: the path in `IGGY_CONFIG_PATH`, or `core/server/config.toml` resolved against the current working directory. A missing file is only a warning. The server continues on embedded defaults. 3. **Environment variables**: any `IGGY_`-prefixed override. -Before the environment is read, the server loads a `.env` file from the working directory, or from the path named by `IGGY_ENV_PATH`. +Before the environment is read, the server loads a `.env` file from the working directory or its parents, or from the path named by `IGGY_ENV_PATH`. Existing environment values take precedence over the `.env` file. After boot the server writes the effective configuration, including the addresses it actually bound, to `{path}/runtime/current_config.toml` with the default runtime subdirectory. @@ -41,15 +41,15 @@ After boot the server writes the effective configuration, including the addresse Every configuration key can be overridden with an `IGGY_` variable. The name is the TOML path, uppercased, with dots turned into underscores: ```bash -IGGY_TCP_ADDRESS=0.0.0.0:8090 # [tcp] address -IGGY_NODE_ADVERTISED_ADDRESS=iggy-1 # [node] advertised_address -IGGY_HTTP_ENABLED=true # [http] enabled -IGGY_PATH=/var/lib/iggy # root path -IGGY_LOGGING_LEVEL=debug # [logging] level -IGGY_SHARDING_CPU_ALLOCATION=4 # [sharding] cpu_allocation +export IGGY_TCP_ADDRESS=0.0.0.0:8090 # [tcp] address +export IGGY_NODE_ADVERTISED_ADDRESS=iggy-1 # [node] advertised_address +export IGGY_HTTP_ENABLED=true # [http] enabled +export IGGY_PATH=/var/lib/iggy # root path +export IGGY_LOGGING_LEVEL=debug # [logging] level +export IGGY_SHARDING_CPU_ALLOCATION=4 # [sharding] cpu_allocation ``` -Two variables live outside the config schema: `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` set the root credentials, always as a pair. **Only the first creation** of the root user reads them. On an existing data directory the stored root user is recovered unchanged. +Two variables live outside the config schema: `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` set the root credentials, always as a pair. They initialize the root user **only at first creation**. On an existing data directory the stored root user is recovered unchanged, but supplied environment credentials must still pass validation. ### Secrets @@ -57,6 +57,7 @@ Five values are secret-flagged: `http.jwt.encoding_secret`, `http.jwt.decoding_s ### Validation at boot +- Array-valued environment settings replace the corresponding TOML array. For an indexed array such as `cluster.nodes`, supply every entry and its required fields through environment variables; a single field override does not merge into the existing TOML roster. - Unknown top-level TOML fields and unknown server `IGGY_` environment names **reject startup**. Some nested tables can still ignore unknown fields, so compare the effective configuration with the intended settings. - The old `[system]` table and its environment mappings are rejected. Removed placeholders such as `archive_expired` and `recreate_missing_state` must be removed, not set to `false`. - Several sections validate relationships between keys (QUIC windows, sharding shutdown budgets, metadata journal sizing, partition transfer floors). A violation aborts boot with an error naming the keys. The constraints are listed with their sections below. @@ -64,12 +65,12 @@ Five values are secret-flagged: `http.jwt.encoding_secret`, `http.jwt.decoding_s ## Command-line flags -`iggy-server` accepts exactly three flags: +`iggy-server` accepts these three startup options, plus `--help` (`-h`) and `--version` (`-V`): | Flag | Description | |------|-------------| | `--fresh`, `-f` | Delete the configured data directory (`local_data` by default, see `IGGY_PATH`) before boot and start on empty state. In cluster mode this wipes **this replica only**; it rejoins and refills by state transfer from the others. Wiping a quorum at the same time can destroy committed data. Do not put `--fresh` in a service unit: it would re-transfer the whole dataset on every restart. | -| `--with-default-root-credentials` | Set `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` to `iggy` unless they are already present in the environment. Only the first creation of the root user reads these values. Development only. | +| `--with-default-root-credentials` | Set `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` to `iggy` unless they are already present in the environment. These values initialize the root user only at first creation. Development only. | | `--replica-id <N>` | Identify this node within `cluster.nodes`. Required when `cluster.enabled = true`; the value must match exactly one `replica_id` in the roster. | ## Relocated configuration keys @@ -109,8 +110,8 @@ The tables below list every section with its shipped defaults. In cluster mode, |-----|---------|-------------| | `enabled` | `true` | Serve the HTTP REST API. | | `address` | `"127.0.0.1:3000"` | Bind address and port. | -| `max_request_size` | `"2 MB"` | Maximum request body size. | -| `web_ui` | `false` | Serve the embedded Web UI at `/ui`. Requires a server built with the `iggy-web` feature; without it, `true` logs a warning and the server continues. | +| `max_request_size` | `"2 MB"` | Maximum request body size, at most `"256 MiB"`. Keep it at or below `message_bus.max_message_size` for multi-replica topics so an admitted batch fits replica frames. | +| `web_ui` | `false` | Serve the embedded Web UI at `/ui`. Requires the `iggy-web` feature and static assets built with `npm --prefix web ci` and `npm --prefix web run build:static` before compiling the server. Without the feature, `true` logs a warning; missing assets return 404. | HTTP sessions hold live server-side session state (they count against `metadata.clients_table_max`). Consumer group management (create, get, delete) is available over HTTP. Group membership (join, leave) is not and needs a stateful transport. @@ -140,17 +141,17 @@ In cluster mode, followers forward control-plane requests (streams, topics, user | `access_token_expiry` | `"1 h"` | Access token lifetime. | | `clock_skew` | `"5 s"` | Tolerance for clock differences during validation. | | `not_before` | `"0 s"` | Time before which a token is not valid. | -| `encoding_secret` | `""` (empty) | Signing key. Empty means a secure random secret is generated on each server start. | -| `decoding_secret` | `""` (empty) | Verification key. Same empty-means-random behavior. | +| `encoding_secret` | `""` (empty) | Signing key. If only one secret is configured, it is used for both signing and verification. See the fallback rules below. | +| `decoding_secret` | `""` (empty) | Verification key. If both secrets are set, they are used as supplied and must agree for self-issued tokens to verify. | | `use_base64_secret` | `false` | Treat the configured secrets as base64-encoded. | -There is **no default secret**. With the secrets left empty, each server start mints a random signing key, so issued tokens **do not survive a restart** and are valid only on the node that issued them. If you configure a secret, set it through the environment (`IGGY_HTTP_JWT_ENCODING_SECRET` / `IGGY_HTTP_JWT_DECODING_SECRET`), never commit it, and use a long random value. +There is **no default secret**. When both secrets are empty and cluster authentication supplies no key, each server start mints a random signing key, so issued tokens **do not survive a restart** and are valid only on the node that issued them. If you configure a secret, set it through the environment (`IGGY_HTTP_JWT_ENCODING_SECRET` / `IGGY_HTTP_JWT_DECODING_SECRET`), never commit it, and use a long random value. -In cluster mode the secret has an extra role: a configured secret, identical on every node, makes bearer tokens valid cluster-wide and activates follower-to-primary HTTP forwarding. With `cluster.auth` enabled the JWT key is instead derived from the cluster PSK. Without either, tokens are node-local and forwarding stays disabled. +In cluster mode the secret has an extra role: a configured secret, identical on every node, makes bearer tokens valid cluster-wide and activates follower-to-primary HTTP forwarding. When both JWT secrets are empty and `cluster.auth` is enabled, the JWT key is derived from the cluster PSK. Explicit JWT secrets take precedence. Without either, tokens are node-local and forwarding stays disabled. #### `[[http.jwt.trusted_issuers]]` -Opt-in trust of external token issuers for application-to-application authentication. With none configured, the listener accepts only self-issued HS256 tokens. +Opt-in trust of external token issuers for application-to-application authentication. With none configured, the listener accepts only self-issued tokens using the configured HMAC algorithm (`HS256`, `HS384`, or `HS512`). ```toml [[http.jwt.trusted_issuers]] @@ -171,7 +172,7 @@ Enabling an issuer opens an outbound JWKS fetch that is reachable before a token | `enabled` | `true` | Expose Prometheus metrics. | | `endpoint` | `"/metrics"` | Metrics path. Must start with `/`. | -The metrics route **requires authentication** like every other read: a missing or invalid bearer credential is rejected with 401. Any authenticated user may scrape. There's no extra RBAC rule. Scrapers present a JWT or a personal access token as the bearer: +The metrics route **requires authentication**: a missing or invalid bearer credential is rejected with 401. Any authenticated user may scrape. There's no extra RBAC rule. Scrapers present a JWT or a personal access token as the bearer: ```yaml scrape_configs: @@ -288,8 +289,8 @@ Set `path` before any TOML table header. `[runtime] path = "runtime"` and `[logg | `path` | `"logs"` | Log directory, relative to the root `path`. | | `level` | `"info"` | Filter directive in `RUST_LOG` syntax: simple levels or directives like `"warn,server=debug,iggy=trace"`. The `RUST_LOG` environment variable always takes precedence. | | `file_enabled` | `true` | Write logs to file as well as stdout. | -| `max_file_size` | `"500 MB"` | Size at which a log file rotates. `0` means one unbounded file, which disables size-based rotation. | -| `max_total_size` | `"4 GB"` | Total log budget; oldest files are deleted first. `0` means unlimited archives. Time-based rotation still applies. | +| `max_file_size` | `"500 MB"` | Size at which a log file rotates. `0` disables both size-based and hourly rotation. | +| `max_total_size` | `"4 GB"` | Total log budget; oldest files are deleted first. `0` disables the total-size cleanup limit; the rolling appender still caps archives at 100000. Retention and, when `max_file_size` is nonzero, hourly rotation still apply. | | `rotation_check_interval` | `"1 h"` | How often rotation status is checked. Avoid values below 1 s. | | `retention` | `"7 days"` | How long log files are kept. Avoid values below 1 s. | @@ -304,8 +305,8 @@ Set `path` before any TOML table header. `[runtime] path = "runtime"` and `[logg | Key | Default | Description | |-----|---------|-------------| -| `enabled` | `true` | Use the pre-allocated buffer pool. | -| `size` | `"4 GiB"` | Total pool memory. Minimum 512 MiB; must be a multiple of 4096 (page size). | +| `enabled` | `true` | Reuse aligned buffers through the memory pool. Buffers are allocated on demand. | +| `size` | `"4 GiB"` | Pool allocation budget, not a process memory limit. When the pool cannot supply a buffer within this budget, allocation continues outside it. Minimum 512 MiB; must be a multiple of 4096 (page size). | | `bucket_capacity` | `8192` | Maximum buffers per bucket. Must be a power of two; minimum 128. | The pool has 28 buckets with buffer sizes from 4 KiB to 512 MiB. @@ -314,7 +315,7 @@ The pool has 28 buckets with buffer sizes from 4 KiB to 512 MiB. | Key | Default | Description | |-----|---------|-------------| -| `cleaner_enabled` | `true` | Run the segment cleaner. It deletes the oldest **sealed** segments of topics with a finite `message_expiry` or `max_topic_size`, per partition, best-effort. The active segment is never touched. | +| `cleaner_enabled` | `true` | Run the segment cleaner. It deletes the oldest **sealed** segments of topics with a finite `message_expiry` or `max_topic_size`, per partition, best-effort. The active segment is never touched, and stored consumer/group offsets can hold back deletion. | | `interval` | `"1 m"` | Cleaner run interval. | ### `[heartbeat]` @@ -324,9 +325,9 @@ The pool has 28 buckets with buffer sizes from 4 KiB to 512 MiB. | `enabled` | `true` | Verify client heartbeats. | | `interval` | `"30 s"` | Expected heartbeat interval. | -When enabled, a connection that sends nothing (no request, no PING) for 1.2 x `interval` (36 s at the default) **and** still holds a consumer group membership has its session released, so its groups rebalance off it. A connection holding no membership is left alone and reaped when its socket closes. +When enabled, a connection that sends nothing (no request, no PING) for 1.2 x `interval` (36 s at the default) **and** still holds a consumer group membership has its session released, so its groups rebalance off it. The verifier checks once per `interval`, so eviction can occur later than the staleness threshold. A connection holding no membership is left alone and reaped when its socket closes. -The Rust, Go, Python, Node, and async Java SDKs ping automatically every 5 s, well inside the staleness window, but only from a connected high-level client: the Rust and async Java pingers are armed by `connect()`, so a session that logs in without it never pings. The blocking Java and C# SDKs have **no automatic heartbeat**, only a manual ping. Wherever nothing pings, an idle consumer group member is evicted, and only the application can keep it alive (ping) or bring it back (reconnect). +The Rust, Go, Python, Node, async Java, and C# TCP clients ping automatically every 5 s by default, well inside the staleness window. Call `connect()` to start the heartbeat in the Rust, Go, Python, and async Java high-level clients; logging in without it does not start their pingers. The blocking Java client has **no automatic heartbeat**, only a manual ping. Wherever nothing pings, an idle consumer group member is evicted, and only the application can keep it alive (ping) or bring it b [...] ### `[telemetry]` @@ -345,7 +346,8 @@ The Rust, Go, Python, Node, and async Java SDKs ping automatically every 5 s, we |-----|---------|-------------| | `cpu_allocation` | `"numa:auto"` | Number of shards and their CPU affinity. See syntaxes below. | | `pin_cores` | `true` | Pin shard threads to dedicated cores, drawn from the process's allowed CPU set (cooperates with systemd `AllowedCPUs=` and container cpusets). Set `false` on hosts where the server shares cores with other workloads. | -| `inbox_capacity` | `1024` | Per-shard inter-shard inbox capacity. Bounded by design; size for the consensus working set plus peak client-reply fan-out. Raising `[metadata]` or `[partition]` `prepare_queue_depth` raises the capacity needed here. | +| `inbox_capacity` | `1024` | Per-shard inbox capacity for consensus, connection setup, and reconciliation. Raising `[metadata]` or `[partition]` `prepare_queue_depth` raises the capacity needed here. | +| `reply_inbox_capacity` | `1024` | Separate per-shard channel for forwarded client replies. Size for peak reply fan-out; dropped replies have no bus retransmission. | | `shutdown_drain_timeout` | `"10 s"` | Per-shard bus drain budget on shutdown. Slow-fsync hosts may need more. | | `shutdown_poll_interval` | `"50 ms"` | Poll cadence for the shutdown flag. Must be less than or equal to `shutdown_drain_timeout`. | | `shutdown_join_timeout` | `"30 s"` | Hard deadline for joining shard threads at exit; a wedged shard is abandoned with an error log. Must be at least `shutdown_drain_timeout`. | @@ -377,9 +379,12 @@ Partition storage and consensus tunables share this table. Unlike `[metadata]` ( | Key | Default | Description | |-----|---------|-------------| -| `prepare_queue_depth` | `32` | Uncommitted produce and consumer-offset ops in flight per partition. Submits past it spill into a request queue of twice this depth; once both are full the server drops the request without a reply and the client retries on its own timeout. Must be between 1 and 127. | +| `prepare_queue_depth` | `32` | Uncommitted produce and consumer-offset ops in flight per partition. Submits past it spill into a request queue of twice this depth; once both are full the server rejects the request with `TransientNotAccepted`. A lost rejection reply can still make the client wait for its timeout. Must be between 1 and 127. | | `validate_checksum` | `true` | Re-hash batches read from segment storage and report a mismatch instead of serving them. | | `wal_bytes_max` | `"256 MiB"` | Active WAL plus queued/in-flight prepare budget per multi-replica partition when either topic policy is `persisted`. A 4 KiB multiple, from 128 MiB + 8 KiB through 4 GiB. Checkpointing reclaims history only after materialized state is synchronized. Temporary rewrites require extra disk space. Environment override: `IGGY_PARTITION_WAL_BYTES_MAX`. | +| `dedup_clients_max` | `4096` | Client request watermarks retained per partition to deduplicate retries. At capacity, the client with the oldest latest commit is evicted and loses that coverage. Between 1 and 65536. | +| `consumer_offsets_max` | `4096` | Durable offset keys admitted per partition, counted separately for standalone consumers and groups. Existing keys remain writable at the limit; new keys are rejected with `TooManyConsumerOffsets`. Between 1 and 262144. | +| `offset_reservation_lease` | `65536` | Offsets reserved ahead in the superblock for single-replica partitions. Crash recovery skips the unused reservation to avoid reusing acknowledged offsets. Multi-replica groups ignore it. Between 1 and 16777216. | | `evicted_ring_capacity` | `4096` | Entries retained per multi-replica partition for journal repair after a peer rejoins. Must be between 1 and 65536. Single-replica partitions retain nothing. | | `evicted_ring_bytes_max` | `"16 MiB"` | Byte ceiling for the evicted ring; whichever ring cap trips first evicts. At most `"256 MiB"`. | | `transfer_served_cache_bytes_max` | `"2176 MiB"` | Byte budget, **per shard**, for segment payloads kept resident to serve state-transfer chunk requests. The default fits two sealed segments at the 1 GiB ceiling, each with one max-message overshoot. Serving concurrency is `floor(this / max(transfer_artifact_bytes_max, 1 GiB + 64 MiB))`, minimum one; boot warns when it drops below two. At most `"64 GiB"`. | @@ -392,7 +397,7 @@ Tunables for the internal bus that ships consensus traffic between replicas and | Key | Default | Description | |-----|---------|-------------| | `max_batch` | `256` | Messages coalesced into one `writev(2)` call. Hard upper bound 512 (`IOV_MAX/2` on Linux). | -| `max_message_size` | `"64 MiB"` | Wire-level cap on a single framed message. Coupled to `partition.transfer_artifact_bytes_max` and `websocket.max_message_size` (see those keys). | +| `max_message_size` | `"64 MiB"` | Wire-level cap on a single framed message, at most `"256 MiB"`. Values above `"64 MiB"` exceed the Go SDK's frame limit. Coupled to `partition.transfer_artifact_bytes_max` and `websocket.max_message_size` (see those keys). | | `peer_queue_capacity` | `256` | Bound on the per-peer queue. `cluster.repair_chunk_max` must stay strictly below it. | | `reconnect_period` | `"5 s"` | Interval between outbound reconnect attempts to peers. | | `close_peer_timeout` | `"2 s"` | Per-peer close drain budget before force-cancellation. | @@ -437,12 +442,15 @@ Cluster mode is configured here but documented in [Clustering](/docs/clustering/ | `request_start_view_retransmit_interval` | `"1s"` | Re-request cadence for the current view's StartView. | | `view_probe_attempts_max` | `5` | Unanswered probes a recovering replica tolerates before electing on its recovered log. Between 1 and 100. | | `repair_retry_interval` | `"1s"` | Re-request cadence for a stalled journal-repair stream. | +| `repair_gap_debounce_interval` | `"1s"` | How long a committed journal gap waits before repair starts, floored at 500 ms. Nonzero. Separate from retries of an already-open repair stream. | | `repair_chunk_max` | `128` | Prepares served per repair round. Must stay strictly below `message_bus.peer_queue_capacity`. Between 1 and 1024. | +| `superblock_wedged_fatal_timeout` | `"2m"` | Exit if a metadata or partition superblock remains unwritable past this window. `"0"` leaves it fenced indefinitely; nonzero values must be at least `"30s"`. | Durations are **rounded down** to the consensus 10 ms tick. Values under 10 ms become one tick. Sub-sections: - `[cluster.auth]`: replica-to-replica authentication (PSK plus BLAKE3 keyed-MAC handshake). `shared_secret` must be at least 32 bytes of CSPRNG output, identical on every node. Prefer `IGGY_CLUSTER_AUTH_SHARED_SECRET`. `previous_shared_secret` enables rolling key rotation. +- `[cluster.coordinator]`: `skip_shard_zero_for_replicas = true` and `skip_shard_zero_for_clients = false` control connection placement when more than one shard runs. - `[cluster.tls]`: TLS 1.3 on the replica port. Requires `cluster.auth.enabled`. The PSK authenticates the peer, TLS supplies confidentiality. -- `[[cluster.nodes]]`: the full roster, byte-identical on every node. Each entry has `name`, `ip`, `replica_id`, and `ports` (`tcp`, `quic`, `http`, `websocket`, `tcp_replica`). In cluster mode `ports` is the single source of listener ports: every enabled transport needs an explicit per-node port or the server refuses to start. `advertised_address` and per-CIDR `advertised_addresses` selectors control what clients are told to dial. +- `[[cluster.nodes]]`: the full roster, byte-identical on every node. Each entry has `name`, `ip`, `replica_id`, and `ports` (`tcp`, `quic`, `http`, `websocket`, `tcp_replica`). In cluster mode `ports` is the single source of listener ports: `tcp` and `tcp_replica` ports are always required, and every other enabled transport also needs an explicit per-node port or the server refuses to start. `advertised_address` and per-CIDR `advertised_addresses` selectors control what clients are told [...] diff --git a/content/docs/server/docker.mdx b/content/docs/server/docker.mdx index c46b89a3..f39ce495 100644 --- a/content/docs/server/docker.mdx +++ b/content/docs/server/docker.mdx @@ -5,7 +5,7 @@ description: "Run the Iggy server from the official Docker images, and deploy it ## Docker -You can easily run the Iggy server with Docker - the official images can be found [here](https://hub.docker.com/r/apache/iggy), simply type `docker pull apache/iggy`. +You can easily run the Iggy server with Docker - the official images can be found [here](https://hub.docker.com/r/apache/iggy), use `docker pull apache/iggy:edge` while preparing for 0.9.0, or `apache/iggy:0.9.0` once released. These properties of the published image matter for deployment: @@ -20,7 +20,7 @@ The examples use a permissive syscall profile and unlimited locked memory for de ```yaml services: iggy: - image: apache/iggy:latest + image: apache/iggy:edge container_name: iggy restart: unless-stopped cap_add: @@ -59,7 +59,7 @@ docker run -d --name iggy \ -e IGGY_NODE_ADVERTISED_ADDRESS=localhost \ -p 8090:8090 -p 3000:3000 \ -v iggy:/app/local_data \ - apache/iggy:latest + apache/iggy:edge ``` ### Why these capabilities? @@ -105,17 +105,17 @@ Helm charts for Kubernetes deployment are available in the [repository](https:// ### Quick start ```bash -helm install iggy ./helm/charts/iggy +helm install iggy ./helm/charts/iggy --set server.image.tag=edge ``` ### Chart components -- **Server Deployment** - runs `apache/iggy` with the pod security context the server needs: seccomp profile `Unconfined` (for `io_uring`) plus the `IPC_LOCK` capability (for memory locking). Listener addresses are set to `0.0.0.0` via `server.env`, and the data volume mounts at `/app/local_data`. The chart supplies `IGGY_NODE_ADVERTISED_ADDRESS` as the in-cluster Service DNS name; override it with `server.advertisedAddress` when clients arrive through a LoadBalancer or an Ingress. Image [...] -- **Server Service** - exposes the `http` (3000), `quic` (8080), and `tcp` (8090) ports. WebSocket is not exposed by the chart. +- **Server Deployment** - runs `apache/iggy` with the pod security context the server needs: seccomp profile `Unconfined` (for `io_uring`) plus the `IPC_LOCK` capability (for memory locking). Listener addresses are set to `0.0.0.0` via `server.env`, and the data volume mounts at `/app/local_data`. The chart supplies `IGGY_NODE_ADVERTISED_ADDRESS` as the in-cluster Service DNS name; override it with `server.advertisedAddress` when clients arrive through a LoadBalancer or an Ingress. The s [...] +- **Server Service** - exposes the `http` (3000), `quic` (8080/UDP), `tcp` (8090), and `websocket` (8092) ports. - **Web UI Deployment + Service** - a separate `apache/iggy-web-ui` deployment on port 3050, enabled by default (`ui.enabled`). -- **Secret** - root user credentials from `server.users.root` (default `iggy`/`changeit`). Point `existingSecret` at your own Secret in production. +- **Secret** - root user credentials from `server.users.root` (default `iggy`/`changeit`). Point `server.users.root.existingSecret.name` at your own Secret in production; `usernameKey` and `passwordKey` select its keys. - **PersistentVolumeClaim** - storage for `/app/local_data`, **disabled by default** (`server.persistence.enabled`), 8Gi when enabled. -- **ServiceAccount**, **HPA**, **Ingress** - the usual optional plumbing, for both server and UI. +- **ServiceAccount** and **Ingress** - a shared ServiceAccount and separate optional server/UI ingresses. The chart has no HPA template. Server `replicaCount` must not exceed 1; cluster mode uses one release per node, each with its own replica ID and storage. - **ServiceMonitor** - Prometheus scrape config (optional). The `/metrics` endpoint requires a bearer credential like every other read, so wire a token (e.g. a personal access token stored in a Secret) into `server.serviceMonitor.authorization` - an unauthenticated scrape gets 401. ### Key values @@ -131,11 +131,12 @@ server: advertisedAddress: "" image: repository: apache/iggy - tag: "0.7.0" + tag: "" # Falls back to Chart.yaml appVersion ports: http: 3000 quic: 8080 tcp: 8090 + websocket: 8092 users: root: username: iggy @@ -169,8 +170,8 @@ securityContext: resources: {} ``` -Customize the values file for your environment and deploy with: +Save the excerpt as `my-values.yaml`, customize it for your environment, and deploy with: ```bash -helm install iggy ./helm/charts/iggy -f my-values.yaml +helm install iggy ./helm/charts/iggy -f my-values.yaml --set server.image.tag=edge ``` diff --git a/content/docs/server/introduction.mdx b/content/docs/server/introduction.mdx index bdac48bd..adbfbfed 100644 --- a/content/docs/server/introduction.mdx +++ b/content/docs/server/introduction.mdx @@ -3,17 +3,17 @@ title: Introduction description: "What the Iggy server does, and where its releases and Docker images are published." --- -Iggy server is the most important part of the system as it's responsible for handling all the incoming connections, managing the data and providing the API for the clients. The server is written in Rust and can be run on any platform that supports it. +Iggy server is the most important part of the system as it's responsible for handling all the incoming connections, managing the data and providing the API for the clients. The server is written in Rust. It uses `io_uring` on Linux and a polling backend on macOS. <ServerEcosystem /> -The releases are published to GitHub and can be found [here](https://github.com/apache/iggy/tags). The official Docker images can be found [here](https://hub.docker.com/r/apache/iggy), simply type `docker pull apache/iggy`. +The releases are published to GitHub and can be found [here](https://github.com/apache/iggy/tags). The official Docker images can be found [here](https://hub.docker.com/r/apache/iggy), use `docker pull apache/iggy:edge` while preparing for 0.9.0, or `apache/iggy:0.9.0` once released. -If you compile the source code in release mode, the longer compilation time comes from [LTO](https://doc.rust-lang.org/rustc/linker-plugin-lto.html) enabled in the `[profile.release]` section of the workspace [Cargo.toml](https://github.com/apache/iggy/blob/master/Cargo.toml). +If you compile the source code in release mode, linking takes longer because [LTO](https://doc.rust-lang.org/cargo/reference/profiles.html#lto) is enabled in the `[profile.release]` section of the workspace [Cargo.toml](https://github.com/apache/iggy/blob/master/Cargo.toml). ## Running the server -One `iggy-server` binary serves both the single-node and the clustered deployment. The loaded configuration decides which one you get. The server accepts three CLI flags: +One `iggy-server` binary serves both the single-node and the clustered deployment. The loaded configuration decides which one you get. The server accepts these startup flags, plus `--help` and `--version`: | Flag | Purpose | |------|---------| @@ -21,18 +21,23 @@ One `iggy-server` binary serves both the single-node and the clustered deploymen | `--with-default-root-credentials` | Set the root credentials to `iggy`/`iggy` on first start, unless `IGGY_ROOT_USERNAME`/`IGGY_ROOT_PASSWORD` are already set. Development only. | | `--replica-id <N>` | Select this node's entry in `cluster.nodes`. Required when `cluster.enabled = true`. See [Clustering](/docs/clustering/vsr). | -Configuration comes from the TOML file named by `IGGY_CONFIG_PATH`. Without one, the server boots on the defaults embedded in the binary. Any key can be overridden with an `IGGY_`-prefixed environment variable, and a `.env` file in the working directory (or the file named by `IGGY_ENV_PATH`) is loaded at startup. See [Configuration](/docs/server/configuration) for the full reference. +Configuration comes from the TOML file named by `IGGY_CONFIG_PATH`, or `core/server/config.toml` relative to the working directory. If that file is missing, the server uses the defaults embedded in the binary. Any key can be overridden with an `IGGY_`-prefixed environment variable, and a `.env` file in the working directory or its parents (or the file named by `IGGY_ENV_PATH`) is loaded at startup. See [Configuration](/docs/server/configuration) for the full reference. -When no root credentials are provided on the very first start, the server generates a random root password and prints it to the log. That's the *only* time it can be read. +When no root credentials are provided on the first single-node start, the server generates a random root password and prints it to the log. That's the *only* time it can be read. The HTTP API endpoints can be found in [server.http](https://github.com/apache/iggy/blob/master/core/server/server.http) file, which can be used with [REST Client](https://marketplace.visualstudio.com/items?itemName=humao.rest-client) extension for VS Code. In order to see the detailed logs from the server, run it with `RUST_LOG=trace` environment variable. -To seed the example data, start the server with known credentials, then run the seeder from the root of the repository: +To seed the example data, start a development server with known credentials from the root of the repository: ```bash cargo run --bin iggy-server -- --fresh --with-default-root-credentials +``` + +In another terminal at the repository root: + +```bash cargo run --bin data-seeder-tool ``` @@ -40,4 +45,4 @@ The seeder logs in as `iggy`/`iggy` by default. Pass `--username` and `--passwor ## Authentication -Only the ping liveness probe and the login handshake itself (username/password, personal access token, or HTTP token refresh, all of which prove a credential) are served without an authenticated session. Every other request requires one and is subject to [permissions](/docs/server/security): fetching server stats, for example, needs the `read_servers` permission (the root user has it), and even the Prometheus `/metrics` scrape **must present a bearer credential**. A session is created by [...] +For broker API commands, only the ping liveness probe and the login handshake itself (username/password, personal access token, or HTTP token refresh, all of which prove a credential) are served without an authenticated session. Every other request requires one and is subject to [permissions](/docs/server/security): fetching server stats, for example, needs the `read_servers` permission (the root user has it), and even the Prometheus `/metrics` scrape **must present a bearer credential** [...] diff --git a/content/docs/server/networking.mdx b/content/docs/server/networking.mdx index 9deb6b91..7fcd92c4 100644 --- a/content/docs/server/networking.mdx +++ b/content/docs/server/networking.mdx @@ -22,7 +22,7 @@ Every request and reply on the stateful transports starts with a fixed 256-byte ## Connection handling across shards -The server runs one shard per core (thread-per-core, shared-nothing). All listeners (every client transport) bind on shard 0 only. Shard 0's coordinator hands accepted plaintext TCP and WebSocket connections to peer shards round-robin by transferring the file descriptor at accept time. From then on the connection lives entirely on its shard. QUIC, TCP with TLS, and HTTP terminate on shard 0, because their per-connection state cannot move between threads. Sockets **never migrate** after a [...] +The server runs one thread and runtime per selected shard (thread-per-core, shared-nothing), with CPU selection and optional pinning controlled by `[sharding]`. All listeners (every client transport) bind on shard 0 only. Shard 0's coordinator hands accepted plaintext TCP and WebSocket connections to peer shards round-robin by transferring the file descriptor at accept time. From then on the connection lives entirely on its shard. QUIC, TCP with TLS, WebSocket with TLS, and HTTP terminat [...] ## TCP @@ -64,7 +64,7 @@ key_file = "" A few of these are deliberate rather than arbitrary: -- `max_concurrent_bidi_streams = 1`: the SDK opens a fresh stream per command and the server accepts one at a time, so a connection carries one in-flight command. Raising the cap does **not** make handlers concurrent. +- `max_concurrent_bidi_streams = 1`: the Rust SDK opens a fresh stream per command and the server accepts one at a time, so a connection carries one in-flight command. Raising the cap does **not** make handlers concurrent. - `initial_mtu = "1200 B"` is the QUIC minimum. MTU discovery probes upward automatically, and boot rejects values below 1200. - `stream_receive_window` is the per-stream slice of `receive_window`, equal to it because there is a single stream. Lower it if you ever raise the stream cap. - `keep_alive_interval` is one third of `max_idle_timeout`, so two consecutive lost keep-alives fit before the idle timer closes the connection. @@ -73,20 +73,20 @@ A few of these are deliberate rather than arbitrary: The HTTP API is built on [axum](https://github.com/tokio-rs/axum) and provides a standard REST interface. It includes JWT authentication, CORS configuration, and optional TLS. -HTTP is the most accessible protocol but has the highest overhead due to JSON serialization and the stateless nature of HTTP (no persistent connections for consumer groups). You can find all the available endpoints in the [server.http](https://github.com/apache/iggy/blob/master/core/server/server.http) file. +HTTP is the most accessible protocol but has the highest overhead due to JSON serialization and the stateless nature of HTTP (no persistent connections for consumer groups). You can find request examples in the [server.http](https://github.com/apache/iggy/blob/master/core/server/server.http) file. The HTTP server also hosts: -- **Prometheus metrics** at `/metrics` (endpoint configurable via `[http.metrics]`). The route authenticates like every other read: scrapers must present a bearer credential (JWT or personal access token). `/ping` is the *only* route that requires no credential at all. -- **Embedded Web UI** at `/ui`, which requires `web_ui = true` in `[http]` (**off by default**) on a server built with the `iggy-web` feature (part of the default build). +- **Prometheus metrics** at `/metrics` (endpoint configurable via `[http.metrics]`). The route authenticates like every other read: scrapers must present a bearer credential (JWT or personal access token). The `/ping` API route, CORS preflight responses, and enabled `/ui` static assets are public. Login and refresh routes prove credentials in the request body. +- **Embedded Web UI** at `/ui`, which requires `web_ui = true` in `[http]` (**off by default**) on a server built with the `iggy-web` feature (part of the default build) and the Web UI static assets. Build the assets before compiling the server; see [Configuration](/docs/server/configuration). ## WebSocket -WebSocket provides bidirectional streaming over HTTP upgrade. Iggy uses its own `compio-ws` implementation that bridges tungstenite's poll-based model with compio's completion-based I/O, reading frames through a buffered stream with a 128 KiB base buffer that can grow to a 64 MiB cap. +WebSocket provides bidirectional streaming over HTTP upgrade. Iggy uses the `compio-ws` crate to connect tungstenite framing to compio's completion-based I/O. The default WebSocket read buffer is 128 KiB and the default maximum message size is 64 MiB; these are separate settings. The `[websocket]` section exposes the frame-tuning knobs (`read_buffer_size`, `write_buffer_size`, `max_write_buffer_size`, `max_message_size`, `max_frame_size`, `accept_unmasked_frames`). See [Configuration](/docs/server/configuration). TLS for WebSocket has its own `[websocket.tls]` section. -**Benchmark comparison** (AWS i3en.3xlarge, fsync per message, 4 producers, 40M messages): +**Historical benchmark comparison** from the [WebSocket implementation article](/blogs/2025/11/17/websocket-io-uring) (AWS i3en.3xlarge, 4 producers or consumers, 40M messages, 1,000 messages per batch, fsync enabled). These measurements predate 0.9.0: - Producer avg latency: TCP 2.61ms vs WebSocket 3.43ms (+31%) - Consumer avg latency: TCP 0.70ms vs WebSocket 1.44ms (+106%) @@ -100,9 +100,9 @@ Each transport configures TLS in its own section, and the `self_signed` semantic | TCP | `[tcp.tls]` | `self_signed = true` generates an ephemeral certificate only while `cert_file` does not exist; an existing PEM pair is loaded instead. | | WebSocket | `[websocket.tls]` | Same load-or-generate rule as TCP. | | QUIC | `[quic.certificate]` | TLS is mandatory (part of the QUIC spec). `self_signed = true` **always** generates an ephemeral certificate and ignores `cert_file`/`key_file`, logging a warning if the files exist. | -| HTTP | `[http.tls]` | No `self_signed` option: HTTPS requires real `cert_file`/`key_file`. | +| HTTP | `[http.tls]` | No `self_signed` option: HTTPS requires certificate and key files in `cert_file`/`key_file`. | -With `self_signed = false`, `cert_file` and `key_file` must both exist. For production, provide proper certificate files everywhere. Ephemeral certificates **change on every start** and cannot be verified by clients. +With `self_signed = false`, `cert_file` and `key_file` must both exist. For production, provide proper certificate files everywhere. Ephemeral certificates **change on every start**, so a trust configuration pinned to one generated certificate does not survive a restart. ## Heartbeat @@ -112,9 +112,9 @@ enabled = true interval = "30 s" ``` -A connection that sends nothing (no request, no ping) for 1.2x the interval has its session released so its consumer groups rebalance off it. Only connections holding a consumer-group membership are evicted. Others are left alone until their socket closes. +The verifier checks once per interval and releases a session whose last heartbeat is more than 1.2x the interval old, so its consumer groups rebalance off it. Requests and pings refresh the heartbeat. Only connections holding a consumer-group membership are evicted. Others are left alone until their socket closes. -Most SDKs (Rust, Go, Python, Node, async Java) ping automatically every 5 seconds from a connected high-level client, well inside the resulting 36-second window. The blocking Java and C# SDKs have **no automatic heartbeat**, so an idle consumer-group member there must ping manually or reconnect after eviction. +The Rust, Go, Python, Node, async Java and C# TCP clients have automatic heartbeats, normally every 5 seconds once connected. This is below the default 36-second inactivity threshold; the periodic verifier means eviction does not occur at an exact 36-second deadline. The blocking Java client has no automatic heartbeat, so an idle consumer-group member must ping manually or reconnect after eviction. ## Cluster networking diff --git a/content/docs/server/security.mdx b/content/docs/server/security.mdx index fa638600..c88137f4 100644 --- a/content/docs/server/security.mdx +++ b/content/docs/server/security.mdx @@ -11,34 +11,34 @@ Iggy supports two authentication mechanisms: ### Username and password -Users authenticate with a username and password via `login_user()`. Passwords are hashed using **Argon2id** (a memory-hard hashing algorithm). On first startup, the server generates a random password for the `root` user and logs it to the console. You can override this by setting environment variables: +Users authenticate with a username and password, for example via the Rust SDK's `login_user()`. Passwords are hashed using **Argon2id** (a memory-hard hashing algorithm). On first single-node startup, the server generates a random password for the initial root account (username `iggy` by default) and writes it to the logs. A first cluster boot requires explicit root credentials. You can override this by setting environment variables: ```bash -IGGY_ROOT_USERNAME=iggy -IGGY_ROOT_PASSWORD=my-secret-password +export IGGY_ROOT_USERNAME=iggy +export IGGY_ROOT_PASSWORD=my-secret-password ``` Or use the `--with-default-root-credentials` flag for development (sets root credentials to `iggy`/`iggy`). -**Important**: once the data directory exists, environment variable credentials are **ignored**. To reset credentials, you must use the `--fresh` flag (which deletes all data). +These variables initialize the root account only when it is first created. Supplied values are still validated on later starts, but they do not replace the stored credentials. Use the password-change API or `iggy user password` with the current password to change an existing account. The `--fresh` flag deletes all data; it is not needed for a password change. The root user **cannot be deleted**, and its permissions are fixed: it always holds every permission. ### Personal Access Tokens (PAT) -PATs provide programmatic access with optional expiry. Each user can have up to `max_tokens_per_user` (default 100) active tokens. Tokens are **hashed before storage** and can be revoked at any time. +PATs provide programmatic access with optional expiry. Each user can have up to `max_tokens_per_user` (default 100) stored tokens, including expired tokens until the cleaner removes them. Tokens are **hashed before storage** and can be revoked at any time. ```bash # Create a PAT via CLI -iggy -u iggy -p secret pat create my-token 7d +IGGY_TOKEN=$(iggy --quiet -u iggy -p my-secret-password pat create my-token 7d) # Use the PAT for authentication -iggy -t my-token-value stream list +iggy -t "$IGGY_TOKEN" stream list ``` An automatic cleaner removes expired tokens at a configurable interval. -Only the ping liveness probe and the login endpoints themselves are served without authentication. **Everything else** - including the Prometheus `/metrics` scrape - requires an authenticated session or bearer credential. +Broker API commands require an authenticated session or bearer credential, except for ping and the login/refresh flows that establish or prove a credential. The Prometheus `/metrics` scrape requires a JWT or personal access token. HTTP CORS preflight responses and embedded `/ui` static assets are public; the UI's broker API calls still require authentication. ## Authorization @@ -70,7 +70,7 @@ The global permissions and the operations they unlock: Two kinds of implication apply on top of the table: -- **Supersets**: every `manage_*` permission includes its `read_*` counterpart. In addition `manage_streams` includes `manage_topics`, `read_streams` includes `read_topics`, and `read_topics` includes `poll_messages`. +- **Supersets**: every `manage_*` permission includes its `read_*` counterpart. In addition `manage_streams` includes `manage_topics`, `read_streams` includes `read_topics`, and `read_topics` includes `poll_messages`. `manage_topics` also includes `send_messages`, so `manage_streams` permits sending too. - **Self-service**: an authenticated user can always read their own account, change their own password, and manage their own personal access tokens, without any of the user permissions above. ### Scoped permissions @@ -90,7 +90,7 @@ Permissions are checked from top to bottom: global, then stream, then topic. A p If a stream has no entry in the user's stream permissions, only global permissions apply to it. The same holds for topics within a stream. -For example, a user that may only consume from stream 42 needs no global permissions at all: grant stream-scoped `read_stream` and `poll_messages` on stream 42, and the user can read that stream, list its topics, and poll messages from any topic in it. Nothing else. +For example, a user that may only consume from stream 42 needs no global permissions at all: grant stream-scoped `read_stream` and `poll_messages` on stream 42, and the user can read that stream, list its topics, and poll messages from any topic in it. The read grant also permits consumer-group operations in that stream. For polling without those read and group permissions, grant only `poll_messages`. ## Transport encryption (TLS) @@ -101,9 +101,9 @@ Each transport configures TLS in its own section, and the `self_signed` semantic | TCP | `[tcp.tls]` | `self_signed = true` generates an ephemeral certificate only while `cert_file` does not exist; an existing PEM pair is loaded instead. | | WebSocket | `[websocket.tls]` | Same load-or-generate rule as TCP. | | QUIC | `[quic.certificate]` | TLS is mandatory (part of the QUIC spec). `self_signed = true` **always** generates an ephemeral certificate and ignores `cert_file`/`key_file`, logging a warning if the files exist. | -| HTTP | `[http.tls]` | No `self_signed` option: HTTPS requires real `cert_file`/`key_file`. | +| HTTP | `[http.tls]` | No `self_signed` option: HTTPS requires certificate and key files in `cert_file`/`key_file`. | -For production deployments, provide proper certificates via `cert_file` and `key_file`. Ephemeral certificates change on every start and cannot be verified by clients. See [Networking](/docs/server/networking) for the surrounding transport configuration. +For production deployments, provide proper certificates via `cert_file` and `key_file`. Ephemeral certificates change on every start, so a trust configuration pinned to one generated certificate does not survive a restart. See [Networking](/docs/server/networking) for the surrounding transport configuration. ## Data encryption at rest @@ -130,7 +130,7 @@ clock_skew = "5 s" Further keys (`valid_issuers`, `valid_audiences`, `not_before`, `use_base64_secret`, the signing secrets) are covered in [Configuration](/docs/server/configuration). -**Signing secrets**: `encoding_secret` and `decoding_secret` default to empty, which makes the server generate a secure random secret on every start. That's a safe single-node default with two consequences: issued tokens **die on restart**, and in a cluster each node signs with its own key, so bearers are node-local and follower-to-primary request forwarding stays disabled. For clusters, configure an identical secret on every node (prefer the `IGGY_HTTP_JWT_ENCODING_SECRET`/`IGGY_HTTP_JW [...] +**Signing secrets**: `encoding_secret` and `decoding_secret` default to empty. Without `cluster.auth`, the server generates a secure random secret on every start. That's a safe single-node default with two consequences: issued tokens **die on restart**, and in a cluster each node signs with its own key, so bearers are node-local and follower-to-primary request forwarding stays disabled. For clusters, configure an identical secret on every node (prefer the `IGGY_HTTP_JWT_ENCODING_SECRET`/ [...] **Refresh tokens**: `POST /users/refresh-token` re-issues an access token from a still-valid one presented in the request body, answering the same identity shape as login, so HTTP clients can extend a session without re-sending credentials. diff --git a/content/docs/server/storage-engine.mdx b/content/docs/server/storage-engine.mdx index a9fcca05..21aa67ee 100644 --- a/content/docs/server/storage-engine.mdx +++ b/content/docs/server/storage-engine.mdx @@ -3,7 +3,7 @@ title: Storage Engine description: "The segmented append-only log, and how streams, topics, partitions and segments map onto files on disk." --- -Iggy's storage engine is built around the concept of a **segmented append-only log**. Every piece of data flows through a well-defined hierarchy: System -> Streams -> Topics -> Partitions -> Segments. This page covers how data is stored, indexed, flushed, recovered, and cleaned up on disk. +Iggy's storage engine is built around the concept of a **segmented append-only log**. Message data follows a hierarchy: System -> Streams -> Topics -> Partitions -> Segments. This page covers how data is stored, indexed, flushed, recovered, and cleaned up on disk. <StreamHierarchy /> @@ -42,7 +42,7 @@ local_data/ └── 00000000000016000000.index ``` -Stream, topic, and partition ids are numeric and **0-based**. Each partition directory holds pairs of `.log` and `.index` files. The filename is the start offset of the segment's first message, zero-padded to 20 digits. Next to the segments, every partition keeps its own superblock pair (replica identity and consensus state for that partition) and an `offsets/` tree for consumer offset storage. Multi-replica partitions also keep a prepare WAL when either topic durability policy is `persisted`. +Stream, topic, and partition ids are numeric and **0-based**. Each partition directory holds pairs of `.log` and `.index` files. The filename is the segment's start offset, zero-padded to 20 digits. Recovery can create an empty active segment at a reserved offset beyond the last stored message. Next to the segments, every partition keeps its own superblock pair (replica identity and consensus state for that partition) and an `offsets/` tree for consumer offset storage. Multi-replica part [...] ## Segmented log @@ -129,13 +129,13 @@ Disk reads are verified before they reach a consumer: validate_checksum = true ``` -With `validate_checksum = true` (the default), every batch a disk poll reads is re-hashed and compared against its stored checksum. A mismatch **fails the poll closed**, so a segment damaged at rest is reported instead of served. Setting it to `false` skips the re-hash and serves whatever decodes, which can hand a consumer bytes provably not the ones written. Only disable it with a corruption guard elsewhere in the stack. +With `validate_checksum = true` (the default), every batch a disk poll reads is re-hashed and compared against its stored checksum. A mismatch stops the disk walk and logs an error on the server. The consumer receives any valid prefix already read, or an ordinary empty poll if none was read; the protocol does not report the checksum failure to the consumer. Repeated polls can therefore wait indefinitely at damaged data. Setting it to `false` skips the re-hash and serves whatever decodes, [...] -Polls serve stored batch records as-is (a reply may be a server-sliced view of a larger stored batch), so there is no re-encoding on the read path. +Binary polls reuse the stored batch layout. A reply may slice a larger stored batch and rewrite its header, and at-rest encryption requires decryption before the reply. HTTP additionally serializes the result as JSON. ## Memory pool -Iggy includes a custom memory pool to eliminate allocation overhead on the hot path. The pool has **28 buckets** with buffer sizes from 4 KiB up to 512 MiB (non-uniform spacing, denser around common message sizes, with sizes above 2 MiB rounded to hugepage-friendly steps). Components request a buffer from the appropriate bucket and return it when done. +Iggy includes a custom memory pool to reuse buffers on the hot path. The pool has **28 buckets** with buffer sizes from 4 KiB up to 512 MiB (non-uniform spacing, denser around common message sizes, with sizes above 2 MiB rounded to hugepage-friendly steps). Components request a buffer from the appropriate bucket and return it when done. ```toml [memory_pool] @@ -144,11 +144,11 @@ size = "4 GiB" # Total pool size (minimum 512 MiB, multiple of the 40 bucket_capacity = 8192 # Buffers per bucket (power of 2, minimum 128) ``` -This avoids heap allocations during message processing and enables zero-copy message passing between internal components. +Buffers are allocated lazily. A pool miss can allocate outside the pool, so message processing is not allocation-free. Internal components can pass ownership of existing buffers without copying their contents. ## Retention and cleanup -Two independent retention policies exist, both **per-topic creation options** (see [Topic options](/docs/server/topic-options)). The segment cleaner enforces them: +Two independent retention policies exist, both **per-topic options configurable at creation or update** (see [Topic options](/docs/server/topic-options)). The segment cleaner enforces them: ```toml [data_maintenance.messages] @@ -160,7 +160,7 @@ interval = "1 m" # default **Time-based retention** (`message_expiry`): sealed segments whose newest message is older than the expiry are deleted. -Both policies can be active at once. They only ever touch sealed segments. The active segment is **never deleted**, even if its messages have expired. +Both policies can be active at once. They only ever touch sealed segments. Deletion also stops at the minimum committed consumer or consumer-group offset, and pending persistence checkpoints can defer it. The active segment is **never deleted by retention**, even if its messages have expired. ## Metadata plane diff --git a/content/docs/server/topic-options.mdx b/content/docs/server/topic-options.mdx index e32c42b8..d2833fce 100644 --- a/content/docs/server/topic-options.mdx +++ b/content/docs/server/topic-options.mdx @@ -11,23 +11,23 @@ Options are key-value pairs sent with `CreateTopic`. Unknown keys are **rejected | Option | Default | Constraints | Description | |--------|---------|-------------|-------------| -| `max_topic_size` | unlimited | | Delete the oldest sealed segments once the topic grows past this size. | -| `message_expiry` | none | | Delete sealed segments older than this. | +| `max_topic_size` | unlimited | finite values at least `segment_size` | Per-partition limit on sealed segment bytes. Delete the oldest sealed segments once that partition exceeds it. | +| `message_expiry` | none | | Delete the oldest sealed segments whose newest message timestamp is older than this duration. | | `compression_algorithm` | `none` | `none` or `gzip` | Placeholder: stored and reported, no compression applied yet. | -| `segment_size` | 1 GiB | 512-byte multiple, at least 1 MiB, at most 1 GiB | Soft size limit per segment: a segment may close one whole batch past it. | +| `segment_size` | 1 GiB | 0 selects the default; otherwise a 512-byte multiple from 1 MiB through 1 GiB | Soft size limit per segment: a segment may close one whole batch past it. | | `durability` | `replicated` | `replicated` or `persisted` | Message completion policy. `persisted` requires recoverable stable-storage copies at the replication quorum before success. | | `consumer_offset_durability` | `replicated` | `replicated` or `persisted` | Completion policy for explicit consumer-offset stores and deletes, independent of message durability. | | `messages_required_to_save` | 1024 | non-zero, at most 16777216 | Flush the journal once it holds this many messages. | -| `size_of_messages_required_to_save` | 1 MiB | at most 1 GiB | Flush the journal once it holds this many bytes. Paired with the message count; whichever threshold trips first flushes. | +| `size_of_messages_required_to_save` | 1 MiB | 0 selects the default; at most 1 GiB | Flush the journal once it holds this many bytes. Paired with the message count; whichever threshold trips first flushes. | | `preallocate_segments` | `false` | `segment_size` x partitions at most 64 GiB per create | Reserve each segment's bytes up front where the filesystem supports it. | -Both retention policies can be active at once. The active segment is **never touched**. Deletion is done by the server's segment cleaner (`[data_maintenance.messages]`, enabled by default). +Both retention policies can be active at once. The active segment is **never touched**, and its bytes are excluded from the size limit. A segment whose end offset exceeds the lowest stored consumer or consumer-group offset is retained; with no stored offsets, this barrier is absent. Deletion is done by the server's segment cleaner (`[data_maintenance.messages]`, enabled by default). Both durability policies write data to disk and default independently to `replicated`. Neither inherits the other. Flush thresholds schedule ordinary segment writes; required persistence, capacity pressure, or lifecycle operations can flush earlier. They do not weaken the `persisted` completion guarantee. Value forms are forgiving: byte sizes accept a raw number of bytes or a string like `"128 MiB"`, expiry accepts microseconds or a humantime string like `"7 days"`, booleans accept `true`/`false`. Create admission re-parses and re-encodes what you send, so a string `segment_size=128MiB` is stored as the number it names. -`preallocate_segments` reserves exactly `segment_size` of real disk per partition the moment the topic is created (and again as segments rotate). With the default 1 GiB segment size that's 1 GiB per partition up front, which is why it's opt-in and why one create is **capped at 64 GiB** of total reservation. +`preallocate_segments` requests a `segment_size` reservation for each partition's segment file when it is opened, including during topic creation and rotation. On Linux this reserves disk space without changing the file's logical length. Unsupported or failed reservations log a warning and fall back to ordinary allocation. With the default 1 GiB segment size, the request is 1 GiB per partition up front, which is why it's opt-in and why one create is **capped at 64 GiB** of requested rese [...] ## Setting options @@ -46,7 +46,7 @@ In the CLI, both durability policies have named flags. `--set` is repeatable and SDKs expose typed durability values in their topic creation options. In Rust, set `TopicCreateOptions::durability` and `TopicCreateOptions::consumer_offset_durability` to `Durability::Replicated` or `Durability::Persisted`. The HTTP API takes `"durability"` and `"consumer_offset_durability"` as string values in the create body's `options` map. -Keys absent from the wire request are resolved by the admitting server and stored as **derived** entries. Typed SDKs can send their default durability values explicitly. `GetTopic` returns explicit and derived blocks, so the effective values and the provenance of the request remain visible. +Keys absent from the wire request are resolved by the admitting server and stored as **derived** entries. Typed SDKs can send their default durability values explicitly. Binary `GetTopic` responses carry explicit and derived blocks; HTTP reports an `explicit` flag per option. Both report option values and the provenance of the request. See the update limitation below. ## Create-only vs updatable @@ -60,19 +60,25 @@ Updates are **patches**: a key you don't send keeps its current value. The storage options (`segment_size`, `durability`, `consumer_offset_durability`, `messages_required_to_save`, `size_of_messages_required_to_save`, `preallocate_segments`) are **create-only**. A topic gets them at creation and keeps them. `UpdateTopic` rejects changes to either durability policy, so select both before creating the topic. +An update that explicitly sends `server_default` (zero) for `message_expiry` or `max_topic_size` has inconsistent reporting: the server retains the previous fixed response fields but stores zero in the corresponding option entries. The CLI sends these sentinels when those update arguments are omitted. Supply explicit expiry and size values when updating a topic; do not rely on zero to reset or preserve its retention settings. + ## Discovering the catalog Ask the server which keys it accepts, with their types, defaults, and descriptions: ```bash -# CLI iggy options topic +``` -# HTTP -GET /options/topic +For HTTP, set `IGGY_TOKEN` to a valid JWT or personal access token: + +```bash +curl --fail-with-body \ + -H "Authorization: Bearer $IGGY_TOKEN" \ + http://localhost:3000/options/topic ``` -SDKs expose the same call as `describe_options`. The scope is `topic`, `stream`, or `user`. +The Rust SDK exposes the same call as `describe_options`. The scope is `topic`, `stream`, or `user`. Discovery matters most on the binary transports: TCP, QUIC, and WebSocket carry **only an error code** for a rejected key, not its name, so the catalog is how a client finds out what this server supports. Over HTTP the error message names the offending key directly. diff --git a/src/components/architecture-diagrams.tsx b/src/components/architecture-diagrams.tsx index e97563d2..02849431 100644 --- a/src/components/architecture-diagrams.tsx +++ b/src/components/architecture-diagrams.tsx @@ -280,7 +280,7 @@ export function SegmentVisualization() { <div className="my-8 rounded-xl border border-fd-border bg-fd-card p-6"> <h3 className="text-lg font-semibold text-fd-foreground m-0 mb-2">Partition Storage Layout</h3> <p className="text-xs text-fd-muted-foreground m-0 mb-5"> - Each partition contains a segmented log. Segments are sealed at 1 GiB and new ones created automatically. Click a segment to inspect its files. + Example partition at the default 1 GiB segment size. Rotation can exceed that size by one batch; message counts depend on payload and batch sizes. Click a segment to inspect its files. </p> <div className="space-y-3"> @@ -343,7 +343,7 @@ export function SegmentVisualization() { <span className="text-[11px] font-mono font-bold text-fd-foreground">.log</span> </div> <span className="text-[10px] text-fd-muted-foreground block">Message data (headers + payloads)</span> - <span className="text-[10px] text-fd-muted-foreground block">batch records, 48-byte frame per message</span> + <span className="text-[10px] text-fd-muted-foreground block">batch records, 48-byte header per message</span> </div> <div className="rounded-lg bg-fd-accent/30 p-3"> <div className="flex items-center gap-2 mb-1"> @@ -356,8 +356,8 @@ export function SegmentVisualization() { </div> <div className="flex gap-4 text-[10px] text-fd-muted-foreground"> <span>Offsets: <code className="text-fd-foreground">{seg.startOffset.toLocaleString()}</code> .. <code className="text-fd-foreground">{seg.endOffset.toLocaleString()}</code></span> - {seg.status === "sealed" && <span>Read-only, safe to archive</span>} - {seg.status === "active" && <span>Accepting writes via vectored I/O</span>} + {seg.status === "sealed" && <span>Read-only segment</span>} + {seg.status === "active" && <span>Active; flushed with vectored I/O</span>} </div> </div> )} @@ -850,7 +850,7 @@ export function ServerEcosystem() { <div className="relative rounded-2xl border-2 border-fd-primary/40 bg-gradient-to-br from-fd-primary/10 to-fd-primary/5 px-8 py-5 text-center shadow-lg shadow-fd-primary/10"> <div className="absolute -top-1 -right-1 w-3 h-3 rounded-full bg-fd-primary shard-pulse" /> <span className="text-lg font-bold text-fd-primary block">Iggy Server</span> - <span className="text-xs text-fd-muted-foreground block mt-1">Thread-per-core + io_uring</span> + <span className="text-xs text-fd-muted-foreground block mt-1">Thread-per-core + io_uring (Linux)</span> <div className="flex justify-center gap-1.5 mt-2"> {["TCP :8090", "QUIC :8080", "HTTP :3000", "WS :8092"].map((p) => ( <span key={p} className="px-1.5 py-0.5 rounded text-[8px] font-mono font-medium bg-fd-primary/15 text-fd-primary">{p}</span> @@ -1213,7 +1213,7 @@ export function DocsHero() { <a href="/docs/introduction/architecture" className="flex-1 rounded-lg border-2 border-fd-primary/30 bg-fd-primary/5 p-3 text-center no-underline group hover:border-fd-primary/50 transition-colors"> <span className="text-sm font-bold text-fd-primary block">Iggy Server</span> - <span className="text-[10px] text-fd-muted-foreground block mt-1">Thread-per-core + io_uring</span> + <span className="text-[10px] text-fd-muted-foreground block mt-1">Thread-per-core + io_uring (Linux)</span> <div className="flex justify-center gap-1 mt-2"> {["TCP", "QUIC", "WS", "HTTP"].map((p) => ( <span key={p} className="px-1.5 py-0.5 rounded text-[8px] font-mono font-bold bg-fd-primary/15 text-fd-primary">{p}</span>
