This is an automated email from the ASF dual-hosted git repository. hubcio pushed a commit to branch docs/align-with-topic-durability in repository https://gitbox.apache.org/repos/asf/iggy-website.git
commit 3dd207a500884e6783318f7ce9c4ede1a1dc0c8f Author: hubcio <[email protected]> AuthorDate: Fri Sep 11 00:47:51 2026 +0200 fix(docs): align introduction with server 0.9.0 --- content/docs/introduction/about.mdx | 46 +++++++------ content/docs/introduction/architecture.mdx | 37 ++++++----- content/docs/introduction/concepts.mdx | 34 +++++----- content/docs/introduction/getting-started.mdx | 96 ++++++++++++++++++--------- src/components/architecture-diagrams.tsx | 36 +++++----- 5 files changed, 146 insertions(+), 103 deletions(-) diff --git a/content/docs/introduction/about.mdx b/content/docs/introduction/about.mdx index 86531739..9c8129b4 100644 --- a/content/docs/introduction/about.mdx +++ b/content/docs/introduction/about.mdx @@ -15,6 +15,8 @@ The name is an abbreviation for the Italian Greyhound - small yet extremely fast  +Historical startup screenshot from server 0.5.0. See [configuration](/docs/server/configuration) for the 0.9.0 settings. + --- ### Features @@ -31,29 +33,29 @@ The name is an abbreviation for the Italian Greyhound - small yet extremely fast - Available client SDK in multiple languages - **Works directly with binary data**, avoiding enforced schema and serialization/deserialization overhead - Custom **zero-copy (de)serialization**, which greatly improves the performance and reduces memory usage -- **Custom memory pool** with 28 buckets (buffer sizes from 4 KiB to 512 MiB) for pre-allocated, zero-copy message passing -- Configurable server features (e.g. caching, segment size, data flush interval, transport protocols etc.) +- **Custom memory pool** with 28 buckets (buffer sizes from 4 KiB to 512 MiB) for on-demand allocation, buffer reuse and sharing without copying buffer contents +- Configurable server features (e.g. caching and transport protocols), plus per-topic segment size, durability and flush thresholds - Server-side storage of **consumer offsets** - Multiple ways of polling the messages: - By offset (using the indexes) - By timestamp (using the time indexes) - First/Last N messages - Next N messages for the specific consumer -- Possibility of **auto committing the offset** (e.g. to achieve *at-most-once* delivery) -- **Consumer groups** providing the message ordering and horizontal scaling across the connected clients, with cooperative partition rebalancing +- Optional **poll auto-commit**, with processing and failure semantics explained in [concepts](/docs/introduction/concepts#polling-messages) +- **Consumer groups** distributing partitions across connected clients for horizontal scaling; ordering is per partition, with cooperative partition rebalancing - **Message expiry** with auto deletion based on the configurable **retention policy** - **Multi-tenant** support via abstraction of **streams** which group **topics** - **TLS** support for all transport protocols (TCP, QUIC, WebSocket, HTTPS) - **[Connectors](/docs/connectors/introduction)** - sinks, sources and data transformations based on the **custom Rust plugins** loaded dynamically at runtime -- **[Model Context Protocol](/docs/ai/mcp)** - provide context to LLM with **MCP server** exposing 40+ tools for full Iggy management +- **[Model Context Protocol](/docs/ai/mcp)** - provide context to LLM with **MCP server** exposing 40+ tools for streaming management - Optional server-side as well as client-side **data encryption** using AES-256-GCM - Optional metadata support in the form of **message headers** - Support for **OpenTelemetry** logs & traces + Prometheus metrics -- Built-in **[CLI](/docs/cli/start)** to manage the streaming server installable via `cargo install iggy-cli` +- Built-in **[CLI](/docs/cli/start)** to manage the streaming server installable via `cargo install iggy-cli --version 0.14.0-edge.7 --locked` - Built-in **[Web UI](/docs/web_ui/start)** dashboard (Svelte) that can be embedded directly in the server binary or run as a standalone container - Built-in **benchmarking app** to test the performance -- **Single binary deployment** (no external dependencies) -- **Accept-time connection distribution** across shards via file descriptor transfer for load balancing +- **Single binary deployment** without an external broker or database; operating-system libraries are still required by dynamically linked builds +- **Accept-time connection distribution** for plaintext TCP and WebSocket across shards via file descriptor transfer - Built on **Viewstamped Replication (VSR)** consensus: runs as a single node by default, with multi-node [clustering](/docs/clustering/vsr) available via the `[cluster]` configuration ## Supported languages SDK @@ -66,49 +68,49 @@ The name is an abbreviation for the Italian Greyhound - small yet extremely fast | Python | [apache-iggy](https://pypi.org/project/apache-iggy/) | PyPI | | Node.js | [apache-iggy](https://www.npmjs.com/package/apache-iggy) | npm | | Go | [iggy-go](https://pkg.go.dev/github.com/apache/iggy/foreign/go) | pkg.go.dev | -| PHP | [apache/iggy-php](https://packagist.org/packages/apache/iggy-php) | Packagist | +| PHP | [apache/iggy-php](https://github.com/apache/iggy/tree/master/foreign/php) | GitHub (source build) | | C++ | [iggy-cpp](https://github.com/apache/iggy/tree/master/foreign/cpp) | GitHub (WIP) | ## CLI -The interactive CLI is implemented under the `cli` project, to provide the best developer experience. This is a great addition to the Web UI, especially for developers who prefer using the console tools. +The interactive CLI is implemented under `core/cli`, to provide the best developer experience. This is a great addition to the Web UI, especially for developers who prefer using the console tools. -Iggy CLI can be installed with `cargo install iggy-cli` and then simply accessed by typing `iggy` in your terminal. It supports named connection contexts (profiles) for managing multiple server connections, shell completions for bash/zsh/fish/elvish/powershell, and session-based login on Linux. +Iggy CLI can be installed with `cargo install iggy-cli --version 0.14.0-edge.7 --locked` and then simply accessed by typing `iggy` in your terminal. It supports named connection contexts (profiles) for managing multiple server connections, shell completions for bash/zsh/fish/elvish/powershell, and session-based login through platform credential stores on Linux, macOS and Windows when the default `login-session` feature is enabled. ### Web UI The Web UI provides a comprehensive dashboard for the Iggy server, built with SvelteKit and TypeScript. It can run in two modes: -- **Embedded** - compiled into the server binary (with `iggy-web` feature flag), served at the `/ui` endpoint -- **Standalone** - as a separate container via `docker pull apache/iggy-web-ui` +- **Embedded** - compiled into the server binary (with `iggy-web` feature flag), served at the `/ui` endpoint when `http.web_ui = true` +- **Standalone** - as a separate container via `docker pull apache/iggy-web-ui:edge` -Features include stream/topic/partition management, message browser with JSON/string/XML decoders, user management, server logs viewer, real-time terminal, and server configuration overview. +Features include stream/topic/partition management, a message browser with JSON/string/XML decoders, and user management. The logs and terminal pages are placeholders, and the server settings page has a disabled Save action rather than an operational configuration editor. ## Connectors Iggy provides a highly performant and modular **[runtime](/docs/connectors/runtime)** for statically typed, yet dynamically loaded connectors. You can ingest data from external sources and push the data to Iggy streams, or fetch data from Iggy streams and forward it to external systems. **Create your own Rust plugins** by simply implementing either the `Source` or `Sink` trait and **build custom pipelines for the data processing**. -There are currently 14 sinks and 4 sources available. Sinks include PostgreSQL, MongoDB, Elasticsearch, ClickHouse, InfluxDB, Apache Iceberg, Delta Lake, Quickwit and S3. Sources include PostgreSQL, Elasticsearch, InfluxDB and random data generation. See the [connectors documentation](/docs/connectors/introduction) for the full catalog. +The source tree contains 16 sinks and 4 sources. Sinks include PostgreSQL, MongoDB, Elasticsearch, ClickHouse, InfluxDB, Apache Iceberg, Delta Lake, Quickwit and S3. Sources include PostgreSQL, Elasticsearch, InfluxDB and random data generation. See the [connectors documentation](/docs/connectors/introduction) for the full catalog. -The [docker image](https://hub.docker.com/r/apache/iggy-connect) is available, and can be fetched via `docker pull apache/iggy-connect`. +The [docker image](https://hub.docker.com/r/apache/iggy-connect) is available, and can be fetched via `docker pull apache/iggy-connect:edge`. ## Model Context Protocol The [Model Context Protocol](https://modelcontextprotocol.io) (MCP) is an open protocol that standardizes how applications provide context to LLMs. The **[Iggy MCP Server](/docs/ai/mcp)** is an implementation of the MCP protocol for message streaming infrastructure. It exposes 40+ tools covering streams, topics, partitions, messages, consumer groups, users, and more. It supports both HTTP and stdio transports, making it compatible with tools like Claude Desktop and other MCP clients. -The [docker image](https://hub.docker.com/r/apache/iggy-mcp) is available, and can be fetched via `docker pull apache/iggy-mcp`. +The [docker image](https://hub.docker.com/r/apache/iggy-mcp) is available, and can be fetched via `docker pull apache/iggy-mcp:edge`. ### Docker -The official Apache Iggy images can be found on [Docker Hub](https://hub.docker.com/r/apache/iggy). Simply type `docker pull apache/iggy` to pull the image. +The official Apache Iggy images can be found on [Docker Hub](https://hub.docker.com/r/apache/iggy). These docs prepare for server 0.9.0. Use `docker pull apache/iggy:0.9.0` when that release is available, or `docker pull apache/iggy:edge` for a development image. SDK and tooling versions are independent of the server version; the CLI command above pins a published edge version. You can also find the images for all the different tooling such as Connectors, MCP Server etc. [here](https://hub.docker.com/u/apache?page=1&search=iggy). Please note that the images tagged as `latest` are based on the official, stable releases, while the `edge` ones are updated directly from latest version of the `master` branch. -You can find the `Dockerfile` and `docker-compose` in the root of the repository. To build and start the server, run: `docker compose up`. +You can find the development `Dockerfile` and `docker-compose.yml` in the root of the repository. Published server images use `core/server/Dockerfile`. To build and start the server, run: `docker compose up`. -Additionally, you can run the `CLI` which is available in the running container, by executing: `docker exec -it iggy-server /iggy`. +For the root Compose build, run the included CLI with `docker exec -it iggy-server /iggy`. In published images, its path is `/usr/local/bin/iggy`. Keep in mind that running the container on operating systems other than Linux, where the Docker is running in the VM, might result in the performance degradation. @@ -128,12 +130,12 @@ ulimits: Or when running with `docker run`: ``` -docker run --cap-add=SYS_NICE --security-opt seccomp=unconfined --ulimit memlock=-1:-1 apache/iggy:latest +docker run --cap-add=SYS_NICE --security-opt seccomp=unconfined --ulimit memlock=-1:-1 -p 8090:8090 -e IGGY_TCP_ADDRESS=0.0.0.0:8090 -e IGGY_NODE_ADVERTISED_ADDRESS=localhost apache/iggy:edge ``` ### Helm Charts -Helm charts for Kubernetes deployment are available in the [repository](https://github.com/apache/iggy/tree/master/helm/charts/iggy). The chart includes templates for Deployment, Service, ServiceAccount, HPA (Horizontal Pod Autoscaler), Ingress, PersistentVolumeClaim, ServiceMonitor (for Prometheus), and root user credentials Secret. +Helm charts for Kubernetes deployment are available in the [repository](https://github.com/apache/iggy/tree/master/helm/charts/iggy). The chart includes templates for Deployment, Service, ServiceAccount, Ingress, PersistentVolumeClaim, ServiceMonitor (for Prometheus), and root user credentials Secret. ### Versioning diff --git a/content/docs/introduction/architecture.mdx b/content/docs/introduction/architecture.mdx index f6032bca..5968a618 100644 --- a/content/docs/introduction/architecture.mdx +++ b/content/docs/introduction/architecture.mdx @@ -7,7 +7,7 @@ This page covers the internals of the Iggy server: how work is scheduled across ## How a message flows through Iggy -Before diving into the architecture details, here's the complete journey of a message from client to disk: +Before diving into the architecture details, here's an overview of a message's journey from client to disk: <MessageFlowDiagram /> @@ -19,20 +19,20 @@ Iggy uses a **thread-per-core shared nothing architecture** combined with `io_ur ### How it works -Each CPU core runs its own **shard** (an instance of `IggyShard`), pinned to a specific core via `sched_setaffinity` on Linux. Each shard has its own single-threaded `compio` async runtime, which means there is no cross-thread synchronization needed within a shard. Memory is bound to the NUMA node of the core via `hwlocality` for optimal memory access latency. +Each configured **shard** (an instance of `IggyShard`) has its own single-threaded `compio` async runtime. With `pin_cores = true`, Linux shard threads are pinned to their selected CPUs via `sched_setaffinity`. NUMA allocation modes also bind memory via `hwlocality`. Setting `pin_cores = false` disables both bindings; shared metadata and inter-shard communication still require synchronization. ### Shard roles and connection distribution Shard 0 has a special role: it binds **every listener** - the replica plane and all client transports (TCP, QUIC, WebSocket, HTTP). Connections are then spread across shards at accept time: - **Plaintext TCP and WebSocket** connections are handed off round-robin to peer shards. Shard 0's coordinator duplicates the socket's file descriptor, ships a connection-setup frame to the target shard, and drops its own handle, so the owning shard serves the connection from then on - before a single byte is read. -- **QUIC and TLS-wrapped TCP** connections terminate on shard 0, because their per-connection state cannot be moved between shards. +- **QUIC, TLS-wrapped TCP and secure WebSocket (WSS)** connections terminate on shard 0, because their per-connection state cannot be moved between shards. HTTP is also served on shard 0. All shards, **including shard 0**, own partitions and serve partition requests. ### Request routing -Requests are routed between shards using **message passing** (via `crossfire` bounded mpsc channels), which avoids locking entirely. The routing logic splits operations into two planes: +Requests are routed between shards using **message passing** (via `crossfire` bounded mpsc channels), so partition state remains on its owning shard. The routing logic splits operations into two planes: - **Metadata operations** (create/delete stream/topic/user etc.) always execute on **shard 0** - it is the only shard that commits metadata - **Partition operations** (send_messages, poll_messages, store_consumer_offset) are routed to the shard owning that partition via a lock-free concurrent map lookup. A request that lands on a non-owning shard rides the inter-shard message bus to the owner @@ -50,27 +50,29 @@ The `IggyNamespace` packs stream_id (20 bits), topic_id (12 bits), and partition The sharding system supports multiple allocation modes via the `cpu_allocation` config: - `"all"` - one shard per available CPU core -- A numeric value (e.g. `4`) - exactly N shards pinned to cores 0..N -- A range (e.g. `"5..8"`) - shards on specified core range -- `"numa:auto"` - automatically detect NUMA topology and bind accordingly +- A numeric value (e.g. `4`) - exactly N shards, pinned to the first N CPUs in the process's allowed CPU set when pinning is enabled +- A range (e.g. `"5..8"`) - shards on CPUs 5, 6 and 7 when pinning is enabled; those CPUs must be allowed for the process +- `"numa:auto"` - automatically detect NUMA topology and select physical cores, avoiding sibling hyperthreads - `"numa:nodes=0,1;cores=4;no_ht=true"` - fine-grained NUMA control per node with hyperthread avoidance ### io_uring and compio -Traditional async runtimes like tokio use `epoll` which is **readiness-based** - you ask the kernel "is this file descriptor ready?" and then perform the I/O yourself. The Linux kernel considers regular files "always ready" for epoll, which means tokio has to outsource file I/O to a blocking thread pool (up to 512 threads). This does not scale well. +Traditional async runtimes like tokio use `epoll` which is **readiness-based** - you ask the kernel "is this file descriptor ready?" and then perform the I/O yourself. [Regular files cannot be registered with epoll](https://man7.org/linux/man-pages/man2/epoll_ctl.2.html). Tokio runs file I/O on a blocking thread pool (512 threads by default, configurable). This does not scale well. -`io_uring` is **completion-based** - you submit I/O requests to a submission queue (SQ), and the kernel completes them and places results in a completion queue (CQ). Both queues are lock-free ring buffers shared between user space and kernel. This is fundamentally better for disk I/O. +`io_uring` is **completion-based** - you submit I/O requests to a submission queue (SQ), and the kernel completes them and places results in a completion queue (CQ). Both queues are shared ring buffers between user space and kernel. Submissions and completions can be batched to reduce syscalls. This is fundamentally better for disk I/O. <IoUringComparison /> Iggy uses **compio** as its async runtime, which provides a driver-disaggregated architecture on top of io_uring (Linux) and IOCP (Windows). Each shard gets its own compio executor configured with: - Capacity: 4096 concurrent I/O operations (by default) -- Event interval: 128 events per loop iteration +- Event interval: poll the I/O driver after 128 scheduler ticks (roughly task polls) by default - Cooperative task running enabled +`IGGY_SHARD_RUNTIME_CAPACITY` and `IGGY_SHARD_EVENT_INTERVAL` override the two numeric defaults. On macOS, compio uses its polling driver and a blocking pool for file I/O. + ### Performance: Tokio vs Thread-per-Core -The migration from Tokio to thread-per-core with compio delivered significant latency improvements across the board: +The [historical migration benchmarks](https://iggy.apache.org/blogs/2026/02/27/thread-per-core-io_uring/) compared v0.5.0 with v0.7.0 at approximately 1,000 MB/s per node. The chart shows selected latency reductions for 8, 16 and 32 producers, each using its own stream. The 8-producer case also reported higher P95 and P99 latency; these are not measurements of 0.9.0: <BenchmarkChart /> @@ -97,6 +99,7 @@ local_data/ ├── logs/ │ └── iggy-server.log ├── state/ +│ └── log/ └── streams/ └── 0/ └── topics/ @@ -105,14 +108,18 @@ local_data/ └── 0/ ├── 00000000000000000000.index ├── 00000000000000000000.log + ├── superblock.a + ├── superblock.b └── offsets/ + ├── consumers/ + └── groups/ ``` -The stream, topic and partition directories are named after their numeric IDs, **assigned from 0**. The `metadata/journal.wal` file is the VSR write-ahead log that persists all metadata operations (stream/topic/user creation, etc.). The `runtime/current_config.toml` file captures the configuration the server actually booted with. Segment files are named by the 20-digit start offset of their first record. The `.index` file is created automatically and speeds up searches by keeping track o [...] +This example shows a fresh partition with the default paths. The stream, topic and partition directories are named after their numeric IDs, **assigned from 0**. The `metadata/journal.wal` file is the VSR write-ahead log that persists all metadata operations (stream/topic/user creation, etc.). The `runtime/current_config.toml` file captures the configuration the server actually booted with. Segment files are named by their 20-digit start offset. Recovery can create an empty active segment [...] ## Memory pool -Iggy uses a custom memory pool with 28 buckets holding buffer sizes from 4 KiB to 512 MiB. The default pool size is 4 GiB with up to 8192 buffers per bucket. This eliminates allocation overhead on the hot path and enables zero-copy message passing between components. The pool is page-aligned (4096-byte multiples) and requires a **minimum of 512 MiB**. +Iggy uses a custom memory pool with 28 buckets holding buffer sizes from 4 KiB to 512 MiB. The default pool size is 4 GiB with up to 8192 buffers per bucket. Buffers are allocated on demand and reused; requests that exceed the pool budget can allocate outside it. Frozen buffers can be shared between components without copying their contents. The pool is page-aligned (4096-byte multiples) and requires a **minimum of 512 MiB**. ## Write pipeline @@ -120,8 +127,8 @@ Messages flow through a multi-stage write pipeline: 1. Messages arrive on the owning shard and are buffered in the partition journal. 2. Partition VSR replicates prepares. With `durability=persisted`, a multi-replica group requires recoverable prepare-WAL copies at the replication quorum before commit. -3. Committed operations are applied before success is returned. A singleton with `durability=persisted` synchronizes local segment state before replying. -4. Ordinary segment writes use per-topic count and byte thresholds (defaults: 1024 messages, 1 MiB); required persistence, capacity pressure, and lifecycle work can flush earlier. The `MessagesWriter` uses **vectored I/O** with up to 1024 buffers per syscall. +3. Awaited writes apply committed operations before success is returned. A singleton with `durability=persisted` synchronizes local segment state before replying. +4. Ordinary segment writes use per-topic count and byte thresholds (defaults: 1024 messages, 1 MiB); required persistence, capacity pressure, and lifecycle work can flush earlier. The `MessagesWriter` uses **vectored I/O** in chunks of up to 1024 buffers. Partial writes can require more than one I/O submission. 5. When a segment reaches the topic's segment size (default 1 GiB), it is **sealed** and a new segment is created. Message `durability` and `consumer_offset_durability` default independently to `replicated`. Both policies write data to disk; `persisted` adds a stable-storage requirement at completion. See [Durability](/docs/server/durability). diff --git a/content/docs/introduction/concepts.mdx b/content/docs/introduction/concepts.mdx index e1157cd1..101cfff1 100644 --- a/content/docs/introduction/concepts.mdx +++ b/content/docs/introduction/concepts.mdx @@ -3,15 +3,15 @@ title: Concepts description: "The domain model behind Iggy, and how an append-only streaming log differs from a message broker." --- -Iggy is a persistent message streaming platform: messages are stored in a form of an **append-only log**. You can create multiple streams, consisting of topics, which might have one or more partitions assigned, e.g. to achieve the horizontal scalability between many independent consumers or higher system resiliency. You can think of Iggy as an alternative to Kafka or RabbitMQ streams. +Iggy is a persistent message streaming platform: messages are stored in a form of an **append-only log**. You can create multiple streams, consisting of topics, which might have one or more partitions assigned, e.g. to divide consumption among independent consumers. Multi-node replication provides redundancy. You can think of Iggy as an alternative to Kafka or RabbitMQ streams. ## Message streaming You've probably used RabbitMQ or Kafka already. They look similar at the first glance, and you can achieve the similar results with both (e.g. publishing and consuming the events by the different applications built on top of microservices architecture), but they work differently underneath. -The main difference is that RabbitMQ (except the recently released Streams plugin) is the **message broker**, which means that it's responsible for delivering the messages to the consumers. It works in the FIFO (First In, First Out) manner and the messages are being kept in the queues. For example, if you have multiple, distinct consumers, then each one would create its own queue, the message would be replicated between each queue and each consumer would be responsible for reading the me [...] +RabbitMQ is a **message broker** with both queues and [streams](https://www.rabbitmq.com/docs/streams). Traditional queues deliver messages to consumers and remove them after acknowledgement, or on delivery when automatic acknowledgement is used, so acknowledged messages cannot be replayed from the queue. Multiple consumers can share a queue and divide its messages. Independent subscribers that each need a copy use separate queues bound to an exchange. Queues normally use FIFO (First In, [...] -On the other hand, Kafka is a **message streaming platform**, meaning that it's not responsible for delivering the messages to the consumers, but rather it's storing them in a form of an append-only log. The consumers are responsible for reading the messages from the log and processing them. You might have multiple distinct consumers, and it doesn't affect the resource usage as there's only one log. The consumers can read the messages from the beginning, or from the specific offset, thus [...] +On the other hand, Kafka is a **message streaming platform** that stores messages in an append-only log and serves [consumer fetch requests](https://kafka.apache.org/41/design/design/). The consumers are responsible for reading the messages from the log and processing them. Multiple consumer groups can read the same retained log without a separate stored copy for each group, although their fetch requests still consume CPU and network resources. The consumers can read the messages from th [...] Both approaches have advantages and disadvantages. The message broker is a more mature concept, but the message streaming platform is gaining more and more popularity, especially in the cloud-native world. And you can achieve much higher performance and throughput with the message streaming platform, since it acts as a simple database, being optimized for the append-only operations and can be queried in a very efficient way. @@ -19,9 +19,9 @@ Iggy is the latter, a message streaming platform. ## Append-only log -The append-only log is the core concept of Iggy. It's a simple data structure, which is optimized for the append-only operations. It's a sequence of records, that are being appended to the end of the log. The records are **immutable**, so that they can't be changed once they are written to the log. The records are being written in the order they are received, which results in the log being ordered. +The append-only log is the core concept of Iggy. It's a simple data structure, which is optimized for the append-only operations. It's a sequence of records, that are being appended to the end of the log. The records are **immutable**, so that they can't be changed once they are written to the log. Records within a partition follow the order admitted by that partition's primary. There is no global ordering across partitions. -You address the log by **offset**, the position of the record in the log. The offset is a simple integer that starts from 0 and is incremented by 1 for each record. When the client reads the records, it specifies the offset to start from and the maximum number of records it wants. Starting from the beginning, or from any earlier offset, is how you replay the messages. +You address the log by **offset**, the position of the record in the log. Offsets start from 0 and increase within a partition. They are not guaranteed to be contiguous: after recovery, the server can skip reserved offsets to avoid reusing them. When the client reads the records, it specifies the offset to start from and the maximum number of records it wants. Starting from the beginning, or from any earlier offset, is how you replay the messages. <AppendOnlyLogViz /> @@ -30,27 +30,27 @@ You address the log by **offset**, the position of the record in the log. The of ## Stream While we could put an equal sign between the log and the stream, they are not the same, at least in a case of Iggy streaming server. -The stream is a logical concept, and you might think of it as a **namespace**. For example, you could have a single stream for the whole system, or multiple streams e.g. representing the different environments, such as `dev`, `staging` and `production`. The stream is identified by its unique ID. The stream can have one or more topics assigned, which results in the records being published to the specific topics that belong to the particular stream. +The stream is a logical concept, and you might think of it as a **namespace**. For example, you could have a single stream for the whole system, or multiple streams e.g. representing the different environments, such as `dev`, `staging` and `production`. The stream is identified by its unique ID. The stream can have zero or more topics assigned, which results in the records being published to the specific topics that belong to the particular stream. ## Topic -The topic is also the logical concept, which is a part of the stream. The topic is identified by its unique ID. You could think of topic as an entity being responsible for storing the specific type of the records. For example, you could have a topic for the user events, and another topic for the order events, etc. +The topic is also the logical concept, which is a part of the stream. The topic is identified by its ID, which is unique within its stream. You could think of topic as an entity being responsible for storing the specific type of the records. For example, you could have a topic for the user events, and another topic for the order events, etc. -The messages are not being stored in the topic directly, but rather in the **partitions**, which are assigned to the topic. The topic can have one or more partitions assigned, that could help achieve higher parallelism and throughput. The topic can also have the **retention policy** assigned, which means that the records are being deleted automatically once they are older than the specified retention period. Topics also support maximum size limits and per-topic storage options (`segment_ [...] +The messages are not being stored in the topic directly, but rather in the **partitions**, which are assigned to the topic. The topic can have one or more partitions assigned, that could help achieve higher parallelism and throughput. The topic can also have a **retention policy**. Expiry removes whole sealed segments after their newest message has expired, subject to the stored consumer-offset barrier; it does not delete each message immediately at its expiry time. See [topic options](/ [...] ## Partition -The partition has its own unique ID and belongs to the topic. The partition is responsible for storing the records. The records are being distributed between the partitions, therefore the partition acts as a simple database, which is optimized for the append-only operations. The partition is identified by its unique ID, which is an integer. Stream, topic and partition IDs are all assigned **starting from 0**, incremented by 1 for each new one. The partition ID is unique per topic, thus t [...] +The partition has its own unique ID and belongs to the topic. The partition is responsible for storing the records. The records are being distributed between the partitions, therefore the partition acts as a simple database, which is optimized for the append-only operations. The partition is identified by its unique ID, which is an integer. Stream, topic and partition IDs are assigned **starting from 0**. Deleted stream and topic IDs can be reused. New partitions use IDs above the highes [...] Thanks to having multiple partitions, we can achieve the horizontal scalability between many independent consumers, since each consumer can read the messages from the different partitions. This can be achieved by using more advanced concepts such as consumer groups. -Each partition in the thread-per-core architecture is owned by **exactly one shard** and includes: +Each partition in the thread-per-core architecture is owned by **exactly one shard per replica** and includes: - A `SegmentedLog` with sealed segments and one active segment - Consumer offsets and consumer group offsets ## Segment -The segment, being a part of the partition, is the actual **physical layer** which stores the records in the binary format in a form of the files. Each segment has the limited size (1 GiB unless the topic sets its own `segment_size`) and once it's full, the new segment is being created automatically. The segment name is based on the start offset of the first record in the segment and is unique per partition. +The segment, being a part of the partition, is the actual **physical layer** which stores the records in the binary format in a form of the files. Each segment has a soft size limit (1 GiB unless the topic sets its own `segment_size`). A segment can exceed it by one whole batch before it is sealed and a new segment is created. The segment name is based on the start offset of the first record in the segment and is unique per partition. Each segment consists of: - `.log` file - the actual message data @@ -63,11 +63,11 @@ Consumers can poll the messages in multiple ways: - **By offset** - start reading from the specified offset. The client tracks its own position. - **By timestamp** - start reading from the first message at or after the given timestamp. - **First / Last** - start from the beginning or the end of the partition. -- **Next** - continue from the consumer offset stored on the server side. The client no longer needs to track the offset itself. Combine it with `auto_commit: true` to commit the offset automatically once the messages are fetched (*at-most-once* delivery), or call `store_offset()` explicitly after processing. +- **Next** - continue from the consumer offset stored on the server side. The server tracks the cursor. On the primary, `auto_commit: true` submits an offset update before the poll response is delivered, without waiting for that update to commit. Follower polls do not advance it. This alone guarantees neither at-most-once nor at-least-once processing. For processing-before-commit ordering, explicitly store the offset after processing, using `store_consumer_offset()` in the Rust SDK. ## Consumer groups -Consumer groups provide horizontal scaling for message consumption. When multiple consumers join the same consumer group, the server automatically distributes partitions among group members so that each partition is consumed by **exactly one member**. When members join or leave, the server triggers a **cooperative partition rebalancing** with a pending revocation phase (configurable timeout, default 30s) to ensure smooth transitions without message loss. +Consumer groups provide horizontal scaling for message consumption. When multiple consumers join the same consumer group, the server automatically distributes partitions among group members so that each partition has **at most one member permitted to poll it** within that group. When members join or leave, the server triggers a **cooperative partition rebalancing** with a pending revocation phase (configurable timeout, default 30s) to let the previous owner finish processing and commit b [...] <ConsumerGroupViz /> @@ -75,19 +75,19 @@ Consumer groups provide horizontal scaling for message consumption. When multipl <MessageHeaderDiagram /> -In the SDKs, each message carries a 64-byte in-memory header (little-endian fields). On the wire and on disk, messages travel inside batch records with a compact 48-byte per-message frame. See the [Binary Protocol](/docs/binary-protocol) section for the exact encodings. +The Rust SDK's `IggyMessageHeader::to_bytes()` representation is 64 bytes, with little-endian fields. The diagram and table describe that SDK representation, not Rust struct memory layout. On the wire and on disk, messages travel inside batch records with a compact 48-byte per-message frame header. See the [Binary Protocol](/docs/binary-protocol) section for the exact encodings. | Field | Bytes | Type | Description | |-------|-------|------|-------------| | checksum | 0-8 | u64 | xxHash3 integrity checksum | -| id | 8-24 | u128 | Unique message ID (UUIDv4) | -| offset | 24-32 | u64 | Sequential offset in partition | +| id | 8-24 | u128 | Client-supplied 128-bit ID; generated as UUIDv4 when omitted by the Rust SDK | +| offset | 24-32 | u64 | Increasing offset within the partition | | timestamp | 32-40 | u64 | Server-assigned timestamp | | origin_timestamp | 40-48 | u64 | Client-provided timestamp | | user_headers_length | 48-52 | u32 | Length of optional headers | | payload_length | 52-56 | u32 | Length of payload | | reserved | 56-64 | u64 | Reserved (must be 0) | -After the header comes the optional user headers bytes, followed by the payload bytes. +In the SDK message representation, the header is followed by payload bytes and then optional user-header bytes. To see how the server schedules these concepts across CPU cores and stores them on disk, head over to [architecture](/docs/introduction/architecture). diff --git a/content/docs/introduction/getting-started.mdx b/content/docs/introduction/getting-started.mdx index 1f5bc12f..00f87c17 100644 --- a/content/docs/introduction/getting-started.mdx +++ b/content/docs/introduction/getting-started.mdx @@ -21,7 +21,7 @@ The completed sample can be found in the [repository](https://github.com/apache/ For our purpose, we will focus on the basic scenario in order to keep things simple. Before we begin implementing the consumer and producer apps, we need to start the Iggy streaming server. -The quickest way is Docker, using the [official images](https://hub.docker.com/r/apache/iggy): +This guide targets server **0.9.0**. Once that release is published, you can use its [official Docker image](https://hub.docker.com/r/apache/iggy). During release preparation, use a matching `edge` build or the source build below: ```bash docker run --rm \ @@ -30,7 +30,7 @@ docker run --rm \ -e IGGY_TCP_ADDRESS=0.0.0.0:8090 \ -e IGGY_NODE_ADVERTISED_ADDRESS=localhost \ -e IGGY_ROOT_USERNAME=iggy -e IGGY_ROOT_PASSWORD=iggy \ - apache/iggy:latest + apache/iggy:0.9.0 ``` The capabilities, seccomp setting, and memlock limit form a permissive development setup. Production deployments can use narrower syscall permissions and a finite memory budget; see [Docker & Helm](/docs/server/docker#why-these-capabilities) for the details. `IGGY_TCP_ADDRESS` is needed because the server binds to `127.0.0.1` inside the container by default, which a published port cannot reach. `IGGY_NODE_ADVERTISED_ADDRESS` is needed because that wildcard leaves the server with no addre [...] @@ -41,7 +41,7 @@ Alternatively, build from source by cloning the [repository](https://github.com/ cargo run --bin iggy-server -- --fresh --with-default-root-credentials ``` -A bare `iggy-server` boot does **not** create the root user with the well-known `iggy`/`iggy` pair. It generates a random password and prints it **exactly once** to the logs. The `--with-default-root-credentials` flag switches to the development credentials (username: `iggy`, password: `iggy`), and `--fresh` wipes the data directory first, so the root user is created anew with them. You can also set `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` yourself. The environment **takes precedenc [...] +A bare `iggy-server` boot does **not** create the root user with the well-known `iggy`/`iggy` pair. It generates a random password and prints it **exactly once** to the logs. The `--with-default-root-credentials` flag switches to the development credentials (username: `iggy`, password: `iggy`), and `--fresh` wipes the data directory first, so the root user is created anew with them. You can also set `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD` yourself. The environment **takes precedenc [...] All the data used by the server will be persisted under the `local_data` directory, unless specified differently in the configuration. @@ -95,7 +95,7 @@ From that point on, we will focus on implementing the message streaming between ## Building the producer -We will begin with installing the Iggy client crate - execute `cargo add iggy` in your terminal. Next, install [tokio.rs](https://tokio.rs) dependency with `cargo add tokio` as we will use the asynchronous runtime. Eventually, modify your `main.rs`, so it looks like this: +We will begin with installing the Iggy client crate - use the SDK from the same source checkout as your server. With `iggy-sample` beside the `iggy` repository, execute `cargo add iggy --path ../iggy/core/sdk` in your terminal. Published SDK releases can expose different APIs. Next, install [tokio.rs](https://tokio.rs) dependency with `cargo add tokio --features macros,rt-multi-thread,time` as we will use the asynchronous runtime. Eventually, modify your `main.rs`, so it looks like this: ```rust use std::error::Error; @@ -113,9 +113,9 @@ let client = IggyClient::default(); client.connect().await?; ``` -The default server address being `127.0.0.1:8090` is configured on the server side, and can be easily adjusted by updating `config.toml` in the `core/server` directory. +Both the default TCP client and the server use `127.0.0.1:8090`. If you change the server address in `core/server/config.toml`, configure the client to connect to that address too. -We could make use of more advanced components such as [`ClientProvider`](https://github.com/apache/iggy/blob/master/core/sdk/src/client_provider.rs), pass the custom configuration built via console args to choose between the different protocols as all the available clients implement the same [Client](https://github.com/apache/iggy/blob/master/core/sdk/src/clients/client.rs) trait and so on. +We could make use of more advanced components such as [`ClientProvider`](https://github.com/apache/iggy/blob/master/core/sdk/src/client_provider.rs), pass the custom configuration built via console args to choose between the different protocols as all the available clients implement the same [Client](https://github.com/apache/iggy/blob/master/core/common/src/traits/client.rs) trait and so on. If you're eager to find out how to build more advanced (and configurable) applications, check the Rust [examples](/docs/sdk/rust/examples). Nevertheless, let's focus on implementing our producer side :) @@ -129,9 +129,9 @@ client When you start the application now, by running `cargo r --bin producer` it will execute immediately, however, you should be able to see the logs on Iggy server - connection should be accepted, user logged in, and then connection should be closed right away, meaning that we've just established our very first communication with the streaming server. -Before we will move on with the code, let's include some logging in our apps as well (Iggy client already has some logging in place on different levels). Let's make use of the [tracing.rs](https://tracing.rs) crates: `cargo add tracing tracing_subscriber`. +Before we will move on with the code, let's include some logging in our apps as well (Iggy client already has some logging in place on different levels). Let's make use of the [tracing.rs](https://tracing.rs) crates: `cargo add tracing` and `cargo add tracing-subscriber --features env-filter`. -To make use of logging, simply invoke `tracing_subscriber::fmt::init()` at the beginning of `main()` method: +Initialize the tracing subscriber at the beginning of `main()`, using `RUST_LOG` when set and `info` otherwise: ```rust use std::error::Error; @@ -139,7 +139,12 @@ use iggy::prelude::*; #[tokio::main] async fn main() -> Result<(), Box<dyn Error>> { - tracing_subscriber::fmt::init(); + tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| "info".into()), + ) + .init(); let client = IggyClient::default(); client.connect().await?; client @@ -153,9 +158,9 @@ From that point on, when starting the application, you should be able to see at So far, so good, however, before we will be able to publish any messages to our streaming server, at first, we need to create the stream, topic and partition(s) - if you're unfamiliar with these concepts, please refer to [concepts](/docs/introduction/concepts) where all of them are described in-depth. -Since our `IggyClient` implements the common [Client](https://github.com/apache/iggy/blob/master/core/sdk/src/clients/client.rs) trait, you can find lots of the different methods to interact with the server, also from the administrative point of view, e.g. creating the streams, topics etc. +Since our `IggyClient` implements the common [Client](https://github.com/apache/iggy/blob/master/core/common/src/traits/client.rs) trait, you can find lots of the different methods to interact with the server, also from the administrative point of view, e.g. creating the streams, topics etc. -These methods are **not idempotent** - for example, if you were to try creating the stream with the same name which already exists on the server, you would receive the specific error. In such a case, you can simply check for an error and move on. When creating a stream, we only provide its name - the server assigns the numeric ID automatically. Stream, topic and partition IDs are all assigned **starting from 0**. Let's do this then :) +These methods are **not idempotent** - for example, if you were to try creating the stream with the same name which already exists on the server, you would receive the specific error. In such a case, handle the specific already-exists error and propagate other failures. When creating a stream, we only provide its name - the server assigns the numeric ID automatically. Stream, topic and partition IDs are all assigned **starting from 0**. Let's do this then :) ```rust use iggy::prelude::*; @@ -167,20 +172,28 @@ const TOPIC_NAME: &str = "sample-topic"; #[tokio::main] async fn main() -> Result<(), Box<dyn Error>> { - tracing_subscriber::fmt::init(); + tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| "info".into()), + ) + .init(); let client = IggyClient::default(); client.connect().await?; client .login_user(DEFAULT_ROOT_USERNAME, DEFAULT_ROOT_PASSWORD) .await?; - init_system(&client).await; + init_system(&client).await?; Ok(()) } -async fn init_system(client: &IggyClient) { +async fn init_system(client: &IggyClient) -> Result<(), IggyError> { match client.create_stream(STREAM_NAME).await { Ok(_) => info!("Stream was created."), - Err(_) => warn!("Stream already exists and will not be created again."), + Err(IggyError::StreamNameAlreadyExists(_)) => { + warn!("Stream already exists and will not be created again."); + } + Err(error) => return Err(error), } match client @@ -196,8 +209,12 @@ async fn init_system(client: &IggyClient) { .await { Ok(_) => info!("Topic was created."), - Err(_) => warn!("Topic already exists and will not be created again."), + Err(IggyError::TopicNameAlreadyExists(..)) => { + warn!("Topic already exists and will not be created again."); + } + Err(error) => return Err(error), } + Ok(()) } ``` @@ -240,7 +257,7 @@ pub struct Identifier { } ``` -Whenever we interact with the streaming server in terms of e.g. sending or polling the messages, managing the streams, topics etc. we need to provide the unique identifier of the stream and the topic that we want to use. Since each stream and topic have a unique numeric ID, as well as a unique name, we can use either of them. Here, we use the string name by calling `Identifier::named()` (implicitly via `TryFrom` on a `&str`), however, you can also use the numeric identifier by invoking t [...] +Whenever we interact with the streaming server in terms of e.g. sending or polling the messages, managing the streams, topics etc. we need to provide the unique identifier of the stream and the topic that we want to use. Since each stream and topic have a unique numeric ID, as well as a unique name, we can use either of them. Use `Identifier::named()` to force a name, or `Identifier::numeric()` for a numeric ID. `TryFrom<&str>` interprets a string that parses as a `u32` as a numeric ID; [...] Next, let's move onto the `partitioning` field: @@ -257,7 +274,7 @@ In our scenario, we simply make use of `PartitioningKind::PartitionId` (by invok However, once your system grows, you might want to parallelize the messages across the independent consumers, in order to achieve the horizontal scaling, higher resiliency etc. In that case, you might consider using either `PartitioningKind::Balanced` (the SDK picks the next partition using a client-local round-robin e.g. 0->1->2->0->1->2 etc.) or `PartitioningKind::MessagesKey` instead (e.g. by invoking one of the helper methods `messages_key()`), where the value wouldn't be a partition [...] -For example, given that you process the set of messages related to the specific order ID (e.g. created, confirmed, paid, delivered etc.), you could use that as a key value to ensure that all the messages which are part of the specific workflow, will always be put onto the same partition. The value of the key can be anything (e.g. string, number) with the **maximum length of 255 bytes**. +For example, given that you process the set of messages related to the specific order ID (e.g. created, confirmed, paid, delivered etc.), you could use that as a key value to ensure that all the messages which are part of the specific workflow, will be put onto the same partition while the partition count and partitioning strategy stay unchanged. The value of the key can be anything (e.g. string, number) with the **maximum length of 255 bytes**. Anyway, let's get back to our scenario, and consider the following code responsible for publishing the messages: @@ -297,7 +314,7 @@ async fn produce_messages(client: &IggyClient) -> Result<(), Box<dyn Error>> { } ``` -The reason behind passing a mutable reference is that the underlying client (especially when using the `IggyClient` wrapper on top of the low-level client) might want to modify the command before sending it to the server. For example, the client might want to encrypt the payload, include the default headers, or provide a custom `Partitioner` implementation etc. thus by using `&mut` we can avoid copying the command each time. +The mutable slice lets `IggyClient` encrypt message payloads and user headers in place when client-side encryption is enabled. Finally, let's complete the implementation of the producer - once you start the application after the latest changes, you shall see the messages being sent to the newly created stream. @@ -315,21 +332,29 @@ const PARTITION_ID: u32 = 0; #[tokio::main] async fn main() -> Result<(), Box<dyn Error>> { - tracing_subscriber::fmt::init(); + tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| "info".into()), + ) + .init(); let client = IggyClient::default(); client.connect().await?; client .login_user(DEFAULT_ROOT_USERNAME, DEFAULT_ROOT_PASSWORD) .await?; - init_system(&client).await; + init_system(&client).await?; produce_messages(&client).await?; Ok(()) } -async fn init_system(client: &IggyClient) { +async fn init_system(client: &IggyClient) -> Result<(), IggyError> { match client.create_stream(STREAM_NAME).await { Ok(_) => info!("Stream was created."), - Err(_) => warn!("Stream already exists and will not be created again."), + Err(IggyError::StreamNameAlreadyExists(_)) => { + warn!("Stream already exists and will not be created again."); + } + Err(error) => return Err(error), } match client @@ -345,8 +370,12 @@ async fn init_system(client: &IggyClient) { .await { Ok(_) => info!("Topic was created."), - Err(_) => warn!("Topic already exists and will not be created again."), + Err(IggyError::TopicNameAlreadyExists(..)) => { + warn!("Topic already exists and will not be created again."); + } + Err(error) => return Err(error), } + Ok(()) } async fn produce_messages(client: &IggyClient) -> Result<(), Box<dyn Error>> { @@ -408,15 +437,15 @@ At the first glance, it might look a bit more complicated than `send_messages` f - `topic_id` - the ID of the topic (numeric or string) from which we want to poll the messages. -- `partition_id` - the ID of the partition from which we want to poll the messages. The partition has to be specified for the regular `Consumer`, while for the `ConsumerGroup` it's **ignored** (`None` value), as the server will automatically assign the partition to the consumer from the group. +- `partition_id` - the ID of the partition from which we want to poll the messages. Specify the partition for a regular `Consumer`. For a joined `ConsumerGroup`, pass `None` to let the SDK select a partition from the assignments it synchronizes with the server. An explicit partition is also accepted, but the server checks that the member owns it. -- `consumer` - the type of the consumer (kind + ID), either the default `Consumer` means the standalone client which does the message polling on its own, independently of the other consumers (unless they would use the same ID), or the `ConsumerGroup` which might be used to create the group of consumers sharing the common identifier - this is especially useful in the case of the horizontal scaling, where we want to ensure, that the same (and only one) consumer, will poll the messages from [...] +- `consumer` - the type of the consumer (kind + ID), either the default `Consumer` means the standalone client which does the message polling on its own, independently of the other consumers (unless they would use the same ID), or the `ConsumerGroup` which might be used to create the group of consumers sharing the common identifier - this is especially useful in the case of the horizontal scaling, where we want to ensure, that the same (and only one) consumer, will poll the messages from [...] -- `strategy` - the way in which we want to poll the messages. The default one being `Offset` (underlying `PollingKind` enum) means that we will start polling the messages from the particular offset provided in the `value` field - it's on the client, to keep track of the most recent offset. On the other hand, we could also use the different kind, for example `Next`, which means, that the next messages will be returned to the client, depending on the so-called `consumer offset` value store [...] +- `strategy` - the way in which we want to poll the messages. The default one being `Offset` (underlying `PollingKind` enum) means that we will start polling the messages from the particular offset provided in the `value` field - it's on the client, to keep track of the most recent offset. On the other hand, we could also use the different kind, for example `Next`, which means, that the next messages will be returned to the client, depending on the so-called `consumer offset` value store [...] - `count` - amount of the messages that the consumer would like to receive in the single response from the server. -- `auto_commit` - whether the consumer offset should be automatically committed on the server side, once the messages are fetched. +- `auto_commit` - whether to request server-managed offset advancement when messages are fetched, with the best-effort behavior described above. Next, let's take a look at the following method responsible for polling the messages based on the specified interval. @@ -445,9 +474,9 @@ async fn consume_messages(client: &IggyClient) -> Result<(), Box<dyn Error>> { continue; } - offset += polled_messages.messages.len() as u64; for message in polled_messages.messages { handle_message(&message)?; + offset = message.header.offset + 1; } sleep(interval).await; } @@ -469,7 +498,12 @@ const PARTITION_ID: u32 = 0; #[tokio::main] async fn main() -> Result<(), Box<dyn Error>> { - tracing_subscriber::fmt::init(); + tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| "info".into()), + ) + .init(); let client = IggyClient::default(); client.connect().await?; client @@ -510,9 +544,9 @@ async fn consume_messages(client: &IggyClient) -> Result<(), Box<dyn Error>> { continue; } - offset += polled_messages.messages.len() as u64; for message in polled_messages.messages { handle_message(&message)?; + offset = message.header.offset + 1; } sleep(interval).await; } diff --git a/src/components/architecture-diagrams.tsx b/src/components/architecture-diagrams.tsx index b6c8cbfe..e97563d2 100644 --- a/src/components/architecture-diagrams.tsx +++ b/src/components/architecture-diagrams.tsx @@ -37,11 +37,11 @@ export function MessageFlowDiagram() { const steps = [ { label: "Client", desc: "Client sends messages via TCP/QUIC/WS/HTTP", icon: "M12 2C6.48 2 2 6.48 2 12s4.48 10 10 10 10-4.48 10-10S17.52 2 12 2zm-1 17.93c-3.95-.49-7-3.85-7-7.93 0-.62.08-1.21.21-1.79L9 15v1c0 1.1.9 2 2 2v1.93z" }, - { label: "Listener", desc: "Transport listener receives the request on a shard thread", icon: "M21 3L3 10.53v.98l6.84 2.65L12.48 21h.98L21 3z" }, + { label: "Listener", desc: "Shard 0 accepts connections; the connection owner decodes requests", icon: "M21 3L3 10.53v.98l6.84 2.65L12.48 21h.98L21 3z" }, + { label: "Stream", desc: "Resolve the stream ID or name from local metadata", icon: "M4 6h16v2H4zm0 5h16v2H4zm0 5h16v2H4z" }, + { label: "Topic", desc: "Resolve the topic and target partition; compression is not applied", icon: "M3 18h6v-2H3v2zM3 6v2h18V6H3zm0 7h12v-2H3v2z" }, { label: "Router", desc: "Request routed to owning shard via IggyNamespace hash", icon: "M12 2C6.48 2 2 6.48 2 12s4.48 10 10 10 10-4.48 10-10S17.52 2 12 2zm1 15h-2v-2h2v2zm0-4h-2V7h2v6z" }, - { label: "Stream", desc: "Stream lookup by ID (metadata read from left-right)", icon: "M4 6h16v2H4zm0 5h16v2H4zm0 5h16v2H4z" }, - { label: "Topic", desc: "Topic lookup within stream, compression applied", icon: "M3 18h6v-2H3v2zM3 6v2h18V6H3zm0 7h12v-2H3v2z" }, - { label: "Partition", desc: "Messages buffered in the partition journal (PartitionJournal)", icon: "M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zm0 16H5V5h14v14z" }, + { label: "Partition", desc: "The partition primary admits and replicates the write through VSR", icon: "M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zm0 16H5V5h14v14z" }, { label: "Segment", desc: "Flushed to .log file via vectored I/O (io_uring)", icon: "M14 2H6c-1.1 0-2 .9-2 2v16c0 1.1.9 2 2 2h12c1.1 0 2-.9 2-2V8l-6-6zm4 18H6V4h7v5h5v11z" }, ]; @@ -126,7 +126,7 @@ export function ShardDiagram() { id: 0, label: "Shard 0 (Coordinator)", color: "var(--color-fd-primary)", - features: ["Binds all listeners: TCP, QUIC, HTTP, WS", "Replica plane listener", "Metadata plane (left-right write handle)", "QUIC + TCP-TLS terminate here", "Hands plaintext TCP/WS to peers (fd transfer)"], + features: ["Binds all listeners: TCP, QUIC, HTTP, WS", "Replica plane listener", "Metadata plane (left-right write handle)", "QUIC + TCP-TLS + WSS + HTTP terminate here", "Hands plaintext TCP/WS to peers (fd transfer)"], partitions: ["P0", "P3", "P6"], }, { @@ -174,7 +174,7 @@ export function ShardDiagram() { <span className="text-[10px] font-mono text-fd-muted-foreground block mb-1">compio runtime</span> <div className="flex items-center gap-1"> <span className="inline-block w-1.5 h-1.5 rounded-full bg-emerald-500 ring-glow" /> - <span className="text-[10px] text-fd-foreground">io_uring (4096 ops)</span> + <span className="text-[10px] text-fd-foreground">io_uring (default capacity: 4096)</span> </div> </div> @@ -213,8 +213,8 @@ export function ShardDiagram() { <span className="text-xs font-medium text-fd-muted-foreground">Inter-shard communication</span> </div> <p className="text-xs text-fd-muted-foreground m-0"> - Shards communicate via <code className="text-[11px]">crossfire</code> bounded mpsc channels. Metadata mutations route to Shard 0, the only shard that commits; peers hold left-right read handles. - Partition ops route to the owning shard via the lock-free <code className="text-[11px]">papaya::HashMap<IggyNamespace, PartitionLocation></code>. + Shards communicate via <code className="text-[11px]">crossfire</code> bounded mpsc channels. Metadata mutations route to Shard 0, the only shard that commits metadata; peers hold left-right read handles. + CPU labels and partition assignments above are illustrative. Partition ops route to the owning shard via the lock-free <code className="text-[11px]">papaya::HashMap<IggyNamespace, PartitionLocation></code>. </p> </div> </div> @@ -239,7 +239,7 @@ export function IoUringComparison() { </div> ))} <div className="rounded-md bg-red-500/10 border border-red-500/20 p-2 mt-2"> - <span className="text-[11px] text-red-400 block">Files are "always ready" for epoll. Tokio uses a blocking thread pool (up to 512 threads) for file I/O.</span> + <span className="text-[11px] text-red-400 block">Regular files cannot be registered with epoll. Tokio uses a blocking pool for file I/O (512 threads by default, configurable).</span> </div> </div> </div> @@ -259,7 +259,7 @@ export function IoUringComparison() { </div> ))} <div className="rounded-md bg-emerald-500/10 border border-emerald-500/20 p-2 mt-2"> - <span className="text-[11px] text-emerald-400 block">Both SQ and CQ are lock-free ring buffers shared between user space and kernel. No syscall per I/O in the hot path.</span> + <span className="text-[11px] text-emerald-400 block">SQ and CQ are shared ring buffers. Batching amortizes submission and completion syscalls across operations.</span> </div> </div> </div> @@ -373,8 +373,8 @@ export function MessageHeaderDiagram() { const fields = [ { name: "checksum", bytes: "0-8", size: 8, type: "u64", desc: "xxHash3 integrity checksum", color: "#ef4444" }, - { name: "id", bytes: "8-24", size: 16, type: "u128", desc: "Unique message ID (UUIDv4)", color: "#f59e0b" }, - { name: "offset", bytes: "24-32", size: 8, type: "u64", desc: "Sequential offset in partition", color: "#10b981" }, + { name: "id", bytes: "8-24", size: 16, type: "u128", desc: "Client-supplied 128-bit ID; Rust SDK generates UUIDv4 when omitted", color: "#f59e0b" }, + { name: "offset", bytes: "24-32", size: 8, type: "u64", desc: "Increasing offset within the partition", color: "#10b981" }, { name: "timestamp", bytes: "32-40", size: 8, type: "u64", desc: "Server-assigned timestamp", color: "#3b82f6" }, { name: "origin_ts", bytes: "40-48", size: 8, type: "u64", desc: "Client-provided timestamp", color: "#6366f1" }, { name: "hdrs_len", bytes: "48-52", size: 4, type: "u32", desc: "User headers length", color: "#8b5cf6" }, @@ -386,9 +386,9 @@ export function MessageHeaderDiagram() { return ( <div className="my-8 rounded-xl border border-fd-border bg-fd-card p-6"> - <h3 className="text-lg font-semibold text-fd-foreground m-0 mb-2">Message Header (64 bytes, little-endian)</h3> + <h3 className="text-lg font-semibold text-fd-foreground m-0 mb-2">Rust SDK Header (64 bytes, little-endian)</h3> <p className="text-xs text-fd-muted-foreground m-0 mb-5"> - Every message starts with this fixed-size header for efficient aligned reads. + This is the SDK byte representation. Wire and disk batches use a 48-byte per-message frame header. </p> <div className="flex rounded-lg overflow-hidden mb-4 h-10"> @@ -454,7 +454,7 @@ export function BenchmarkChart() { <div className="my-8 rounded-xl border border-fd-border bg-fd-card p-6"> <h3 className="text-lg font-semibold text-fd-foreground m-0 mb-2">Latency Improvements: Tokio vs Thread-per-Core</h3> <p className="text-xs text-fd-muted-foreground m-0 mb-5"> - Relative latency comparison (lower is better). Thread-per-core with io_uring vs Tokio work-stealing. + Selected historical results (lower is better): v0.5.0 vs v0.7.0 at approximately 1,000 MB/s per node. See the linked migration benchmark for workloads. </p> <div className="space-y-4"> @@ -483,7 +483,7 @@ export function BenchmarkChart() { export function NamespacePacking() { const bitGroups = [ - { label: "unused (12 bits)", bits: 12, color: undefined, range: "63..52" }, + { label: "zero for partitions (12 bits)", bits: 12, color: undefined, range: "63..52" }, { label: "stream", bits: 20, color: "#f59e0b88", range: "51..32", sub: "20 bits" }, { label: "topic", bits: 12, color: "#3b82f688", range: "31..20", sub: "12 bits" }, { label: "partition", bits: 20, color: "#10b98188", range: "19..0", sub: "20 bits" }, @@ -499,7 +499,7 @@ export function NamespacePacking() { <div className="my-8 rounded-xl border border-fd-border bg-fd-card p-6"> <h3 className="text-lg font-semibold text-fd-foreground m-0 mb-2">IggyNamespace Bit Packing (u64)</h3> <p className="text-xs text-fd-muted-foreground m-0 mb-5"> - Stream, topic, and partition IDs are packed into a single u64 for efficient hashing and shard routing. + Stream, topic, and partition IDs are packed into a single u64 for efficient hashing and shard routing. Bit 63 is reserved for the separate metadata consensus group. </p> <div className="flex rounded-lg overflow-hidden h-12 mb-3"> @@ -761,7 +761,7 @@ export function ConsumerGroupViz() { <div className="my-8 rounded-xl border border-fd-border bg-fd-card p-6"> <h3 className="text-lg font-semibold text-fd-foreground m-0 mb-2">Consumer Group</h3> <p className="text-xs text-fd-muted-foreground m-0 mb-6"> - Each partition is assigned to exactly one consumer. When a consumer joins or leaves, partitions are rebalanced. + Within this group, each partition has at most one member permitted to poll it. Joins and leaves trigger rebalancing; pending handoffs can temporarily pause polling. </p> <div className="grid grid-cols-1 md:grid-cols-[1fr_auto_1fr] gap-6 items-center">
