hubcio opened a new issue, #4021: URL: https://github.com/apache/iggy/issues/4021
## Why [OpenMessaging Benchmark](https://github.com/openmessaging/benchmark) (OMB) is the suite behind Confluent's 2020 Kafka vs Pulsar vs RabbitMQ comparison and Redpanda's 2022 Redpanda vs Kafka comparison (each ran its own fork, confluentinc/openmessaging-benchmark and redpanda-data/openmessaging-benchmark). It ships 15 drivers, 130 workloads under `workloads/`, and a result JSON that `bin/create_charts.py` turns into the charts people compare. Iggy has no driver, so there's no like-for-like number for it. #217 (2023) asked for a Kafka-style single-node benchmark and linked the OMB docs. It was closed as completed in 2025 with iggy-bench, without an OMB driver. `iggy-bench` stays the in-house tool (8 benchmark kinds, one microsecond latency sample per batch, `report.json` for the dashboard). OMB is coarser on the consumer side (end-to-end latency is a wall-clock millisecond difference, 0 ms samples are dropped) and reports in 10-second windows, but it's the baseline other systems already publish against. ## When After the Iggy 0.9.0 release. That release puts Java SDK 0.9.0 on Maven Central, the first release that speaks the VSR wire protocol (#3841). The current 0.8.0 (on Maven Central since 2026-04-22) uses the old TCP framing, which the server removed in #3856, so it can't even log in. OMB's build resolves third-party artifacts from Maven Central only and its CI runs without a custom `settings.xml`, so the driver isn't buildable upstream before then. ## What a driver has to provide `driver-api` (`io.openmessaging.benchmark:driver-api:0.0.1-SNAPSHOT`, compiled with `--release 17`) is four interfaces plus a `ResourceCreator` helper nobody needs. Compiling against it needs only that jar and its transitive `bookkeeper-stats-api`. `BenchmarkDriver` has `initialize(File yaml, StatsLogger)`, `getTopicNamePrefix()`, `createTopic(name, partitions)`, `createProducer(topic)`, `createConsumer(topic, subscriptionName, callback)` and `close()`. The worker instantiates it by reflection from `driverClass` in the driver yaml, through a no-arg constructor. `BenchmarkProducer.sendAsync(Optional<String> key, byte[] payload)` returns a `CompletableFuture<Void>`. The worker pins each producer to one load thread and calls it once per message, measuring publish latency from just before the call until the future completes, so batching is the driver's job (the Kafka driver gets it from `linger.ms` and `batch.size`). Payload arrays are shared across calls and never mutated, so holding a reference until flush is fine. Before the load starts the worker sends one 10-byte probe per producer with key `"key"` and waits up to 60 s for the consumers to see it. `ConsumerCallback.messageReceived(byte[] payload, long publishTimestamp)` (there's also a `ByteBuffer` overload) is where the worker computes end-to-end latency as `System.currentTimeMillis() - publishTimestamp`, so the driver has to carry a producer-side wall-clock timestamp in milliseconds through the broker. `BenchmarkConsumer` is only `AutoCloseable`. Consumers are created before producers. The `consumerPerSubscription` consumers of one `subscriptionName` share the topic, and every subscription gets every message: one Iggy consumer group per subscription, and the readiness and backlog logic depends on it. Topics are named `<prefix>-<7 digits>-<7 base64url chars>`, subscriptions `sub-<3 digits>-<7 base64url chars>`. Iggy only limits names to 255 bytes, so they pass as-is. ## MVP - [ ] `driver-iggy` Maven module in openmessaging/benchmark, developed in a fork and sent upstream as a PR. Depends on `org.apache.iggy:iggy:0.9.0`. Package `io.openmessaging.benchmark.driver.iggy`, classes `IggyBenchmarkDriver`, `IggyBenchmarkProducer`, `IggyBenchmarkConsumer`, `IggyConfig`. Registered as a `<module>` in the root `pom.xml` and as a compile-scope `<dependency>` in `benchmark-framework/pom.xml` (sorted between `driver-bookkeeper` and `driver-jms`, spotless checks the order). That's what puts the jar into `lib/` of the tarball, `package/pom.xml` lists no drivers. `driver-iggy/*.yaml` is packaged automatically. A bullet in OMB's README platform list, like #444 did. Passes checkstyle, spotless (`mvn spotless:apply`), license headers (`mvn license:format`), spotbugs, and the flexmark check on `driver-iggy/README.md`. - [ ] Shaded jar. OMB's assembly keeps only `netty-all` 4.1.65.Final in `lib/` and strips every individual netty module, while the SDK needs netty 4.2 (`MultiThreadIoEventLoopGroup` and `NioIoHandler` exist in no 4.1.x release). The SDK also uses Jackson 3 (`tools.jackson`) while OMB pins `jackson-annotations` 2.13.2. So `driver-iggy` builds with `maven-shade-plugin`: the SDK and its runtime closure go into the driver jar with netty, Jackson and the other SDK-only libraries relocated, `slf4j-api` left alone (OMB provides 1.7.36, the SDK uses only the classic API), and the SDK dependency marked `<optional>true</optional>` so the plain SDK jar and its dependencies stay out of `lib/`. Bumping `netty.version` in OMB is not an option: since 4.1.69 `netty-all` is an empty aggregator whose modules the assembly deletes. - [ ] `IggyBenchmarkDriver`. Reads `iggy.yaml` (host, port, username, password, stream name, producer batch size and linger, consumer poll size), opens one admin `AsyncIggyTcpClient` with `buildAndLogin()`, creates the stream when `getStream` returns empty. `createTopic` maps to `TopicsClient.createTopic(stream, partitions, CompressionAlgorithm.None, ZERO, ZERO, name)`. `close()` deletes the topics it created so repeated runs don't fill the disk (consumer groups go with them). - [ ] `IggyBenchmarkProducer`. One `AsyncIggyTcpClient` per producer (a VSR session is bound to one connection). `sendAsync` builds a `MessageHeader` with `originTimestamp` set to the current time in microseconds and wraps the payload untouched. `Message.of(...)` leaves that field at zero and `BytesSerializer` sends whatever the header holds, the server never writes it, and the polled header returns the absolute value (verified end to end against a server built from master). Messages accumulate in one buffer per producer and flush by size or linger as one `sendMessages` call with `Partitioning.balanced()`: the SDK resolves one partition per call and rotates, so each batch lands on one partition. With a key distributor the producer buckets by `XxHash32.hashUnsigned(key) % partitions` (the mapping `Partitioning.messagesKey` uses) and sends each bucket with `Partitioning.partitionId(p)`. Every message future completes when its batch reply arrives. The reply means committed to the rep licated journal and visible to consumers, not fsynced (defaults `messages_required_to_save = 1024`, `enforce_fsync = false`). Completion callbacks run on the Netty event loop, so the continuation completes OMB's futures and nothing else. The buffer is touched by the load thread, the linger timer and the reply thread, and `close()` can race an in-flight `sendAsync`, so it needs a lock. - [ ] `IggyBenchmarkConsumer`. One client per consumer (group membership is the connection's VSR client id). Creates the consumer group named after `subscriptionName` when `getConsumerGroup` returns empty, tolerating the conflict when sibling consumers race, then joins it. Polls on a dedicated thread with `Consumer.group(ConsumerId.of(name))`, `PollingStrategy.next()` and `autoCommit = true`, joining each poll future on that thread. One poll serves one partition of the assignment, so an empty poll doesn't mean drained: keep polling and back off only after a full empty cycle. Each message goes to `messageReceived(payload, originTimestamp / 1000)`. Backlog workloads pause consumers by blocking inside the callback, which only works because the callback runs on the poll thread and never on the event loop. `close()` stops the loop, leaves the group, closes the client. - [ ] `driver-iggy/iggy.yaml`, a one-minute smoke workload (`warmupDurationMinutes: 0`, `testDurationMinutes: 1`), and `driver-iggy/README.md` with the full sequence: start `iggy-server --with-default-root-credentials` (a fresh data dir otherwise generates a random root password), `mvn -DskipTests install` (at current OMB master add `-Dspotless.check.skip=true -Dspotbugs.skip=true`, `benchmark-framework` fails its own checks since #443 and upstream CI is red), untar `package/target/openmessaging-benchmark-0.0.1-SNAPSHOT-bin.tar.gz`, and from inside it `bin/benchmark --drivers driver-iggy/iggy.yaml workloads/1-topic-1-partition-1kb.yaml`. With no `--workers` and no `workers.yaml` in the working directory OMB runs one in-process worker, enough for the MVP. The README says which topic options a number was taken with. Done when `1-topic-1-partition-1kb.yaml` (50k msg/s, 16 min with warm-up) and `max-rate-1-topic-16-partitions-1kb.yaml` (6 min) run against a local `iggy-server` 0.9.0 with non-zero `publishLatency*` and `endToEndLatency*` series in the result JSON, and `bin/create_charts.py` renders it. `bin/benchmark` exits 0 even when a run fails, so check the JSON, not the exit code. ## Later - Terraform and Ansible under `driver-iggy/deploy/`, mirroring `driver-kafka/deploy`, so cloud runs use the same instance types as the Kafka and Redpanda numbers. - A page on iggy.apache.org on running OMB against Iggy. - Topic knobs (`enforce_fsync`, `messages_required_to_save`, compression) through `TopicOptions`, a 3-node cluster run, TLS, HTTP and QUIC transports. - CI smoke job in apache/iggy that clones OMB and runs the smoke workload against a server built from the PR, asserting on the result JSON. - A shaded SDK artifact published from apache/iggy, so the OMB, Flink and Pinot integrations stop carrying their own relocation. - Converter from OMB result JSON to `bench-report` so runs show up in the bench dashboard. ## Risks - No OMB module shades today, so the shaded driver jar is a first there. If upstream refuses it, the alternatives are an OMB-wide netty modernisation PR (rewrite the `bin.xml` excludes and move bookkeeper, pravega, rabbitmq and async-http-client onto one netty line) or the shaded SDK artifact from the Later list. - Every `AsyncIggyTcpClient` builds its own Netty event loop group (default 2 threads per core) and nothing lets clients share one, so thread count scales with producers plus consumers. Workloads with a large `producersPerTopic` times `topics` need that fixed in the SDK first. - Joining a group is cooperative with a 30 s `rebalancing_timeout`: with `consumerPerSubscription > 1`, late joiners get partitions only after the owner drains them or the timeout passes. Expect a startup transient inside the 60 s readiness window. - Upstream review can be slow. The fork stays runnable on its own, and there is no second copy of the driver in apache/iggy. - Alternative not taken: a Gradle module in `foreign/java` depending on `project(":iggy")`. It would track SDK master but needs `driver-api` from `mavenLocal()` (not on Maven Central) plus the driver jars on an exported `CLASSPATH` (`bin/benchmark` prepends it to `lib/*`), and it would become a duplicate once upstreamed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
