ryerraguntla opened a new pull request, #4231: URL: https://github.com/apache/iggy/pull/4231
### Summary - Wires `Metadata` ([#3534](https://github.com/apache/iggy/issues/3534)), `CreateTopics` ([#3538](https://github.com/apache/iggy/issues/3538)), and `ListOffsets` ([#3537](https://github.com/apache/iggy/issues/3537)) to the `IggyBridge` module - Adds a `TopicCatalog` trait seam (`bridge/mod.rs`) so the protocol-level handler tests stay fast, in-process unit tests against an in-memory `FakeBridge`, while a smaller set of tests exercises real `IggyBridge` behavior against a spawned `iggy-server`. - Fixes several correctness gaps found while wiring these handlers up against real client behavior (see "Bugs fixed" below) — none of these were reachable before this PR, since nothing called the bridge from a live handler yet. - `main.rs` now connects to Iggy before binding the Kafka listener (fail fast and loud if Iggy is unreachable) and cleanly closes the bridge connection on graceful shutdown. Closes #3534, #3537, #3538. Depends on #3533 (already merged to `master`). Produce/Fetch wiring is tracked separately as #3535/#3536. ### Why [#3534](https://github.com/apache/iggy/issues/3534)/[#3537](https://github.com/apache/iggy/issues/3537)/[#3538](https://github.com/apache/iggy/issues/3538) ask for Metadata, ListOffsets, and CreateTopics to stop being stubs and start answering from a real Iggy backend, using the bridge module #3533 already built and merged. This is that wiring — three protocol handlers now call through `TopicCatalog` (backed by a real `IggyBridge` in production) instead of returning a hardcoded error code. Produce and Fetch are deliberately out of scope here: they need to persist and read actual message payloads, not just topic metadata, which is a bigger and separately-tracked change. ### What's included **Metadata (#3534)** - `handle_metadata` lists every topic the bridge currently knows about for a null `topics` array, or looks up specific names individually so each name can carry its own per-topic error code (partial failure doesn't fail the whole response). - Real per-partition broker/leader/replica info in the response. Still single-broker: every partition reports broker id 1, since this gateway has no second broker to be anything else. - Null topics array ("all topics") is no longer conflated with an explicit empty array (the shape `AdminClient.describeCluster()` sends per KIP-4 — "no topics, just cluster info"). **CreateTopics (#3538)** - `create_one_topic` provisions via `bridge.ensure_stream_and_topic`, one result per requested topic in a multi-topic request — one bad topic never fails the whole batch. - Non-empty per-topic Kafka `configs` rejected (`INVALID_CONFIG`, 40) — no Iggy topic option any of them maps onto today. - `validate_only` (KIP-4 dry run) is checked on request format alone, without calling the bridge or creating anything. - An already-existing topic returns `TOPIC_ALREADY_EXISTS` (36), not a silent success — `CreateTopics` is not an upsert, unlike `ensure_stream_and_topic`'s own idempotent internal contract. - `replication_factor` is accepted on the wire but not applied (Iggy replication is cluster-wide Raft, not a per-topic knob) — documented in the README rather than silently misrepresented. **ListOffsets (#3537)** - `resolve_list_offsets_topic` resolves `earliest` (-2) / `latest` (-1) sentinels per requested partition via one `bridge.high_watermarks` call per topic (not one round trip per partition). - A real timestamp value (not a sentinel) returns `INVALID_REQUEST` (42) — no timestamp-to-offset lookup exists yet. - A per-partition failure doesn't fail the topic's other partitions. **Bridge / error mapping** - New `TopicCatalog` trait (`bridge/mod.rs`), implemented for `IggyBridge` as pure delegation — production code pays no vtable cost it didn't already pay for (every real call round-trips over TCP regardless). - New `IggyBridge::get_kafka_topic`/`list_kafka_topics` for read-only Metadata lookups that must not create anything on the Iggy side just by asking. - `ensure_stream_and_topic` now rejects `partition_count == 0` itself (`BridgeError::InvalidPartitionCount`), not only at the `CreateTopics` wire-validation layer — it's reachable directly through the public trait now, so the invariant has to hold there too. **Startup / shutdown** - `main.rs` connects to Iggy (`IggyBridgeConfig::from_env` + `IggyBridge::connect`) before binding the Kafka listener — an unreachable Iggy backend now fails fast with a non-zero exit instead of accepting Kafka connections that could only ever answer bridge-backed requests with a connectivity error. - Graceful shutdown now closes the bridge connection cleanly via `Arc::try_unwrap` once every clone the server handed out is done with it, instead of relying only on `Drop`. ### Bugs fixed Found while wiring real clients against these handlers for the first time — none were reachable before this PR, since nothing called the bridge from a live connection yet: 1. **`CreateTopics`'s `-1` (KIP-464 broker-default) sentinel ignored manual assignments.** `validate_create_topic_shape` hardcoded a 1-partition default for `num_partitions == -1` regardless of whether the request carried a manual partition `assignments` list. A client using `NewTopic`'s manual-assignment constructor (e.g. for rack-aware placement) got a silently wrong 1-partition topic. Now resolves the real count from `assignments.len()` when present. 2. **Mismatched explicit `num_partitions` vs. manual `assignments` length was silently accepted.** Now returns `INVALID_REPLICA_ASSIGNMENT` (39), matching real Kafka's own code for this condition. 3. **`IggyError::TooManyPartitions` mapped to the wrong Kafka error code.** It used `INVALID_PARTITIONS` (37) — whose own text, per `kafka-protocol`'s table, is "number of partitions is below 1," the opposite condition from "too many." A client asking for too many partitions got an error message contradicting its own request. Now maps to `INVALID_REQUEST` (42). 4. **Bulk topic listing could report a topic under the wrong identity.** `list_kafka_topics`'s `default_stream` bulk listing reported every physical topic name found there as a Kafka topic identity-resolving to itself — false when that name also happens to be another mapping override's *key*, since `resolve()` always prefers the override. A client listing all topics could see a name that, looked up individually, answers about a completely different physical topic. Bulk-listed names colliding with an override key are now skipped; the override loop already reports them correctly under their real target. 5. **A real librdkafka client's trailing padding byte was rejected as malformed.** A wire trace against `kcat -L` (librdkafka 2.14.2) showed a fully-valid Metadata v9 request followed by one extra zero byte past the schema's own end. Up to 8 trailing zero bytes are now tolerated as benign encoder padding instead of failing the whole request as malformed. ### Test plan - [x] `cargo build -p iggy-gateway-kafka --all-targets` - [x] `cargo clippy -p iggy-gateway-kafka --all-features --all-targets -- -D warnings` — clean - [x] `cargo fmt --all -- --check` — clean - [x] `cargo sort --no-format --workspace` — clean - [x] `cargo test -p iggy-gateway-kafka --lib --bins` — 104 passed - [x] Protocol-level suites against `FakeBridge` (`api_handler_tests`, `response_negative_tests`, `version_firewall_tests`, `listener_robustness_tests`, `golden_wire_fixtures_tests`, `header_tests`, `broker_advertise_tests`, `server_integration_tests`, `fixtures_canary_tests`) — 137 passed - [x] `cargo test -p iggy-gateway-kafka --test bridge_iggy_integration_tests` against a real, spawned `iggy-server` — 29 passed - [x] `cargo test -p iggy-gateway-kafka --test gateway_bridge_e2e_tests` — real Kafka wire bytes decoded by the real protocol layer, through a real `KafkaGateway`, into a real `IggyBridge` against a real, spawned `iggy-server` (the only suite that exercises this exact combination) — 9 passed All verification run locally against the current working tree (commits + uncommitted changes combined) on 2026-09-19. ## AI Usage If AI tools were used, please answer: 1. Which tools? Claude, Cursor 2. Scope of usage? Implementation and code review 3. How did you verify the generated code works correctly? Human walkthrough and Testing 4. Can you explain every line of the code if asked? Yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
