ryerraguntla opened a new pull request, #3519:
URL: https://github.com/apache/iggy/pull/3519
## Which issue does this PR address?
Closes #3421
## Rationale
Existing Kafka clients cannot talk to Iggy without a protocol translation
layer. Issue #3421 calls for a foundation-layer TCP listener on the Kafka wire
port (9093) so that any Kafka-compatible client can eventually route messages
into Iggy streams without any client-side changes. This PR delivers that
foundation: decode, version-firewall, and stub responses with no Iggy backend
integration yet.
## What changed?
Before this PR, Iggy had no surface that spoke the Kafka binary protocol.
Kafka clients connecting on port 9093 would receive no response or a connection
reset.
This PR adds a new `gateways/kafka/` workspace crate (`iggy-gateway-kafka`)
with a `tokio`-based TCP listener that accepts Kafka wire connections,
auto-detects request header v1/v2, enforces a version-firewall
(`SUPPORTED_RANGES`), decodes and stub-encodes six API keys (ApiVersions 18,
Metadata 3, Produce 0, Fetch 1, ListOffsets 2, CreateTopics 19), and rejects
unsupported keys or versions with `UNSUPPORTED_VERSION` (error 35) without
dropping the connection. A `kafka-message-gen` helper tool generates golden
wire fixtures used by the regression suite.
Key implementation notes:
- Produce hot path keeps RecordBatch as opaque `Bytes` — no RecordBatch
decode in this layer.
- `BrokerAdvertise` is `Arc`-cloned per connection; no per-connection string
allocation.
- Frame read uses `BytesMut` + `read_buf` loop; no zero-initialised heap
allocation.
- `SUPPORTED_RANGES` is the single source of truth; `ApiVersions` advertises
exactly what the firewall allows.
103 regression tests across 12 suites cover: primitive codec round-trips,
adversarial wire inputs, header v1/v2 parsing, version boundary matrix, per-API
golden byte fixtures, stub error codes, and full TCP round-trips via a real
`KafkaServer` listener.
## Local Execution
- Passed
- Pre-commit hooks ran
```bash
# Fixture generation (required for decode_validation_tests)
cargo run -p kafka-message-gen -- generate \
--output gateways/kafka/tools/kafka-tool/kafka_messages \
--api-key 0 --api-key 1 --api-key 2 --api-key 19
# Regression suite
cargo test -p iggy-gateway-kafka
# Lint
cargo clippy -p iggy-gateway-kafka --all-targets -- -D warnings
cargo fmt --all --check
```
All 103 tests pass. `clippy -D warnings` clean. Manual smoke-test procedure
followed per `gateways/kafka/docs/MANUAL_TESTING.md` (ApiVersions, Metadata,
Produce, Fetch round-trips; version firewall; oversized frame rejection).
## AI Usage
1. **Tools:** Claude Sonnet 4.6 (Cursor Cloud Agent)
2. **Scope:** Protocol codec implementation (`requests.rs`, `responses.rs`,
`codec.rs`, `header.rs`), server framing (`server.rs`), error type
(`error.rs`), regression test suites, documentation (`SCOPE.md`,
`TEST_SUITE.md`, `MANUAL_TESTING.md`, `kafka_api_keys_reference.md`), and the
`kafka-tool` fixture generator.
3. **Verification:** Every generated file was reviewed line by line. The
103-test regression suite was run locally against the implementation. Manual
testing followed the procedure in `MANUAL_TESTING.md` using `kcat` and the
`kafka-message-gen` tool against a live listener. Golden byte fixtures pin
exact wire output against known-correct Kafka responses.
4. **Explainability:** Yes — the author can explain every line. The codec
follows the Kafka protocol specification field-by-field; non-obvious decisions
(opaque RecordBatch, `SUPPORTED_RANGES` governance model,
`Arc<BrokerAdvertise>` per-connection, `varint=0` null compact array) are
documented inline or in `SCOPE.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
