ryerraguntla commented on code in PR #4043: URL: https://github.com/apache/iggy/pull/4043#discussion_r4001643188
########## gateways/kafka/README.md: ########## @@ -58,6 +58,79 @@ Before check-in, run the procedure in [docs/MANUAL_TESTING.md](docs/MANUAL_TESTI See [docs/SCOPE.md](docs/SCOPE.md) for [#3421](https://github.com/apache/iggy/issues/3421) deliverables, supported API key/version table, and post-foundation TODO backlog. +## Iggy bridge ([#3533](https://github.com/apache/iggy/issues/3533)) + +`src/bridge/` is the SDK integration layer: connects to Iggy, maps Kafka topics to Iggy +streams/topics, provisions them on demand, and looks up the high watermark for `ListOffsets`. +**Not wired into the live Produce/Fetch dispatch path yet** - that lands with +[#3535](https://github.com/apache/iggy/issues/3535)/[#3536](https://github.com/apache/iggy/issues/3536). +Exercised today by `bridge`'s own unit tests and `tests/bridge_iggy_integration_tests.rs` (spawns a +real `iggy-server`). + +### Connection config + +| Variable | Default | Description | +| --- | --- | --- | +| `IGGY_KAFKA_IGGY_ADDR` | `127.0.0.1:8090` | Address of the Iggy server to bridge to | +| `IGGY_KAFKA_IGGY_USERNAME` | `iggy` | Iggy username | +| `IGGY_KAFKA_IGGY_PASSWORD` | none - **required** | Iggy password. No default: `iggy-server` only uses the well-known `iggy`/`iggy` root credentials when started with `--with-default-root-credentials` (dev-only); otherwise it generates a random password, so a hardcoded default here could never be right and would invite running as root unnoticed | +| `IGGY_KAFKA_IGGY_STREAM` | `kafka` | Default Iggy stream for a Kafka topic with no explicit mapping override | +| `IGGY_KAFKA_TOPIC_MAP_PATH` | unset | Path to a topic-mapping TOML file (see below); omit to use only the default rule | + +The connection retries a fixed, bounded number of times (`RECONNECTION_RETRIES`, not the Iggy SDK +client's own default of unlimited retries, one dial per second, forever), and the whole connect +attempt - retries included - is capped at `CONNECT_TIMEOUT` (15s) wall-clock, so a bridge call +fails in bounded time whether the address refuses the connection or silently drops it, instead of +blocking the calling task indefinitely. See `IggyBridge::connect`'s doc comment. + +### Topic mapping + +Default rule, no config file needed: a Kafka topic `orders` maps to Iggy stream +`IGGY_KAFKA_IGGY_STREAM` (default `kafka`), topic `orders` - the Kafka topic name carries over +unchanged. Override specific topics with a TOML file: + +```toml +default_stream = "kafka" + +[topics.orders] +stream = "billing" +topic = "orders_v2" + +# A Kafka topic name containing dots needs the key quoted, or TOML parses it as nested +# tables ([topics.org] containing [apache] containing [kafka]) instead of one topic named +# "org.apache.kafka.events". +[topics."org.apache.kafka.events"] +stream = "billing" +topic = "kafka_events" +``` + +Point `IGGY_KAFKA_TOPIC_MAP_PATH` at the file to load it; topics not listed under `[topics.*]` Review Comment: **Fixed** - both rules are now stated explicitly in the README's topic mapping section. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
