tokencraftdata opened a new pull request, #4538: URL: https://github.com/apache/iggy/pull/4538
Relates to #3530 ## Why Iggy has sinks for databases and search engines but nothing for Redis. This sink lets a stream feed Redis for caching, materialized views and event buffering, by appending messages to Redis Streams with pipelined `XADD`. ## Design choices - **Idempotent writes.** With `deterministic_ids = true` the entry ID is derived from the message offset. If a batch is sent twice (for example a retry after a lost response), Redis rejects the repeated IDs and the sink counts them as duplicates instead of writing twice. Offsets only grow inside one partition, so each stream, topic and partition gets its own key. - **Invalid messages are skipped and counted.** The runtime does not redeliver a failed batch, so returning an error for one bad `text`/`json` payload or header would drop the valid messages with it. This is documented in the sink README. - **`response_timeout` (default 5s).** The redis crate default is 500ms, which is too short for a loaded server. A timeout counts as a transient failure and goes through the normal retry path. - **Retries** cover connection-level failures only, capped at 3 attempts. ## Testing Run locally: - `cargo fmt --all`, `cargo sort --no-format --workspace`, `cargo clippy --all-features --all-targets -- -D warnings` - 22 unit tests in `redis_sink` - 5 integration tests against a real Redis container (`cargo test -p integration -- connectors::redis`): JSON entries with offset IDs, metadata disabled, raw binary payload, user headers (text and binary), and replay of an existing entry ID being skipped - `prek run --from-ref a2895fdc6 --to-ref HEAD` 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
