mlevkov opened a new pull request, #2925: URL: https://github.com/apache/iggy/pull/2925
## Summary Adds a **generic HTTP sink connector** that delivers consumed Iggy stream messages to any HTTP endpoint — webhooks, Lambda functions, REST APIs, or SaaS integrations. - **4 batch modes**: `individual` (one request per message), `ndjson` (newline-delimited), `json_array` (single array), `raw` (bytes) - **Retry with exponential backoff**: transient error classification (429/5xx), `Retry-After` header respect, configurable max retries/delay - **Metadata envelope**: optional Iggy metadata wrapping (offset, timestamp, stream, topic, partition), base64 for binary payloads - **Partial delivery**: in `individual`/`raw` modes, continues after per-message failure with consecutive failure abort threshold - **47 unit tests**, zero clippy warnings ### Design Follows established connector patterns: | Pattern | Source | Applied | |---------|--------|---------| | `Option<reqwest::Client>` with lazy init in `open()` | MongoDB, ES, PostgreSQL | Connection pooling, configurable timeout | | `AtomicU64` counters | MongoDB | Track requests, messages, errors, retries | | Retry with backoff | PostgreSQL, MongoDB | Transient error classification + exponential backoff | | Config via serde `Deserialize` | All sinks | `HttpSinkConfig` with `Option` fields, resolved in `new()` | | `sink_connector!` macro | All sinks | FFI export | All retry logic lives inside `consume()` — the runtime currently discards the `Result` from `consume()` (`sink.rs:585`) and commits offsets before processing. These are known upstream issues documented in the README. ### Files ``` core/connectors/sinks/http_sink/ ├── Cargo.toml # workspace deps only (no new dependencies) ├── config.toml # example configuration ├── README.md # usage, config reference, known limitations └── src/lib.rs # implementation + 47 unit tests (1757 lines) ``` ### Configuration ```toml [plugin_config] url = "https://api.example.com/ingest"; method = "POST" batch_mode = "ndjson" timeout = "30s" max_retries = 3 retry_delay = "1s" retry_backoff_multiplier = 2.0 include_metadata = true [plugin_config.headers] Authorization = "Bearer my-token" ``` See [README](core/connectors/sinks/http_sink/README.md) for full configuration reference and examples. ### Context This was discussed in [GitHub Discussion #2901](https://github.com/apache/iggy/discussions/2901) with positive feedback from the maintainers. The design follows all patterns established by the existing 6 sink connectors. ## Test plan - [x] 47 unit tests covering config resolution, payload conversion, metadata envelope, retry delay computation, transient status classification, UUID formatting, TOML deserialization, `open()` validation - [x] Zero clippy warnings (`cargo clippy -p iggy_connector_http_sink -- -D warnings`) - [x] Compiles as both `cdylib` and `lib` - [ ] Manual test with `iggy-connectors` runtime (Try It section in README) - [ ] Integration test with mock HTTP server (future follow-up) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
