GitHub user mlevkov added a comment to the discussion: Proposal: Generic HTTP Sink Connector
## PR and Issues Filed The HTTP sink connector implementation is ready for review: - **PR**: #2925 — `feat(connectors): add generic HTTP sink connector` - 4 batch modes, retry with exponential backoff, metadata envelope, 47 unit tests - Follows all patterns from existing sinks (MongoDB, PostgreSQL, Elasticsearch) - No new dependencies — all workspace deps While analyzing the runtime to inform the design, we found two issues affecting **all existing sinks**. Filed separately: - **Issue #2927** — `consume()` return value discarded in sink runtime (`sink.rs:585`) - **Issue #2928** — `PollingMessages` auto-commit commits offsets before sink processing ### A note on integration testing The HTTP sink PR currently includes 47 unit tests covering config resolution, payload conversion, retry logic, and serialization. For full end-to-end testing, I noticed the test harness in `core/integration` has excellent infrastructure — the `#[iggy_harness]` macro, `TestFixture` trait, and even a WireMock container in `fixtures/wiremock.rs`. No existing sink connector has in-crate integration tests — they all live in `core/integration/tests/connectors/`. I would like to add integration tests for the HTTP sink there as well (using WireMock to receive and verify HTTP deliveries). This would cover the full path: send message to Iggy → connector consumes → HTTP request received → verify format/headers/retry behavior. Would this be welcome as part of the same PR, or better as a follow-up? Happy to either way. GitHub link: https://github.com/apache/iggy/discussions/2919#discussioncomment-16104535 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
