GitHub user mlevkov created a discussion: Feature idea: Per-message HTTP header forwarding in sink connectors
## Feature idea: Per-message HTTP header forwarding *Moved from [discussion #2919](https://github.com/apache/iggy/discussions/2919#discussioncomment-16300152) at @hubcio's request.* **Context**: While working on the HTTP sink PR (#2925), a question came up about dynamic per-message headers. Today, all HTTP headers are static (config-derived, same for every request). But there are real use cases for per-message headers. ### Use cases - `X-Iggy-Offset` / `X-Iggy-Topic` / `X-Iggy-Partition` — routing and dedup at the HTTP endpoint without body parsing - Forwarding Iggy user-defined message headers (`ConsumedMessage.headers`) as HTTP headers — e.g., a producer attaches `X-Correlation-ID` and the sink forwards it to the downstream endpoint ### Proposed design — additive layering ```rust // Static config headers (pre-built once in open(), cloned per-request) let mut request = build_request(self.method, client, &self.url) .headers(self.request_headers.clone()) .header("content-type", content_type); // Per-message dynamic headers (opt-in via config) if self.include_metadata_headers { request = request .header("x-iggy-offset", offset.to_string()) .header("x-iggy-topic", &topic_metadata.topic) .header("x-iggy-partition", partition_id.to_string()); } // Forward Iggy user headers as HTTP headers (opt-in) if self.forward_iggy_headers { if let Some(headers) = &message.headers { for (key, value) in headers { request = request.header( format!("x-iggy-{}", key.to_string_value()), value.to_string_value(), ); } } } ``` ### Config surface ```toml [plugin_config] # Include iggy metadata (offset, topic, partition) as HTTP headers include_metadata_headers = false # Forward iggy user-defined message headers as HTTP headers (prefixed with x-iggy-) forward_iggy_headers = false ``` This preserves the current pre-built `HeaderMap` optimization for static headers while layering dynamic per-message headers on top. Both options are opt-in and disabled by default. ### Design considerations - **Batch modes**: For `ndjson` and `json_array`, per-message headers only apply if the batch contains a single message, or we'd need to pick "representative" headers from the batch (e.g., first message, or common subset). For `individual` mode it's straightforward — each message is its own HTTP request. - **Header name conflicts**: If a user-defined Iggy header name conflicts with a static config header, the static header should win (config is explicit intent). Alternatively, we could log a warning. - **Performance**: Adding per-message headers is a few string allocations per request. For `individual` mode at high throughput this could add up, but it's behind an opt-in flag so the default path is unchanged. This could be a follow-up PR after the base HTTP sink (#2925) lands. Thoughts? GitHub link: https://github.com/apache/iggy/discussions/3029 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
