mfyuce opened a new issue, #3814: URL: https://github.com/apache/iggy/issues/3814
### Bug description `quickwit_sink` predates the shared connector retry helpers (`sdk/src/retry.rs` arrived with the InfluxDB sink in #2933, roughly ten months after the sink itself) and was never adapted to them. Compared to the other HTTP sinks it is missing several guards, and each gap turns a recoverable condition into a hang or into silent data loss. **1. No request timeout.** `create_index`, `has_index` and `ingest` all share a bare `reqwest::Client::new()`, which has no timeout. Under a network partition, or against a Quickwit node that is still starting, these calls block indefinitely. The connector neither makes progress nor reports a failure. **2. No retries.** A transient 5xx or a connection reset during `open()` fails the connector outright, and the same during `ingest()` drops the batch. Every other HTTP sink routes these through retry middleware with exponential backoff. **3. Concurrent index creation deadlocks the connector.** `open()` calls `has_index()` and, if absent, `create_index()`. Two instances starting together, or one restarting next to a live one, both observe `has_index() == false` and both POST. The loser receives `409 Conflict`, which is treated as a fatal `InitError`, so that instance never opens even though the index it wanted now exists. **4. Non-JSON payloads are discarded.** `consume()` matches only `Payload::Json`; everything else hits a `warn!` and is dropped. A topic carrying raw or text messages silently loses every record, with the offset already committed. **5. No readiness gate.** There is no health probe before the first request, so startup races the node coming up. **6. Trailing slash in `url` produces a malformed path.** Every request rebuilds its URL from `self.config.url` with `format!` and nothing trims a trailing slash, so a configured `http://host:7280/` yields `//api/v1/...`. ### Affected area / component Connectors ### Deployment Compiled from source ### Reproduction For (3), start two `quickwit_sink` connectors against the same index before it exists. One opens, the other fails with `InitError` on the 409 and stays down. For (4), publish a message with a non-JSON payload to a topic the sink consumes. The record is dropped, the offset advances, and only a `warn!` is emitted. For (6), set `url = "http://localhost:7280/"` in `[plugin_config]` and observe the request path. ### Proposed scope #3523 addresses all six: - Build the HTTP client in `open()` with a configurable `timeout`, wrapped in `build_retry_client` from `iggy_connector_sdk::retry`, with configurable retry delay bounds. - Probe `/health/readyz` through `check_connectivity_with_retry` before touching the index. - Absorb `409 Conflict`, and a `400` whose body reports the index already exists, as success in `create_index`. - Wrap non-JSON payloads instead of dropping them: raw bytes are parsed as JSON when possible, otherwise carried as text, or base64 when not valid UTF-8. - Resolve the base URL once in `open()` with the trailing slash trimmed. - Add unit tests for payload handling and config defaults, alongside the existing integration suite in `core/integration/tests/connectors/quickwit/`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
