ryerraguntla opened a new issue, #3728: URL: https://github.com/apache/iggy/issues/3728
### Bug description ## Body ### Summary The Elasticsearch sink integration test (`elasticsearch_sink_stores_json_messages`) can fail with `HealthCheckFailed` for `iggy-connectors` after many retries. Connectors runtime logs stop at `Opening Elasticsearch sink...` and never reach a successful open. Observed locally against a reused Testcontainers Elasticsearch instance; matches Discord failure reports. [ [elastic_connector_files.zip](https://github.com/user-attachments/files/30239708/elastic_connector_files.zip)](url) Root cause is a combination of: 1. Sink `open()` can hang indefinitely on Elasticsearch HTTP (no client timeout applied), and the connectors HTTP API only binds **after** plugin open completes. 2. Fixture reuse (`ReuseDirective::Always`, container name `iggy-test-elasticsearch`) can attach to a days-old container without re-validating cluster health the same way as a cold start. 3. On macOS, `localhost` can resolve to IPv6 `::1` first while Docker publishes IPv4 `0.0.0.0:HOST→9200`, so the elasticsearch-rs client can hang while a request health probe still looks fine. 4. Harness readiness polled a generic HTTP path and did not fail fast if `iggy-connectors` crashed during open. Stale leftover indices alone do **not** appear to break the happy path on a healthy host, but accumulated leftovers on a reused container are still worth cleaning so the shared node stays healthy across runs. ### Impact - Flaky CI / local integration runs for Elasticsearch sink (and related harness readiness). - Hard to diagnose: failure surfaces as connectors health timeout, not as an Elasticsearch client timeout error. ### Proposed fix Branch in progress: `fix/elasticsearch-sink-test-flakiness` - **Sink:** apply `timeout_seconds` (default 30s) on `TransportBuilder` so hung ES I/O cannot block FFI `open()` forever. - **Fixture container:** prefer `http://127.0.0.1:{port}`; always poll `/_cluster/health` after attach; if the reused container is unusable, `docker rm -f iggy-test-elasticsearch` once and recreate (named harness container only — does not touch other Elasticsearch containers). - **Fixture container:** best-effort sweep of stale test indices (`iggy_messages_*`, `test_documents_*`) older than 30 minutes so prior empty/partial indices do not accumulate on the reused node. - **Sink fixture:** refresh index before each document-count poll; keep last error in timeout messages. - **Harness:** poll connectors `/health` and fail fast with `ProcessCrashed` if the child dies during retries. ### Acceptance criteria - [ ] `timeout_seconds` is applied to the Elasticsearch transport client. - [ ] Elasticsearch fixture readiness is re-checked on every setup, including reuse attach. - [ ] Wedged `iggy-test-elasticsearch` is removed and recreated once; other Docker Elasticsearch containers are never targeted. - [ ] Stale harness test indices older than the age threshold are cleaned best-effort before the test proceeds. - [ ] Connectors runtime readiness uses `/health` and detects a dead process during retries. - [ ] `elasticsearch_sink_stores_json_messages` passes locally against reused container. ### References - Discord report / log set (connectors stuck at "Opening Elasticsearch sink...", harness `HealthCheckFailed`). - Local analysis notes (commit where flake was investigated): `6f1d5485d06c3c43afd2510e432b2f7c1a3ac41a` - Related discussion context: connectors / integration test stability. ### Out of scope - Changing connectors runtime to bind HTTP before plugin `open()` (possible follow-up for clearer "starting" health). - Broader Test containers reuse policy changes for non-Elasticsearch fixtures. [elastic_connector_files.zip](https://github.com/user-attachments/files/30239822/elastic_connector_files.zip) ### Affected area / component Connectors ### Deployment Docker (DockerHub image) ### Versions _No response_ ### Hardware / environment _No response_ ### Sample code _No response_ ### Logs _No response_ ### Iggy server config _No response_ ### Reproduction _No response_ ### Contribution - [x] I'm willing to submit a pull request to fix this bug ### Good first issue - [ ] I think this could be a good first issue for a new contributor -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
