egalpin opened a new pull request, #41463: URL: https://github.com/apache/beam/pull/41463
Addresses #27094. ## Problem `BulkIOBaseFn` (the DoFn behind `ElasticsearchIO.Write` and `BulkIO`) creates its `RestClient` once, in `@Setup`. If that client's I/O reactor stops, every later request fails with `Request cannot be executed; I/O reactor status: STOPPED`, and the client never recovers. A streaming pipeline then keeps failing bundles on that DoFn instance until the instance is discarded, e.g. when the worker restarts. Network-level outages are not the problem. When Elasticsearch refuses connections, hangs until the socket timeout, or resets connections, the `RestClient` stays running and recovers on its own once the cluster is back: requests fail with an `IOException`, and the client retries the host it marked dead. The new Docker tests below confirm this on Elasticsearch 7, 8 and 9, without the fix. The reactor stops for good when an `Error` (e.g. an `OutOfMemoryError` under load) is thrown on one of the client's I/O dispatcher threads: `RestClient` and the Apache async client catch `Exception`s there, but not `Error`s. ## Fix Before using the client, at `@StartBundle` and before each bulk request and retry, `BulkIOBaseFn` checks `RestClient#isRunning()`. If the client has stopped, it creates a new one and closes the stopped one. Cost: - `isRunning()` reads the client's in-memory status and sends no request. - A running client is always reused. A client is created only to replace a stopped one, so each DoFn instance still holds at most one client. - If creating the replacement fails, the bundle fails and the next bundle tries again. `isRunning()` exists in `elasticsearch-rest-client` 7.17 (the version ElasticsearchIO compiles against) and in 8.x and 9.x, which the ElasticsearchIO test modules run with. ## Tests - `ElasticsearchIOClientRecoveryTest` (unit tests, `:sdks:java:io:elasticsearch`, no Docker): a fake `_bulk` endpoint served by a local HTTP server. Stops a real client's I/O reactor and checks that the DoFn replaces the client once, at bundle start or mid-bundle, and keeps reusing a running one (1 client over 100 bundles). - `ElasticsearchIOOutageTest` (Testcontainers, Elasticsearch 7, 8 and 9, via the new `ElasticsearchIOOutageTestCommon`): Elasticsearch behind a Toxiproxy container. One DoFn instance writes before, during and after an outage. Outages: connection refused, hanging connections, connection reset, and a stopped I/O reactor. Asserts the write after the outage succeeds, and that the DoFn creates no client for the network outages and exactly one replacement for the stopped reactor. Results, run locally: | Test | With the fix | Without the `isRunning()` check | |---|---|---| | Unit tests (5) | 5 pass | 2 pass, 3 fail | | Outage: connection refused (ES 7, 8, 9) | pass | pass | | Outage: hanging connections (ES 7, 8, 9) | pass | pass | | Outage: connection reset (ES 7, 8, 9) | pass | pass | | Outage: stopped I/O reactor (ES 7, 8, 9) | pass | fail | Build changes: adds `library.java.testcontainers_toxiproxy` to `BeamModulePlugin` and uses it, plus `toxiproxy-java`, in `elasticsearch-tests-common`; adds junit and hamcrest as test dependencies of `:sdks:java:io:elasticsearch`, which had no tests of its own. ## Notes for reviewers - #27094 has no logs, so it is not confirmed that a stopped reactor is what froze the reporter's pipeline. If that happened, the worker logs would contain `I/O reactor terminated abnormally` (logged by the Apache async client) and `I/O reactor status: STOPPED`. - The issue was closed as stale; it may need reopening. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
