egalpin commented on issue #27094: URL: https://github.com/apache/beam/issues/27094#issuecomment-7022514988
For anyone who finds this later: I reproduced this with a 429 storm against a real Elasticsearch. The likely cause was that, before Beam 2.61.0, Runner v2 did not call `@Teardown` on the DoFn instance of a failed bundle. Each failed bundle therefore leaked ElasticsearchIO's `RestClient` and its I/O threads, until the worker ran out of threads (`OutOfMemoryError: unable to create native thread`) and new clients could not start. This was fixed in 2.61.0 by #32522. On 2.61.0 and later, a 429 storm only fails bundles while it lasts, and writes resume as soon as Elasticsearch recovers. Details in #41463. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
