egalpin commented on issue #27094:
URL: https://github.com/apache/beam/issues/27094#issuecomment-7022514988

   For anyone who finds this later: I reproduced this with a 429 storm against 
a real Elasticsearch. The likely cause was that, before Beam 2.61.0, Runner v2 
did not call `@Teardown` on the DoFn instance of a failed bundle. Each failed 
bundle therefore leaked ElasticsearchIO's `RestClient` and its I/O threads, 
until the worker ran out of threads (`OutOfMemoryError: unable to create native 
thread`) and new clients could not start. This was fixed in 2.61.0 by #32522. 
On 2.61.0 and later, a 429 storm only fails bundles while it lasts, and writes 
resume as soon as Elasticsearch recovers. Details in #41463.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to