egalpin commented on PR #41463:
URL: https://github.com/apache/beam/pull/41463#issuecomment-7022513735
Thanks @Abacn, you were right that the tests showed little being fixed. I
went back to the report in #27094 and tried to reproduce it against a real
Elasticsearch (7.9.2) instead. The result is that this PR doesn't fix anything
that still exists, so I'm closing it.
**What #27094 describes:** a streaming job under a 429 storm that stayed
frozen until a Runner v2 update replaced the DoFn instances. On Runner v2, the
DoFn instance of a failed bundle is discarded, so bundles that kept failing
would have replaced the instance (and its `RestClient`) by themselves.
Something else had to keep the job stuck.
**What I ran:** one long-lived `BulkIO` DoFn writing 200-document bundles in
a loop, against an ES with a write thread pool of size 1 and a queue of 1,
while 8 other clients sent 1000-document bulks to it (60s storm, then 60s of
recovery). Each bundle had a 120s watchdog to catch hangs. I ran it with and
without `RetryConfiguration`, both reusing the DoFn instance after a failure
and discarding it the way Runner v2 does.
Results, on master's code:
- Reusing the instance: bundles fail with ordinary 429 write errors during
the storm (or succeed with `RetryConfiguration`), and writes resume as soon as
the storm ends. No bundle hung, and no client stopped.
- Discarding the instance **with** `@Teardown`, as Runner v2 has done since
Beam 2.61.0 (#32522): the same, with a flat thread count.
- Discarding the instance **without** `@Teardown`, as Runner v2 did before
2.61.0, so also when #27094 was filed: every failed bundle leaked its
`RestClient` and that client's I/O dispatcher threads (one per CPU core). After
about 630 failed bundles in 30s the JVM had about 11,600 threads and threw
`OutOfMemoryError: unable to create native thread`, and every new client
started with its I/O reactor already stopped.
That last case matches the report: once a worker has run out of threads,
nothing on it recovers until it is replaced. This PR's `isRunning()` check
doesn't help there: with the fix applied, the run leaked the same ~11,600
threads, and creating any new client failed with the same `OutOfMemoryError`.
Network outages (refused, hanging and reset connections) recover without the
fix, as the tests in this PR showed, and so did a 45s pause of the ES container
in this reproduction.
So the likely cause of #27094 was the missing `@Teardown` on Runner v2,
fixed in 2.61.0 by #32522 ("Fixed tearDown not invoked when DoFn throws on
Portable Runners"). I'll add a note to #27094 saying so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]