SEPURI-SAI-KRISHNA commented on PR #12313: URL: https://github.com/apache/seatunnel/pull/12313#issuecomment-5789049649
Leaving this here at @SEZ9's suggestion from the #12290 thread, so it is on the record before this lands. It is a javadoc accuracy point, not an objection to the change. The new class comment says the reader's inter-split sleep gives barrier injection "a real, contention-free window once per second". Two thirds of that is right and one third is not. Right: the window is one second long, because `FakeSourceReader` sleeps `Thread.sleep(1000L)` once a split is complete. Also right: "that one-second gap drains only a quarter of the queue's capacity", since 1s at ~500 rows/sec is 500 rows against the 2048 capacity. Not right: the frequency. The window occurs once per split, not once per second, and under sustained backpressure a split does not complete in a second. With `row.num = 2000000` and `split.num = 500` each split is 4000 rows, and the sink is throttled to ~500 rows/sec by `write_delay_ms = 2`, so the reader cannot finish emitting a split faster than the sink drains it: roughly 8 seconds, then the 1 second sleep, so the window recurs about every 9 seconds rather than every second. The conclusion still holds against `checkpoint.interval = 15000`, so this does not change the chosen numbers. It does change how much room they leave: the margin is about 1.7x, not the 15x the current wording implies, and that margin is the whole justification for picking `split.num = 500`. Worth stating accurately so a later edit to `row.num`, `split.num` or `write_delay_ms` is made against the real headroom. For context on why I was in this code: I hit `BackpressureSlowSinkIT` while triaging `engine-v2-it` on #12290. The failure there is at the first-checkpoint gate, `completed >= 1` never becoming true within 2 minutes while the job was healthy and RUNNING, which is the same starvation this PR describes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
