HyukjinKwon commented on PR #58:
URL:
https://github.com/apache/spark-connect-rust/pull/58#issuecomment-5446584126
Update: the actual root cause is now found and fixed, and the PR has pivoted
accordingly (title/description updated).
It was **not** the keepalive path and **not** test-specific. Via CI
RPC-tracing + a server jstack: the reattachable-execute driver fires a
`ReleaseExecute` after every response on a background pool, and each release
closure captures the operation's iterator. Our `release_execute` was
**unbounded**, so a stalled release kept its future pending → kept the iterator
alive → kept the iterator's open `ExecutePlan` HTTP/2 stream alive; enough of
those accumulate on the single shared connection that a later `ExecutePlan`
can't get a stream and the client blocks forever while the server is fully
idle. Cumulative → it landed on different tests depending on timing
(`test_raise_stop_iteration`, `test_udf_in_generate`, …).
Fix: bound `release_execute` with a 30s deadline. `ReleaseExecute` is
best-effort — reattach.py runs it on a pool and logs-and-ignores failures
("server is equipped to deal with abandoned executions", identical on v4.2.0
and master) — so a timed-out release frees its thread and closes its stream,
nothing accumulates. Verified in CI: with the hang test **not** deselected,
`test_parity_arrow_python_udf.py` passes 319/320 and all arrow files pass. The
keepalive change (which you reviewed) stays as an independent robustness fix;
all the earlier workarounds are removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
