HyukjinKwon opened a new pull request, #255: URL: https://github.com/apache/spark-connect-rust/pull/255
### What changes were proposed in this pull request? The per-RPC deadlines of PySpark Connect, which SPARK-60997 applied with tonic's `grpc-timeout`, now behave as PySpark's when they pass before the response headers (`crates/spark-connect-core/src/client.rs`): - `within_deadline` bounds each RPC as a whole by its deadline, and reads tonic's `CANCELLED` "Timeout expired", which tonic gives when its `grpc-timeout` passes before the headers, as the `DEADLINE_EXCEEDED` it is. A unary call stalled after its headers is now bounded as well. - A reattachable execution whose ExecutePlan or ReattachExecute passes its deadline reattaches, for the policy's bound of such retries from the first one, as the reference's `RetryException` does. - A reattach that finds `INVALID_HANDLE.OPERATION_NOT_FOUND` before any response, from the call or from its stream, sends the ExecutePlan again, as the reference iterator does. ### Why are the changes needed? [SPARK-61064](https://issues.apache.org/jira/browse/SPARK-61064): with a server that stalls before sending headers (an in-process mock that compares the two clients): | Scenario | Current master | This change and PySpark 4.3.0 | | --- | --- | --- | | First ExecutePlan of an execution stalls past its deadline | fails with `[The operation was cancelled] Timeout expired` after 10 minutes, no reattach | reattaches, executes again on OPERATION_NOT_FOUND, returns the rows | | ReattachExecute stalls past its deadline | fails the same way | reattaches again and returns the rows | | Config stalls past its deadline | `CANCELLED` | `DEADLINE_EXCEEDED`, with PySpark's message about the per-RPC timeouts | The existing tests stall response bodies, never headers, which the client already handled. ### Does this PR introduce _any_ user-facing change? Yes, in these failure cases, as in PySpark. ### How was this patch tested? - `crates/spark-connect-core/tests/deadlines.rs`, 8 tests against a loopback service that stalls before its headers: the three scenarios above, the bound of the retries past deadlines for the first call and for reattaches, and OPERATION_NOT_FOUND from the call, from the stream, and after a response (which still fails). Seven of them fail without the change. - The core tests (lib 119, reattach cleanup 11, user context 4, all passed) and the build of every integration test of `apache-spark-connect`. - The stalled-ExecutePlan scenario end to end through the Python drop-in: the rows are returned after the reattach and the second ExecutePlan. Unchanged and different from PySpark: PySpark also executes again on `INVALID_HANDLE.SESSION_NOT_FOUND`, and raises `RESPONSE_ALREADY_RECEIVED` for OPERATION_NOT_FOUND after a response. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
