Tim Meehan created SPARK-61411:
----------------------------------

             Summary: Retry the errors that carry a RetryInfo, after the delay 
the server asks for, as PySpark does
                 Key: SPARK-61411
                 URL: https://issues.apache.org/jira/browse/SPARK-61411
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Tim Meehan


The Rust client does not retry an error whose status details carry a 
{{google.rpc.RetryInfo}}, and no retry of it waits for a delay that the server 
asks for. PySpark's {{DefaultPolicy}} retries any error that carries a 
{{RetryInfo}}, whatever its status code, and waits at least the {{retry_delay}} 
of it (at most 10 minutes) before the next attempt.

A server that throttles with {{RESOURCE_EXHAUSTED}} and a {{RetryInfo}} of 400 
ms is the case. Consider a loopback gRPC service that answers Config with that 
status once and then succeeds. The PySpark behavior below is read from its 
source (pyspark-client 4.3.0), not from a run; the Rust behavior is from a test 
of the service at spark-connect-rust master 125654b:
* pyspark-client 4.3.0 (read from {{pyspark/sql/connect/client/retries.py}}, 
{{DefaultPolicy.can_retry}} and {{RetryPolicyState.next_attempt}}): retries, 
and waits at least 400 ms.
* spark-connect-rust master 125654b, with a Rust test of the same service 
({{crates/spark-connect-core/tests/retry_info.rs}}, added by the fix): the 
error is raised at the first call; the server sees one call.

{code:rust}
// The service fails Config once with RESOURCE_EXHAUSTED and a RetryInfo of 400 
ms.
let result = client.config(request).await; // Err at 125654b; Ok after >= 400 
ms with the fix
{code}

The same holds for every call that goes through the retry policy (unary calls, 
AddArtifacts, ReleaseExecute, and the reattach of an execution). I did not run 
a stock OSS Spark 4.3.0 Connect server for it: the server does not send a 
{{RetryInfo}} on demand.

>From source, at 125654b:
* {{RetryPolicy::can_retry}} (crates/spark-connect-core/src/retries.rs:76) 
retries only on the status code (UNAVAILABLE, a cursor disconnect, and two 
transport errors); it does not look at the details.
* Every caller passes no server delay: {{state.next_attempt(None)}} at 
crates/spark-connect-core/src/client.rs:1151, 1908, 2101 and 2283. 
{{RetryPolicyState::next_attempt}} (retries.rs:135) takes the delay and clamps 
it to {{max_server_retry_delay_ms}}, but nothing supplies it.
* {{RetryPolicy::default()}} has {{recognize_server_retry_delay: false}} and no 
{{max_server_retry_delay_ms}} (retries.rs:45), where PySpark's 
{{DefaultPolicy}} has {{recognize_server_retry_delay=True}} and 
{{max_server_retry_delay}} of 10 minutes.
* A stream that fails after its headers with a retryable error is reattached at 
once, with no wait and no retry counted, where PySpark raises the error inside 
its {{Retrying}}, which waits and counts it (client.rs:2229-2236 at 125654b).

Known difference, not part of this issue: the other defaults of 
{{RetryPolicy::default()}} (initial backoff, multiplier, maximum backoff, 
jitter) differ from PySpark's {{DefaultPolicy}}.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to