[
https://issues.apache.org/jira/browse/SPARK-61411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18143024#comment-18143024
]
Tim Meehan commented on SPARK-61411:
------------------------------------
SPARK-61390 (ae4dad6) and SPARK-61415 (74ee810), merged after this was filed,
fix it: at spark-connect-rust master c8ae7ef, RetryPolicy::can_retry retries
any status with a google.rpc.RetryInfo detail whatever its code,
RetryPolicy::default() recognizes the server's delay capped at 10 minutes, and
every call of RetryPolicyState::next_attempt in
crates/spark-connect-core/src/client.rs passes retry_delay_ms(&status),
including the reattach after a stream that fails mid-execution. Closing as a
duplicate of SPARK-61390.
> Retry the errors that carry a RetryInfo, after the delay the server asks for,
> as PySpark does
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-61411
> URL: https://issues.apache.org/jira/browse/SPARK-61411
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Tim Meehan
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The Rust client does not retry an error whose status details carry a
> {{google.rpc.RetryInfo}}, and no retry of it waits for a delay that the
> server asks for. PySpark's {{DefaultPolicy}} retries any error that carries a
> {{RetryInfo}}, whatever its status code, and waits at least the
> {{retry_delay}} of it (at most 10 minutes) before the next attempt.
> A server that throttles with {{RESOURCE_EXHAUSTED}} and a {{RetryInfo}} of
> 400 ms is the case. Consider a loopback gRPC service that answers Config with
> that status once and then succeeds. The PySpark behavior below is read from
> its source (pyspark-client 4.3.0), not from a run; the Rust behavior is from
> a test of the service at spark-connect-rust master 125654b:
> * pyspark-client 4.3.0 (read from {{pyspark/sql/connect/client/retries.py}},
> {{DefaultPolicy.can_retry}} and {{RetryPolicyState.next_attempt}}): retries,
> and waits at least 400 ms.
> * spark-connect-rust master 125654b, with a Rust test of the same service
> ({{crates/spark-connect-core/tests/retry_info.rs}}, added by the fix): the
> error is raised at the first call; the server sees one call.
> {code:rust}
> // The service fails Config once with RESOURCE_EXHAUSTED and a RetryInfo of
> 400 ms.
> let result = client.config(request).await; // Err at 125654b; Ok after >= 400
> ms with the fix
> {code}
> The same holds for every call that goes through the retry policy (unary
> calls, AddArtifacts, ReleaseExecute, and the reattach of an execution). I did
> not run a stock OSS Spark 4.3.0 Connect server for it: the server does not
> send a {{RetryInfo}} on demand.
> From source, at 125654b:
> * {{RetryPolicy::can_retry}} (crates/spark-connect-core/src/retries.rs:76)
> retries only on the status code (UNAVAILABLE, a cursor disconnect, and two
> transport errors); it does not look at the details.
> * Every caller passes no server delay: {{state.next_attempt(None)}} at
> crates/spark-connect-core/src/client.rs:1151, 1908, 2101 and 2283.
> {{RetryPolicyState::next_attempt}} (retries.rs:135) takes the delay and
> clamps it to {{max_server_retry_delay_ms}}, but nothing supplies it.
> * {{RetryPolicy::default()}} has {{recognize_server_retry_delay: false}} and
> no {{max_server_retry_delay_ms}} (retries.rs:45), where PySpark's
> {{DefaultPolicy}} has {{recognize_server_retry_delay=True}} and
> {{max_server_retry_delay}} of 10 minutes.
> * A stream that fails after its headers with a retryable error is reattached
> at once, with no wait and no retry counted, where PySpark raises the error
> inside its {{Retrying}}, which waits and counts it (client.rs:2229-2236 at
> 125654b).
> Known difference, not part of this issue: the other defaults of
> {{RetryPolicy::default()}} (initial backoff, multiplier, maximum backoff,
> jitter) differ from PySpark's {{DefaultPolicy}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]