[ 
https://issues.apache.org/jira/browse/SPARK-61415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61415:
-----------------------------------
    Labels: pull-request-available  (was: )

> Back off and bound the reattaches of a result stream that keeps failing, and 
> raise NO_ACTIVE_SESSION when the plan compression lookup closed the client, 
> in Spark Connect Rust as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61415
>                 URL: https://issues.apache.org/jira/browse/SPARK-61415
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Three differences of Spark Connect Rust's transport from PySpark 4.3.0:
> 1. ReattachableResponseStream::message has no backoff and no retry limit for 
> a stream that fails after it opens. When a reattach gets its response headers 
> and the stream then fails before the next response, the client reattaches 
> again at once, without waiting or counting. PySpark's 
> ExecutePlanResponseReattachableIterator._has_next runs the reattach and the 
> read of its stream in one attempt of its Retrying, so each such failure waits 
> the policy's backoff and is raised after max_retries. Through a proxy that 
> resets every connection after 100 KB, a collect of a 300 KB row reattached 
> 197 times in 1.3 s; PySpark reattached 4 times over 4.7 s.
> 2. When the plan compression lookup of the first plan of a client closes the 
> client (the server restarted, so the server-side session ID changed), the 
> client still sends the ExecutePlan or AnalyzePlan. PySpark raises 
> NO_ACTIVE_SESSION and sends nothing.
> 3. The FetchErrorDetails and stack-trace Config requests sent while raising a 
> server error carry no user context extensions (PySpark's carry them), and the 
> AddArtifacts and ArtifactStatus requests carry client_type, which PySpark's 
> ArtifactManager does not send. docs/compatibility.md also gives the wrong old 
> codes of GOAWAY failures.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to