[ 
https://issues.apache.org/jira/browse/SPARK-62123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62123:
-----------------------------------
    Labels: pull-request-available  (was: )

> Make the connections, retries and errors of the Rust transport grpcio's, of 
> its codes, texts and limits, in the Python client
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62123
>                 URL: https://issues.apache.org/jira/browse/SPARK-62123
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> The Rust transport of the Python client (pyspark-client-rust) differs from 
> grpcio, which pyspark-client 4.3.0 uses, in how it connects, retries and 
> reports failures, as dev/parity/known-differences.md lists:
> - A refused connection or a failed DNS lookup gives tonic's messages ("tcp 
> connect error"), where grpcio gives gRPC core's ("failed to connect to all 
> addresses; last error: ..."); the transport does not race the addresses of a 
> host, fail fast, or back off as gRPC's channel does; and each session opens a 
> connection of its own.
> - SparkConnectClient.set_retry_policies takes only DefaultPolicy parameters: 
> a policy of its own can_retry or to_state raises NOT_IMPLEMENTED, where 
> PySpark's Retrying calls them; requests sent in the background do not warn 
> RETRIES_EXCEEDED.
> - The errors that PySpark leaves to grpc (clone, copy_from_local_to_fs, the 
> caching of artifacts) are converted.
> - A response that does not decode, a message over 128 MiB, response metadata 
> past gRPC's limits, and a stream that the server resets get tonic's codes and 
> texts, where grpcio gives its own (Exception deserializing response!, 
> RESOURCE_EXHAUSTED "Received message larger than max", "received metadata 
> size exceeds hard limit", "Stream removed (RST_STREAM ...)"); response 
> metadata of hundreds of KB hangs the client.
> - The trust store of TLS is not grpcio's, and a child of os.fork() that uses 
> or frees an inherited session can hang its parent.
> Expected: as grpcio and pyspark-client, the connections, retries, logs and 
> errors above give grpcio's codes, texts and behavior.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to