Hyukjin Kwon created SPARK-62123:
------------------------------------

             Summary: Make the connections, retries and errors of the Rust 
transport grpcio's, of its codes, texts and limits, in the Python client
                 Key: SPARK-62123
                 URL: https://issues.apache.org/jira/browse/SPARK-62123
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The Rust transport of the Python client (pyspark-client-rust) differs from 
grpcio, which pyspark-client 4.3.0 uses, in how it connects, retries and 
reports failures, as dev/parity/known-differences.md lists:

- A refused connection or a failed DNS lookup gives tonic's messages ("tcp 
connect error"), where grpcio gives gRPC core's ("failed to connect to all 
addresses; last error: ..."); the transport does not race the addresses of a 
host, fail fast, or back off as gRPC's channel does; and each session opens a 
connection of its own.
- SparkConnectClient.set_retry_policies takes only DefaultPolicy parameters: a 
policy of its own can_retry or to_state raises NOT_IMPLEMENTED, where PySpark's 
Retrying calls them; requests sent in the background do not warn 
RETRIES_EXCEEDED.
- The errors that PySpark leaves to grpc (clone, copy_from_local_to_fs, the 
caching of artifacts) are converted.
- A response that does not decode, a message over 128 MiB, response metadata 
past gRPC's limits, and a stream that the server resets get tonic's codes and 
texts, where grpcio gives its own (Exception deserializing response!, 
RESOURCE_EXHAUSTED "Received message larger than max", "received metadata size 
exceeds hard limit", "Stream removed (RST_STREAM ...)"); response metadata of 
hundreds of KB hangs the client.
- The trust store of TLS is not grpcio's, and a child of os.fork() that uses or 
frees an inherited session can hang its parent.

Expected: as grpcio and pyspark-client, the connections, retries, logs and 
errors above give grpcio's codes, texts and behavior.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to