Hyukjin Kwon created SPARK-62123:
------------------------------------
Summary: Make the connections, retries and errors of the Rust
transport grpcio's, of its codes, texts and limits, in the Python client
Key: SPARK-62123
URL: https://issues.apache.org/jira/browse/SPARK-62123
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The Rust transport of the Python client (pyspark-client-rust) differs from
grpcio, which pyspark-client 4.3.0 uses, in how it connects, retries and
reports failures, as dev/parity/known-differences.md lists:
- A refused connection or a failed DNS lookup gives tonic's messages ("tcp
connect error"), where grpcio gives gRPC core's ("failed to connect to all
addresses; last error: ..."); the transport does not race the addresses of a
host, fail fast, or back off as gRPC's channel does; and each session opens a
connection of its own.
- SparkConnectClient.set_retry_policies takes only DefaultPolicy parameters: a
policy of its own can_retry or to_state raises NOT_IMPLEMENTED, where PySpark's
Retrying calls them; requests sent in the background do not warn
RETRIES_EXCEEDED.
- The errors that PySpark leaves to grpc (clone, copy_from_local_to_fs, the
caching of artifacts) are converted.
- A response that does not decode, a message over 128 MiB, response metadata
past gRPC's limits, and a stream that the server resets get tonic's codes and
texts, where grpcio gives its own (Exception deserializing response!,
RESOURCE_EXHAUSTED "Received message larger than max", "received metadata size
exceeds hard limit", "Stream removed (RST_STREAM ...)"); response metadata of
hundreds of KB hangs the client.
- The trust store of TLS is not grpcio's, and a child of os.fork() that uses or
frees an inherited session can hang its parent.
Expected: as grpcio and pyspark-client, the connections, retries, logs and
errors above give grpcio's codes, texts and behavior.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]