[
https://issues.apache.org/jira/browse/SPARK-62123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-62123.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 905
[https://github.com/apache/spark-connect-rust/pull/905]
> Make the connections, retries and errors of the Rust transport grpcio's, of
> its codes, texts and limits, in the Python client
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62123
> URL: https://issues.apache.org/jira/browse/SPARK-62123
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> The Rust transport of the Python client (pyspark-client-rust) differs from
> grpcio, which pyspark-client 4.3.0 uses, in how it connects, retries and
> reports failures, as dev/parity/known-differences.md lists:
> - A refused connection or a failed DNS lookup gives tonic's messages ("tcp
> connect error"), where grpcio gives gRPC core's ("failed to connect to all
> addresses; last error: ..."); the transport does not race the addresses of a
> host, fail fast, or back off as gRPC's channel does; and each session opens a
> connection of its own.
> - SparkConnectClient.set_retry_policies takes only DefaultPolicy parameters:
> a policy of its own can_retry or to_state raises NOT_IMPLEMENTED, where
> PySpark's Retrying calls them; requests sent in the background do not warn
> RETRIES_EXCEEDED.
> - The errors that PySpark leaves to grpc (clone, copy_from_local_to_fs, the
> caching of artifacts) are converted.
> - A response that does not decode, a message over 128 MiB, response metadata
> past gRPC's limits, and a stream that the server resets get tonic's codes and
> texts, where grpcio gives its own (Exception deserializing response!,
> RESOURCE_EXHAUSTED "Received message larger than max", "received metadata
> size exceeds hard limit", "Stream removed (RST_STREAM ...)"); response
> metadata of hundreds of KB hangs the client.
> - The trust store of TLS is not grpcio's, and a child of os.fork() that uses
> or frees an inherited session can hang its parent.
> Expected: as grpcio and pyspark-client, the connections, retries, logs and
> errors above give grpcio's codes, texts and behavior.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]