[ 
https://issues.apache.org/jira/browse/SPARK-61975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61975.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 843
[https://github.com/apache/spark-connect-rust/pull/843]

> Emit PySpark's RETRIES_EXCEEDED warning, log stop's release failure through 
> the connect logger, render gRPC failures as grpcio does, and send PySpark's 
> requests for the first command and the registrations
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61975
>                 URL: https://issues.apache.org/jira/browse/SPARK-61975
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> A follow-up of SPARK-61915, from a round-3 review of the transport against 
> pyspark-client 4.3.0 (grpcio 1.85.1):
> 1. PySpark's [RETRIES_EXCEEDED] UserWarning 
> (connect/client/retries.py:280-287, 309-312) was never emitted, as the core's 
> retry loop did not tell a failure that is not retryable from retries that ran 
> out. With the server down and DefaultPolicy(max_retries=1), PySpark warns on 
> count, conf.get, schema, interruptAll, sql, version and stop, and 
> simplefilter("error") raises the warning instead of the failure. The drop-in 
> warned on none.
> 2. stop() logged a release failure to logging.getLogger("pyspark"), which 
> reaches stderr through logging's lastResort. PySpark logs it through the 
> connect logger (connect/session.py:949-953), which is silent unless 
> SPARK_CONNECT_LOG_LEVEL is set (connect/logging.py:25-50).
> 3. The text of a gRPC failure without status details was tonic's ("[The 
> service is currently unavailable] tcp connect error"). PySpark's str(e) is 
> grpcio's "<_InactiveRpcError of RPC that terminated with: status = 
> StatusCode.X ...>" (core.py:2408-2418, 2456-2459).
> 4. The first command ExecutePlan of a session carried 
> client_observed_server_side_session_id, as the drop-in looked the plan 
> compression settings up before it built the request. PySpark builds the 
> request first (core.py:1416, 1518-1527, 2714-2736). The udf, udtf, 
> data-source and Java-function registrations went through that compression 
> too, with its Config lookup, where PySpark sends them as they are 
> (core.py:1111-1185). So an 11 MB UDF registration was compressed, and the 
> first registration of a session sent a Config request that PySpark does not.
> 5. The stack-trace conf read while raising a server error was tried once, 
> where PySpark's config() retries it (core.py:2071-2080).
> 6. The docs said that the SPARK_CONNECT_USER_AGENT environment variable is 
> read "once", where newSession and cloneSession keep the parent's user agent. 
> They did not say that copyFromLocalToFs and cloneSession convert errors that 
> PySpark leaves as grpc.RpcError.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to