[
https://issues.apache.org/jira/browse/SPARK-61975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61975.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 843
[https://github.com/apache/spark-connect-rust/pull/843]
> Emit PySpark's RETRIES_EXCEEDED warning, log stop's release failure through
> the connect logger, render gRPC failures as grpcio does, and send PySpark's
> requests for the first command and the registrations
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61975
> URL: https://issues.apache.org/jira/browse/SPARK-61975
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> A follow-up of SPARK-61915, from a round-3 review of the transport against
> pyspark-client 4.3.0 (grpcio 1.85.1):
> 1. PySpark's [RETRIES_EXCEEDED] UserWarning
> (connect/client/retries.py:280-287, 309-312) was never emitted, as the core's
> retry loop did not tell a failure that is not retryable from retries that ran
> out. With the server down and DefaultPolicy(max_retries=1), PySpark warns on
> count, conf.get, schema, interruptAll, sql, version and stop, and
> simplefilter("error") raises the warning instead of the failure. The drop-in
> warned on none.
> 2. stop() logged a release failure to logging.getLogger("pyspark"), which
> reaches stderr through logging's lastResort. PySpark logs it through the
> connect logger (connect/session.py:949-953), which is silent unless
> SPARK_CONNECT_LOG_LEVEL is set (connect/logging.py:25-50).
> 3. The text of a gRPC failure without status details was tonic's ("[The
> service is currently unavailable] tcp connect error"). PySpark's str(e) is
> grpcio's "<_InactiveRpcError of RPC that terminated with: status =
> StatusCode.X ...>" (core.py:2408-2418, 2456-2459).
> 4. The first command ExecutePlan of a session carried
> client_observed_server_side_session_id, as the drop-in looked the plan
> compression settings up before it built the request. PySpark builds the
> request first (core.py:1416, 1518-1527, 2714-2736). The udf, udtf,
> data-source and Java-function registrations went through that compression
> too, with its Config lookup, where PySpark sends them as they are
> (core.py:1111-1185). So an 11 MB UDF registration was compressed, and the
> first registration of a session sent a Config request that PySpark does not.
> 5. The stack-trace conf read while raising a server error was tried once,
> where PySpark's config() retries it (core.py:2071-2080).
> 6. The docs said that the SPARK_CONNECT_USER_AGENT environment variable is
> read "once", where newSession and cloneSession keep the parent's user agent.
> They did not say that copyFromLocalToFs and cloneSession convert errors that
> PySpark leaves as grpc.RpcError.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]