[
https://issues.apache.org/jira/browse/SPARK-62114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-62114.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 898
[https://github.com/apache/spark-connect-rust/pull/898]
> Make the attributes and thread-local state of SparkConnectClient, and the
> lifetime of a session, PySpark's, of plain attributes that the requests read
> when they are built, in the Python client
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62114
> URL: https://issues.apache.org/jira/browse/SPARK-62114
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> The Python client (pyspark-client-rust) differs from pyspark-client 4.3.0 in
> the attributes and thread-local state of its SparkConnectClient and in the
> lifetime of a session, as dev/parity/known-differences.md lists:
> - SparkConnectClient.thread_local is a subclass of threading.local whose tags
> and user context extensions are subclasses of set and list, kept in step with
> the Rust transport, and global_user_context_extensions is a subclass of list;
> PySpark's are a plain threading.local, set and lists, which its requests read
> when they are built. A copy.copy of a client does not send its own attributes.
> - _closed and _user_id are not PySpark's plain attributes: _user_id is
> read-only, and setting _closed to False sends requests again after close() or
> stop().
> - A conf or a client kept apart from its session keeps the session open,
> where PySpark's hold the client, so the session closes once the program drops
> it.
> - channelBuilder(...) rejects a channel builder without a host, where PySpark
> takes any object.
> - A call that runs during another call of the same thread (a progress
> handler, a finalizer) changes the request context of the outer call; the
> interrupt of a Ctrl-C and the error details of a failed CloneSession carry
> other extensions than PySpark's, and a failed CloneSession sends extra
> requests that PySpark's clone does not.
> Expected: as pyspark-client, the client's attributes, its thread_local and
> its lists are plain Python objects that the requests read when they are
> built, of each copy of a client its own; a kept conf or client does not keep
> the session open; and the requests of nested calls, interrupts and clones
> carry what PySpark's carry.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]