[ 
https://issues.apache.org/jira/browse/SPARK-62114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62114:
-----------------------------------
    Labels: pull-request-available  (was: )

> Make the attributes and thread-local state of SparkConnectClient, and the 
> lifetime of a session, PySpark's, of plain attributes that the requests read 
> when they are built, in the Python client
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62114
>                 URL: https://issues.apache.org/jira/browse/SPARK-62114
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> The Python client (pyspark-client-rust) differs from pyspark-client 4.3.0 in 
> the attributes and thread-local state of its SparkConnectClient and in the 
> lifetime of a session, as dev/parity/known-differences.md lists:
> - SparkConnectClient.thread_local is a subclass of threading.local whose tags 
> and user context extensions are subclasses of set and list, kept in step with 
> the Rust transport, and global_user_context_extensions is a subclass of list; 
> PySpark's are a plain threading.local, set and lists, which its requests read 
> when they are built. A copy.copy of a client does not send its own attributes.
> - _closed and _user_id are not PySpark's plain attributes: _user_id is 
> read-only, and setting _closed to False sends requests again after close() or 
> stop().
> - A conf or a client kept apart from its session keeps the session open, 
> where PySpark's hold the client, so the session closes once the program drops 
> it.
> - channelBuilder(...) rejects a channel builder without a host, where PySpark 
> takes any object.
> - A call that runs during another call of the same thread (a progress 
> handler, a finalizer) changes the request context of the outer call; the 
> interrupt of a Ctrl-C and the error details of a failed CloneSession carry 
> other extensions than PySpark's, and a failed CloneSession sends extra 
> requests that PySpark's clone does not.
> Expected: as pyspark-client, the client's attributes, its thread_local and 
> its lists are plain Python objects that the requests read when they are 
> built, of each copy of a client its own; a kept conf or client does not keep 
> the session open; and the requests of nested calls, interrupts and clones 
> carry what PySpark's carry.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to