Hyukjin Kwon created SPARK-62114:
------------------------------------

             Summary: Make the attributes and thread-local state of 
SparkConnectClient, and the lifetime of a session, PySpark's, of plain 
attributes that the requests read when they are built, in the Python client
                 Key: SPARK-62114
                 URL: https://issues.apache.org/jira/browse/SPARK-62114
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The Python client (pyspark-client-rust) differs from pyspark-client 4.3.0 in 
the attributes and thread-local state of its SparkConnectClient and in the 
lifetime of a session, as dev/parity/known-differences.md lists:

- SparkConnectClient.thread_local is a subclass of threading.local whose tags 
and user context extensions are subclasses of set and list, kept in step with 
the Rust transport, and global_user_context_extensions is a subclass of list; 
PySpark's are a plain threading.local, set and lists, which its requests read 
when they are built. A copy.copy of a client does not send its own attributes.
- _closed and _user_id are not PySpark's plain attributes: _user_id is 
read-only, and setting _closed to False sends requests again after close() or 
stop().
- A conf or a client kept apart from its session keeps the session open, where 
PySpark's hold the client, so the session closes once the program drops it.
- channelBuilder(...) rejects a channel builder without a host, where PySpark 
takes any object.
- A call that runs during another call of the same thread (a progress handler, 
a finalizer) changes the request context of the outer call; the interrupt of a 
Ctrl-C and the error details of a failed CloneSession carry other extensions 
than PySpark's, and a failed CloneSession sends extra requests that PySpark's 
clone does not.

Expected: as pyspark-client, the client's attributes, its thread_local and its 
lists are plain Python objects that the requests read when they are built, of 
each copy of a client its own; a kept conf or client does not keep the session 
open; and the requests of nested calls, interrupts and clones carry what 
PySpark's carry.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to