Hyukjin Kwon created SPARK-62114:
------------------------------------
Summary: Make the attributes and thread-local state of
SparkConnectClient, and the lifetime of a session, PySpark's, of plain
attributes that the requests read when they are built, in the Python client
Key: SPARK-62114
URL: https://issues.apache.org/jira/browse/SPARK-62114
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The Python client (pyspark-client-rust) differs from pyspark-client 4.3.0 in
the attributes and thread-local state of its SparkConnectClient and in the
lifetime of a session, as dev/parity/known-differences.md lists:
- SparkConnectClient.thread_local is a subclass of threading.local whose tags
and user context extensions are subclasses of set and list, kept in step with
the Rust transport, and global_user_context_extensions is a subclass of list;
PySpark's are a plain threading.local, set and lists, which its requests read
when they are built. A copy.copy of a client does not send its own attributes.
- _closed and _user_id are not PySpark's plain attributes: _user_id is
read-only, and setting _closed to False sends requests again after close() or
stop().
- A conf or a client kept apart from its session keeps the session open, where
PySpark's hold the client, so the session closes once the program drops it.
- channelBuilder(...) rejects a channel builder without a host, where PySpark
takes any object.
- A call that runs during another call of the same thread (a progress handler,
a finalizer) changes the request context of the outer call; the interrupt of a
Ctrl-C and the error details of a failed CloneSession carry other extensions
than PySpark's, and a failed CloneSession sends extra requests that PySpark's
clone does not.
Expected: as pyspark-client, the client's attributes, its thread_local and its
lists are plain Python objects that the requests read when they are built, of
each copy of a client its own; a kept conf or client does not keep the session
open; and the requests of nested calls, interrupts and clones carry what
PySpark's carry.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]