Hyukjin Kwon created SPARK-62037:
------------------------------------
Summary: Share the options and hooks of a builder's shallow
copies, hold PySpark's session and builder locks, let SparkSession, Catalog and
DataSourceRegistration be subclassed, and load PySpark's pickles of readers
Key: SPARK-62037
URL: https://issues.apache.org/jira/browse/SPARK-62037
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
>From a review of the documented differences of the session against PySpark
>4.3.0:
- copy.copy of the builder of pyspark.sql.connect.session.SparkSession has its
own options and hook factories, where PySpark's plain object shares its
_options dict and _hook_factories list with the copy, so a config or a hook set
on the copy is seen by the original.
- The builder makes and stops sessions under a Rust lock, where PySpark holds
the class attribute SparkSession._lock (an RLock) for getOrCreate, around the
default session of create and for stop, and the builder's own Builder._lock for
config, remote, appName, channelBuilder and _registerHook, so a thread that
holds SparkSession._lock does not hold off the drop-in; and getOrCreate of a
new session sends the Config of its options once, where PySpark's create and
getOrCreate both apply them.
- SparkSession, Catalog and DataSourceRegistration cannot be subclassed, where
PySpark's plain classes can.
- PySpark's pickle of a DataFrameReader or a DataStreamReader does not load in
the drop-in (TypeError: missing 'client').
- The drop-in's LogicalPlan has no to_proto, which plugins and tools call.
- The known differences of the default port, of a missing URL, of the rendering
of deep plans and of the builder copies are stale, and they lack that PySpark
4.3.0 hangs at exit for a session that is not the default and is not stopped.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]