[ 
https://issues.apache.org/jira/browse/SPARK-62037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-62037.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 867
[https://github.com/apache/spark-connect-rust/pull/867]

> Share the options and hooks of a builder's shallow copies, hold PySpark's 
> session and builder locks, let SparkSession, Catalog and 
> DataSourceRegistration be subclassed, and load PySpark's pickles of readers
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62037
>                 URL: https://issues.apache.org/jira/browse/SPARK-62037
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> From a review of the documented differences of the session against PySpark 
> 4.3.0:
> - copy.copy of the builder of pyspark.sql.connect.session.SparkSession has 
> its own options and hook factories, where PySpark's plain object shares its 
> _options dict and _hook_factories list with the copy, so a config or a hook 
> set on the copy is seen by the original.
> - The builder makes and stops sessions under a Rust lock, where PySpark holds 
> the class attribute SparkSession._lock (an RLock) for getOrCreate, around the 
> default session of create and for stop, and the builder's own Builder._lock 
> for config, remote, appName, channelBuilder and _registerHook, so a thread 
> that holds SparkSession._lock does not hold off the drop-in; and getOrCreate 
> of a new session sends the Config of its options once, where PySpark's create 
> and getOrCreate both apply them.
> - SparkSession, Catalog and DataSourceRegistration cannot be subclassed, 
> where PySpark's plain classes can.
> - PySpark's pickle of a DataFrameReader or a DataStreamReader does not load 
> in the drop-in (TypeError: missing 'client').
> - The drop-in's LogicalPlan has no to_proto, which plugins and tools call.
> - The known differences of the default port, of a missing URL, of the 
> rendering of deep plans and of the builder copies are stale, and they lack 
> that PySpark 4.3.0 hangs at exit for a session that is not the default and is 
> not stopped.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to