[
https://issues.apache.org/jira/browse/SPARK-61129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61129:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Make SparkSession and DataFrame picklable, of the active session, in the
> Python client as PySpark does
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61129
> URL: https://issues.apache.org/jira/browse/SPARK-61129
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect's SparkSession and DataFrame can be pickled with cloudpickle.
> - A session pickles as a function of its ID, which gives back the active
> session of that ID (SparkSession._getActiveSessionIfMatches). Where no
> session is active, such as on a Python worker of the server, it raises
> PySparkRuntimeError NO_ACTIVE_SESSION. Where another session is active, it
> raises PySparkAssertionError.
> - A DataFrame pickles of its plan and its session.
> So a UDF or UDTF that captures the session or a DataFrame is sent to the
> server. If it is called there, it raises PythonException NO_ACTIVE_SESSION,
> as Apache Spark's test_udtf_access_spark_session_connect checks. In the same
> process, a pickled DataFrame unpickles as a DataFrame of the same plan.
> The Python client of spark-connect-rust cannot pickle either: "TypeError:
> cannot pickle 'pyspark.sql.session.SparkSession' object", and the same for
> its DataFrame. Such a UDF or UDTF fails on the client, and
> test_udtf_access_spark_session_connect fails. The client also lacks
> SparkSession._getActiveSessionIfMatches.
> The client's SparkSession and DataFrame should pickle as PySpark's do.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
