[
https://issues.apache.org/jira/browse/SPARK-61167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61167:
------------------------------------
Assignee: Hyukjin Kwon
> Pickle Columns as their expressions, also for the Python workers of the
> server, in the Python client as PySpark does
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61167
> URL: https://issues.apache.org/jira/browse/SPARK-61167
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect's Columns are Python objects of their expressions, which
> pickle and deepcopy, also with cloudpickle. So a foreachBatch function, which
> runs on a Python worker of the server, can capture a Column defined outside
> of it, and the worker unpickles it as PySpark Connect's.
> The Python client's Column is a class of the Rust extension, which cannot be
> pickled:
> PySparkPicklingError: [STREAMING_CONNECT_SERIALIZATION_ERROR] Cannot
> serialize the function `foreachBatch`. ...
> and pickle.dumps(column) and copy.deepcopy(column) raise TypeError. Apache
> Spark's test_streaming_foreach_batch_external_column of
> test_parity_foreach_batch.py fails on this client.
> A Column should pickle as its spark.connect.Expression and its rendering: on
> the client, it unpickles as a Column that sends the expression as it is; on a
> worker without the Rust client, as PySpark Connect's Column of that
> expression.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
