Hyukjin Kwon created SPARK-61167:
------------------------------------
Summary: Pickle Columns as their expressions, also for the Python
workers of the server, in the Python client as PySpark does
Key: SPARK-61167
URL: https://issues.apache.org/jira/browse/SPARK-61167
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect's Columns are Python objects of their expressions, which pickle
and deepcopy, also with cloudpickle. So a foreachBatch function, which runs on
a Python worker of the server, can capture a Column defined outside of it, and
the worker unpickles it as PySpark Connect's.
The Python client's Column is a class of the Rust extension, which cannot be
pickled:
PySparkPicklingError: [STREAMING_CONNECT_SERIALIZATION_ERROR] Cannot serialize
the function `foreachBatch`. ...
and pickle.dumps(column) and copy.deepcopy(column) raise TypeError. Apache
Spark's test_streaming_foreach_batch_external_column of
test_parity_foreach_batch.py fails on this client.
A Column should pickle as its spark.connect.Expression and its rendering: on
the client, it unpickles as a Column that sends the expression as it is; on a
worker without the Rust client, as PySpark Connect's Column of that expression.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
