Hyukjin Kwon created SPARK-61402:
------------------------------------
Summary: Pickle VariantVal, Geometry and Geography as PySpark's
plain classes in the Python client, so that the server's Python workers load
them
Key: SPARK-61402
URL: https://issues.apache.org/jira/browse/SPARK-61402
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
SPARK-61291 made the Python client's VariantVal pickle its state as a tuple
(value, metadata, __dict__) with a __setstate__ of its own. PySpark's
VariantVal is a plain class without __setstate__, whose pickle BUILD takes a
dict state only, so official PySpark cannot load a VariantVal pickled by the
Python client: "_pickle.UnpicklingError: state is not a dictionary". The
server's Python workers run official PySpark, so a UDF, pandas UDF, UDTF,
foreachBatch or other function that refers to a VariantVal fails on the server;
before SPARK-61291 it ran.
Geometry and Geography values of the Python client cannot be pickled, copied or
subclassed at all ("cannot pickle 'pyspark.sql.types.Geometry' object", "not an
acceptable base type"), so neither can a Row that holds one or a UDF that
refers to one. PySpark's are plain classes, which do all of these.
Fix: pickle the three classes as PySpark pickles its plain classes,
copyreg.__newobj__ with the dict of their attributes (and the slots of a
subclass of slots, as object.__reduce_ex__ does), take that state and PySpark's
pickles of them in __setstate__, and let Geometry and Geography be subclassed
as VariantVal can.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]