Hyukjin Kwon created SPARK-61965:
------------------------------------
Summary: Convert the name and the return type before
spark.client.register_udtf pickles the handler, as PySpark's register_udtf does
Key: SPARK-61965
URL: https://issues.apache.org/jira/browse/SPARK-61965
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Problem
PySpark's SparkConnectClient.register_udtf (core.py:1117-1149) makes a
PythonUDTF, which reads the
__name__ of the handler class (plan.py:3062), and then its udtf_plan sets the
name in the string
field of its command (plan.py:3138) and converts the return type with
pyspark_types_to_proto_types
(PythonUDTF.to_plan, plan.py:3075-3076) before it pickles the handler
(plan.py:3078-3088). The
drop-in's spark.client.register_udtf pickled the handler before
_registerPythonUdtf converted the
name, so of a handler that does not pickle it raised UDTF_SERIALIZATION_ERROR
where PySpark raises
the error of the name or of the return type. spark.udtf.register does this in
PySpark's order since
SPARK-61951.
Repro (a handler class that closes over a threading.Lock, so that cloudpickle
cannot pickle it)
client = spark.client
client.register_udtf(Unpicklable, "x int", 5)
client.register_udtf(Unpicklable, "x int", "\udc80")
client.register_udtf(Unpicklable, StructField("x", IntegerType()), "t")
client.register_udtf(Unpicklable, "x \udc80", "t")
client.register_udtf(Unpicklable, 5, 5)
client.register_udtf(object(), 5, 5)
Actual (drop-in): UDTF_SERIALIZATION_ERROR for the first four, and
UNSUPPORTED_OPERATION of
"data type 5" for the last two, as its own check of the return type came first.
Expected (pyspark-client 4.3.0): TypeError: bad argument type for built-in
operation;
UnicodeEncodeError; PySparkValueError UNSUPPORTED_OPERATION of "data type
StructField('x',
IntegerType(), True)"; UnicodeEncodeError; the TypeError of the name;
AttributeError: 'object' object
has no attribute '__name__'. A valid name and return type raise
UDTF_SERIALIZATION_ERROR on both.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]