Hyukjin Kwon created SPARK-61995:
------------------------------------
Summary: Convert the return type of a UDTF each time its plan is
built, as PySpark's PythonUDTF.to_plan
Key: SPARK-61995
URL: https://issues.apache.org/jira/browse/SPARK-61995
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
A follow-up of SPARK-61951 and SPARK-61962. PySpark's PythonUDTF.to_plan
converts the return type of a UDTF with pyspark_types_to_proto_types each time
it builds a plan of a call, of the type object that the call kept, and pickles
the handler class then. The drop-in converted a valid return type at the call,
and only checked it at the build. So a StructType changed in place after the
call, as by add(), is sent as it was at the call: after rt = StructType([a
long]); d = udtf(H, returnType=rt)(lit(1)); rt.add("b", LongType()), PySpark's
d.schema is struct<a:bigint,b:bigint> and its collect gives Row(a=1, b=2),
where the drop-in's schema had only a and its collect failed with a
PythonException; a lateralJoin of the call had the columns ['id', 'a'] where
PySpark's has ['id', 'a', 'b']. Also, a str assigned to the UDTF's returnType
after udtf() raised UNSUPPORTED_OPERATION where PySpark parses it, and
UDTF_SERIALIZATION_ERROR named a handler class renamed after the call by its
new name, where PySpark reads __name__ at the call.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]