[
https://issues.apache.org/jira/browse/SPARK-61995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61995:
-----------------------------------
Labels: pull-request-available (was: )
> Convert the return type of a UDTF each time its plan is built, as PySpark's
> PythonUDTF.to_plan
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-61995
> URL: https://issues.apache.org/jira/browse/SPARK-61995
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> A follow-up of SPARK-61951 and SPARK-61962. PySpark's PythonUDTF.to_plan
> converts the return type of a UDTF with pyspark_types_to_proto_types each
> time it builds a plan of a call, of the type object that the call kept, and
> pickles the handler class then. The drop-in converted a valid return type at
> the call, and only checked it at the build. So a StructType changed in place
> after the call, as by add(), is sent as it was at the call: after rt =
> StructType([a long]); d = udtf(H, returnType=rt)(lit(1)); rt.add("b",
> LongType()), PySpark's d.schema is struct<a:bigint,b:bigint> and its collect
> gives Row(a=1, b=2), where the drop-in's schema had only a and its collect
> failed with a PythonException; a lateralJoin of the call had the columns
> ['id', 'a'] where PySpark's has ['id', 'a', 'b']. Also, a str assigned to the
> UDTF's returnType after udtf() raised UNSUPPORTED_OPERATION where PySpark
> parses it, and UDTF_SERIALIZATION_ERROR named a handler class renamed after
> the call by its new name, where PySpark reads __name__ at the call.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]