[ 
https://issues.apache.org/jira/browse/SPARK-61995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61995.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 847
[https://github.com/apache/spark-connect-rust/pull/847]

> Convert the return type of a UDTF each time its plan is built, as PySpark's 
> PythonUDTF.to_plan
> ----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61995
>                 URL: https://issues.apache.org/jira/browse/SPARK-61995
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> A follow-up of SPARK-61951 and SPARK-61962. PySpark's PythonUDTF.to_plan 
> converts the return type of a UDTF with pyspark_types_to_proto_types each 
> time it builds a plan of a call, of the type object that the call kept, and 
> pickles the handler class then. The drop-in converted a valid return type at 
> the call, and only checked it at the build. So a StructType changed in place 
> after the call, as by add(), is sent as it was at the call: after rt = 
> StructType([a long]); d = udtf(H, returnType=rt)(lit(1)); rt.add("b", 
> LongType()), PySpark's d.schema is struct<a:bigint,b:bigint> and its collect 
> gives Row(a=1, b=2), where the drop-in's schema had only a and its collect 
> failed with a PythonException; a lateralJoin of the call had the columns 
> ['id', 'a'] where PySpark's has ['id', 'a', 'b']. Also, a str assigned to the 
> UDTF's returnType after udtf() raised UNSUPPORTED_OPERATION where PySpark 
> parses it, and UDTF_SERIALIZATION_ERROR named a handler class renamed after 
> the call by its new name, where PySpark reads __name__ at the call.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to