Hyukjin Kwon created SPARK-61995:
------------------------------------

             Summary: Convert the return type of a UDTF each time its plan is 
built, as PySpark's PythonUDTF.to_plan
                 Key: SPARK-61995
                 URL: https://issues.apache.org/jira/browse/SPARK-61995
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


A follow-up of SPARK-61951 and SPARK-61962. PySpark's PythonUDTF.to_plan 
converts the return type of a UDTF with pyspark_types_to_proto_types each time 
it builds a plan of a call, of the type object that the call kept, and pickles 
the handler class then. The drop-in converted a valid return type at the call, 
and only checked it at the build. So a StructType changed in place after the 
call, as by add(), is sent as it was at the call: after rt = StructType([a 
long]); d = udtf(H, returnType=rt)(lit(1)); rt.add("b", LongType()), PySpark's 
d.schema is struct<a:bigint,b:bigint> and its collect gives Row(a=1, b=2), 
where the drop-in's schema had only a and its collect failed with a 
PythonException; a lateralJoin of the call had the columns ['id', 'a'] where 
PySpark's has ['id', 'a', 'b']. Also, a str assigned to the UDTF's returnType 
after udtf() raised UNSUPPORTED_OPERATION where PySpark parses it, and 
UDTF_SERIALIZATION_ERROR named a handler class renamed after the call by its 
new name, where PySpark reads __name__ at the call.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to