[ 
https://issues.apache.org/jira/browse/SPARK-61965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61965:
---------------------------------
    Fix Version/s: connect-rust-4.4.0

> Convert the name and the return type before spark.client.register_udtf 
> pickles the handler, as PySpark's register_udtf does
> ---------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61965
>                 URL: https://issues.apache.org/jira/browse/SPARK-61965
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark's SparkConnectClient.register_udtf (core.py:1117-1149) makes a 
> PythonUDTF, which reads the
> __name__ of the handler class (plan.py:3062), and then its udtf_plan sets the 
> name in the string
> field of its command (plan.py:3138) and converts the return type with 
> pyspark_types_to_proto_types
> (PythonUDTF.to_plan, plan.py:3075-3076) before it pickles the handler 
> (plan.py:3078-3088). The
> drop-in's spark.client.register_udtf pickled the handler before 
> _registerPythonUdtf converted the
> name, so of a handler that does not pickle it raised UDTF_SERIALIZATION_ERROR 
> where PySpark raises
> the error of the name or of the return type. spark.udtf.register does this in 
> PySpark's order since
> SPARK-61951.
> Repro (a handler class that closes over a threading.Lock, so that cloudpickle 
> cannot pickle it)
>     client = spark.client
>     client.register_udtf(Unpicklable, "x int", 5)
>     client.register_udtf(Unpicklable, "x int", "\udc80")
>     client.register_udtf(Unpicklable, StructField("x", IntegerType()), "t")
>     client.register_udtf(Unpicklable, "x \udc80", "t")
>     client.register_udtf(Unpicklable, 5, 5)
>     client.register_udtf(object(), 5, 5)
> Actual (drop-in): UDTF_SERIALIZATION_ERROR for the first four, and 
> UNSUPPORTED_OPERATION of
> "data type 5" for the last two, as its own check of the return type came 
> first.
> Expected (pyspark-client 4.3.0): TypeError: bad argument type for built-in 
> operation;
> UnicodeEncodeError; PySparkValueError UNSUPPORTED_OPERATION of "data type 
> StructField('x',
> IntegerType(), True)"; UnicodeEncodeError; the TypeError of the name; 
> AttributeError: 'object' object
> has no attribute '__name__'. A valid name and return type raise 
> UDTF_SERIALIZATION_ERROR on both.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to