[
https://issues.apache.org/jira/browse/SPARK-61965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61965:
------------------------------------
Assignee: Hyukjin Kwon
> Convert the name and the return type before spark.client.register_udtf
> pickles the handler, as PySpark's register_udtf does
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61965
> URL: https://issues.apache.org/jira/browse/SPARK-61965
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark's SparkConnectClient.register_udtf (core.py:1117-1149) makes a
> PythonUDTF, which reads the
> __name__ of the handler class (plan.py:3062), and then its udtf_plan sets the
> name in the string
> field of its command (plan.py:3138) and converts the return type with
> pyspark_types_to_proto_types
> (PythonUDTF.to_plan, plan.py:3075-3076) before it pickles the handler
> (plan.py:3078-3088). The
> drop-in's spark.client.register_udtf pickled the handler before
> _registerPythonUdtf converted the
> name, so of a handler that does not pickle it raised UDTF_SERIALIZATION_ERROR
> where PySpark raises
> the error of the name or of the return type. spark.udtf.register does this in
> PySpark's order since
> SPARK-61951.
> Repro (a handler class that closes over a threading.Lock, so that cloudpickle
> cannot pickle it)
> client = spark.client
> client.register_udtf(Unpicklable, "x int", 5)
> client.register_udtf(Unpicklable, "x int", "\udc80")
> client.register_udtf(Unpicklable, StructField("x", IntegerType()), "t")
> client.register_udtf(Unpicklable, "x \udc80", "t")
> client.register_udtf(Unpicklable, 5, 5)
> client.register_udtf(object(), 5, 5)
> Actual (drop-in): UDTF_SERIALIZATION_ERROR for the first four, and
> UNSUPPORTED_OPERATION of
> "data type 5" for the last two, as its own check of the return type came
> first.
> Expected (pyspark-client 4.3.0): TypeError: bad argument type for built-in
> operation;
> UnicodeEncodeError; PySparkValueError UNSUPPORTED_OPERATION of "data type
> StructField('x',
> IntegerType(), True)"; UnicodeEncodeError; the TypeError of the name;
> AttributeError: 'object' object
> has no attribute '__name__'. A valid name and return type raise
> UDTF_SERIALIZATION_ERROR on both.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]