Hyukjin Kwon created SPARK-61965:
------------------------------------

             Summary: Convert the name and the return type before 
spark.client.register_udtf pickles the handler, as PySpark's register_udtf does
                 Key: SPARK-61965
                 URL: https://issues.apache.org/jira/browse/SPARK-61965
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Problem
PySpark's SparkConnectClient.register_udtf (core.py:1117-1149) makes a 
PythonUDTF, which reads the
__name__ of the handler class (plan.py:3062), and then its udtf_plan sets the 
name in the string
field of its command (plan.py:3138) and converts the return type with 
pyspark_types_to_proto_types
(PythonUDTF.to_plan, plan.py:3075-3076) before it pickles the handler 
(plan.py:3078-3088). The
drop-in's spark.client.register_udtf pickled the handler before 
_registerPythonUdtf converted the
name, so of a handler that does not pickle it raised UDTF_SERIALIZATION_ERROR 
where PySpark raises
the error of the name or of the return type. spark.udtf.register does this in 
PySpark's order since
SPARK-61951.

Repro (a handler class that closes over a threading.Lock, so that cloudpickle 
cannot pickle it)
    client = spark.client
    client.register_udtf(Unpicklable, "x int", 5)
    client.register_udtf(Unpicklable, "x int", "\udc80")
    client.register_udtf(Unpicklable, StructField("x", IntegerType()), "t")
    client.register_udtf(Unpicklable, "x \udc80", "t")
    client.register_udtf(Unpicklable, 5, 5)
    client.register_udtf(object(), 5, 5)
Actual (drop-in): UDTF_SERIALIZATION_ERROR for the first four, and 
UNSUPPORTED_OPERATION of
"data type 5" for the last two, as its own check of the return type came first.
Expected (pyspark-client 4.3.0): TypeError: bad argument type for built-in 
operation;
UnicodeEncodeError; PySparkValueError UNSUPPORTED_OPERATION of "data type 
StructField('x',
IntegerType(), True)"; UnicodeEncodeError; the TypeError of the name; 
AttributeError: 'object' object
has no attribute '__name__'. A valid name and return type raise 
UDTF_SERIALIZATION_ERROR on both.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to