Hyukjin Kwon created SPARK-61951:
------------------------------------

             Summary: Convert the return type of a UDTF when its plan is built 
and before spark.udtf.register pickles the handler, and take the name of a data 
source as protobuf's string field, as PySpark does
                 Key: SPARK-61951
                 URL: https://issues.apache.org/jira/browse/SPARK-61951
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Round-3 review of UDTFs and data sources against pyspark-client 4.3.0:

1. A UDTF returnType that is neither a DataType nor a str failed at the call: 
udtf(H, returnType=5)(lit(1)) raised "TypeError: expected a DataType or a DDL 
type string". PySpark keeps the value (connect/udtf.py:159-165) and converts it 
when it builds the plan, before pickling the handler (plan.py:3073-3079). So 
its call succeeds, and schema, collect and register raise 
"[UNSUPPORTED_OPERATION] data type 5 is not supported." 
(connect/types.py:210-214).

2. spark.udtf.register pickled the handler before converting the name and the 
type. So with an unpicklable handler it raised UDTF_SERIALIZATION_ERROR first, 
where PySpark raises:
- TypeError "bad argument type for built-in operation" for register(5, ...);
- UnicodeEncodeError for a lone-surrogate name;
- UNSUPPORTED_OPERATION for returnType=5.

3. spark.dataSource.register took name() only as a str (datasource.rs). PySpark 
puts it into a protobuf string field (core.py:1159-1162, plan.py:3185), so 
bytes of UTF-8 register, and other values raise protobuf's errors, before the 
class is pickled.

4. UDTF object details:
- UserDefinedTableFunction(func) accepted a missing returnType, which PySpark's 
__init__ requires;
- vars() of a UDTF listed its attributes in another order;
- TableArg lacked PySpark's _subquery_expr and _is_partitioned().




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to