[ 
https://issues.apache.org/jira/browse/SPARK-61951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61951:
-----------------------------------
    Labels: pull-request-available  (was: )

> Convert the return type of a UDTF when its plan is built and before 
> spark.udtf.register pickles the handler, and take the name of a data source 
> as protobuf's string field, as PySpark does
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61951
>                 URL: https://issues.apache.org/jira/browse/SPARK-61951
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Round-3 review of UDTFs and data sources against pyspark-client 4.3.0:
> 1. A UDTF returnType that is neither a DataType nor a str failed at the call: 
> udtf(H, returnType=5)(lit(1)) raised "TypeError: expected a DataType or a DDL 
> type string". PySpark keeps the value (connect/udtf.py:159-165) and converts 
> it when it builds the plan, before pickling the handler (plan.py:3073-3079). 
> So its call succeeds, and schema, collect and register raise 
> "[UNSUPPORTED_OPERATION] data type 5 is not supported." 
> (connect/types.py:210-214).
> 2. spark.udtf.register pickled the handler before converting the name and the 
> type. So with an unpicklable handler it raised UDTF_SERIALIZATION_ERROR 
> first, where PySpark raises:
> - TypeError "bad argument type for built-in operation" for register(5, ...);
> - UnicodeEncodeError for a lone-surrogate name;
> - UNSUPPORTED_OPERATION for returnType=5.
> 3. spark.dataSource.register took name() only as a str (datasource.rs). 
> PySpark puts it into a protobuf string field (core.py:1159-1162, 
> plan.py:3185), so bytes of UTF-8 register, and other values raise protobuf's 
> errors, before the class is pickled.
> 4. UDTF object details:
> - UserDefinedTableFunction(func) accepted a missing returnType, which 
> PySpark's __init__ requires;
> - vars() of a UDTF listed its attributes in another order;
> - TableArg lacked PySpark's _subquery_expr and _is_partitioned().



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to