[
https://issues.apache.org/jira/browse/SPARK-61951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61951:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Convert the return type of a UDTF when its plan is built and before
> spark.udtf.register pickles the handler, and take the name of a data source
> as protobuf's string field, as PySpark does
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61951
> URL: https://issues.apache.org/jira/browse/SPARK-61951
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> Round-3 review of UDTFs and data sources against pyspark-client 4.3.0:
> 1. A UDTF returnType that is neither a DataType nor a str failed at the call:
> udtf(H, returnType=5)(lit(1)) raised "TypeError: expected a DataType or a DDL
> type string". PySpark keeps the value (connect/udtf.py:159-165) and converts
> it when it builds the plan, before pickling the handler (plan.py:3073-3079).
> So its call succeeds, and schema, collect and register raise
> "[UNSUPPORTED_OPERATION] data type 5 is not supported."
> (connect/types.py:210-214).
> 2. spark.udtf.register pickled the handler before converting the name and the
> type. So with an unpicklable handler it raised UDTF_SERIALIZATION_ERROR
> first, where PySpark raises:
> - TypeError "bad argument type for built-in operation" for register(5, ...);
> - UnicodeEncodeError for a lone-surrogate name;
> - UNSUPPORTED_OPERATION for returnType=5.
> 3. spark.dataSource.register took name() only as a str (datasource.rs).
> PySpark puts it into a protobuf string field (core.py:1159-1162,
> plan.py:3185), so bytes of UTF-8 register, and other values raise protobuf's
> errors, before the class is pickled.
> 4. UDTF object details:
> - UserDefinedTableFunction(func) accepted a missing returnType, which
> PySpark's __init__ requires;
> - vars() of a UDTF listed its attributes in another order;
> - TableArg lacked PySpark's _subquery_expr and _is_partitioned().
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]