DB Tsai created SPARK-60678:
-------------------------------
Summary: Accept DDL return schemas for Python UDTFs in Spark
Connect Rust
Key: SPARK-60678
URL: https://issues.apache.org/jira/browse/SPARK-60678
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
The Rust Python client rejects the public Python UDTF returnType string form
(for example a struct DDL schema) with TypeError: argument return_type: str
object is not an instance of DataType. Official PySpark Connect accepts this
API. The benchmark udf-udtf-construct, udf-udtf-first-query and
udf-udtf-repeat-query all fail on Rust and pass on official Python. Support the
string schema form with matching parsing/validation and retain support for
StructType. Add focused positive/negative compatibility and plan/protocol
tests; rerun the three benchmark cases and UDF regression guards. Baseline
Rust: d884e347452c. This is the client schema serialization path, not
worker-side UDTF execution.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
