Hyukjin Kwon created SPARK-61094:
------------------------------------
Summary: Add UnparsedDataType of pyspark.sql.connect.types, the
returnType of a UDTF of a DDL string, in the Python client as PySpark does
Key: SPARK-61094
URL: https://issues.apache.org/jira/browse/SPARK-61094
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{pyspark.sql.connect.types.UnparsedDataType}} is the type
of a DDL string that the server parses: its {{simpleString}} is
"unparsed('<ddl>')", its repr "UnparsedDataType('<ddl>')", and its
{{jsonValue}}, {{needConversion}}, {{toInternal}} and {{fromInternal}} raise
{{INVALID_CALL_ON_UNRESOLVED_OBJECT}}. The {{returnType}} of a UDTF of a DDL
string is an {{UnparsedDataType}}, and {{spark.dataSource.register}} of a class
without a {{name}} raises the {{AttributeError}} of {{name()}}.
The Rust-backed Python client lacks {{pyspark.sql.connect.types}}, keeps the
DDL string as the {{returnType}} of a UDTF, and raises a {{TypeError}}
("DataSource class must have a classmethod name()") of {{register}}. Add them
as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
