Hyukjin Kwon created SPARK-61916:
------------------------------------
Summary: Raise PySpark's errors for abstract data types given to
UDTF return types, registerJavaFunction, table schemas and createDataFrame
Key: SPARK-61916
URL: https://issues.apache.org/jira/browse/SPARK-61916
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
After SPARK-61859, the base data type classes such as AtomicType() can be
instantiated, as in PySpark, and PySpark's conversion of a type to its proto
raises UNSUPPORTED_OPERATION for them, also when nested. Four places of the
drop-in that take a type do not use that conversion, so a nested abstract type
reaches the server as a DataType.Unparsed of its JSON, where PySpark raises in
the client:
- the return type of udtf(...)() and of spark.udtf.register;
- the return type of spark.udf.registerJavaFunction;
- the schema of spark.catalog.createTable.
For example, udtf(Fn, returnType=StructType([StructField("a",
AtomicType())]))().schema raises the server's
CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_PROTO_TO_CATALYST {"kindCase":
"UNPARSED"} on the drop-in, and PySpark's UNSUPPORTED_OPERATION {"operation":
"data type AtomicType()"} on pyspark-client 4.3.0. registerJavaFunction("f",
"cls", AtomicType()) raises PARSE_SYNTAX_ERROR. The same already happened for a
StructField nested as a type.
Also, createDataFrame with an abstract type as its schema classifies the schema
by the core type rather than the Python object: createDataFrame([1],
AtomicType()) raises NOT_EXPECTED_TYPE {"arg_type": "UnparsedDataType"}, where
PySpark raises UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION {"data_type":
"AtomicType()"}, and createDataFrame([1], DataType()) names "UnparsedDataType"
where PySpark names "DataType".
These should raise PySpark's errors, as cast and try_cast already do.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]