[
https://issues.apache.org/jira/browse/SPARK-61916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61916:
-----------------------------------
Labels: pull-request-available (was: )
> Raise PySpark's errors for abstract data types given to UDTF return types,
> registerJavaFunction, table schemas and createDataFrame
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61916
> URL: https://issues.apache.org/jira/browse/SPARK-61916
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> After SPARK-61859, the base data type classes such as AtomicType() can be
> instantiated, as in PySpark, and PySpark's conversion of a type to its proto
> raises UNSUPPORTED_OPERATION for them, also when nested. Four places of the
> drop-in that take a type do not use that conversion, so a nested abstract
> type reaches the server as a DataType.Unparsed of its JSON, where PySpark
> raises in the client:
> - the return type of udtf(...)() and of spark.udtf.register;
> - the return type of spark.udf.registerJavaFunction;
> - the schema of spark.catalog.createTable.
> For example, udtf(Fn, returnType=StructType([StructField("a",
> AtomicType())]))().schema raises the server's
> CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_PROTO_TO_CATALYST {"kindCase":
> "UNPARSED"} on the drop-in, and PySpark's UNSUPPORTED_OPERATION {"operation":
> "data type AtomicType()"} on pyspark-client 4.3.0. registerJavaFunction("f",
> "cls", AtomicType()) raises PARSE_SYNTAX_ERROR. The same already happened for
> a StructField nested as a type.
> Also, createDataFrame with an abstract type as its schema classifies the
> schema by the core type rather than the Python object: createDataFrame([1],
> AtomicType()) raises NOT_EXPECTED_TYPE {"arg_type": "UnparsedDataType"},
> where PySpark raises UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION {"data_type":
> "AtomicType()"}, and createDataFrame([1], DataType()) names
> "UnparsedDataType" where PySpark names "DataType".
> These should raise PySpark's errors, as cast and try_cast already do.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]