[ 
https://issues.apache.org/jira/browse/SPARK-61916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61916:
------------------------------------

    Assignee: Hyukjin Kwon

> Raise PySpark's errors for abstract data types given to UDTF return types, 
> registerJavaFunction, table schemas and createDataFrame
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61916
>                 URL: https://issues.apache.org/jira/browse/SPARK-61916
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> After SPARK-61859, the base data type classes such as AtomicType() can be 
> instantiated, as in PySpark, and PySpark's conversion of a type to its proto 
> raises UNSUPPORTED_OPERATION for them, also when nested. Four places of the 
> drop-in that take a type do not use that conversion, so a nested abstract 
> type reaches the server as a DataType.Unparsed of its JSON, where PySpark 
> raises in the client:
> - the return type of udtf(...)() and of spark.udtf.register;
> - the return type of spark.udf.registerJavaFunction;
> - the schema of spark.catalog.createTable.
> For example, udtf(Fn, returnType=StructType([StructField("a", 
> AtomicType())]))().schema raises the server's 
> CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_PROTO_TO_CATALYST {"kindCase": 
> "UNPARSED"} on the drop-in, and PySpark's UNSUPPORTED_OPERATION {"operation": 
> "data type AtomicType()"} on pyspark-client 4.3.0. registerJavaFunction("f", 
> "cls", AtomicType()) raises PARSE_SYNTAX_ERROR. The same already happened for 
> a StructField nested as a type.
> Also, createDataFrame with an abstract type as its schema classifies the 
> schema by the core type rather than the Python object: createDataFrame([1], 
> AtomicType()) raises NOT_EXPECTED_TYPE {"arg_type": "UnparsedDataType"}, 
> where PySpark raises UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION {"data_type": 
> "AtomicType()"}, and createDataFrame([1], DataType()) names 
> "UnparsedDataType" where PySpark names "DataType".
> These should raise PySpark's errors, as cast and try_cast already do.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to