[ 
https://issues.apache.org/jira/browse/SPARK-61980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61980:
------------------------------------

    Assignee: Hyukjin Kwon

> Refuse the types that have no proto or Arrow type where PySpark refuses them: 
> pipeline schemas, the reader's parse of a JSON schema, the SQL type of a 
> user-defined type and local rows
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61980
>                 URL: https://issues.apache.org/jira/browse/SPARK-61980
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark 4.3.0 has no proto and no Arrow type of a type of its class only, 
> such as AtomicType(),
> DataType() or NumericType(), nor of a StructField used as a type. 
> pyspark_types_to_proto_types
> raises UNSUPPORTED_OPERATION, to_arrow_type raises 
> UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION, and
> StructType.fromJson raises CANNOT_PARSE_DATATYPE or UNSUPPORTED_DATA_TYPE of 
> their JSON. The
> drop-in lowers such a type into an unparsed core type of its JSON. It refused 
> it in most places,
> but not in these:
> - the schema of a pipeline's table or materialized view (register_output 
> converts it with
>   pyspark_types_to_proto_types): the definition was sent, and the server 
> answered;
> - the reader's csv/json/xml of a DataFrame of strings, whose Parse plan reads 
> the JSON schema with
>   StructType.fromJson when it is built: the call raised the core's 
> INVALID_DATATYPE_STRING or
>   INVALID_DATATYPE_FORMAT, or the plan was sent, as for the names "null" and 
> "calendarinterval",
>   which PySpark refuses too;
> - such a type in the sqlType() of a user-defined type, which 
> pyspark_types_to_proto_types and
>   to_arrow_type convert too: a cast, a non-Arrow UDF, registerJavaFunction, 
> createTable, a UDTF
>   and DataFrame.to sent their plans for the server to refuse, and 
> createDataFrame of local rows
>   raised TypeError: unsupported value type for createDataFrame;
> - local rows of such a type raised UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION 
> before their values
>   were converted. PySpark's LocalDataToArrowConversion.convert runs the 
> converters first, then
>   to_arrow_schema (python/pyspark/sql/conversion.py:976-1006), so the errors 
> of a UDT's
>   serialize, of a decimal, binary or non-nullable value, and a CharType field 
> before the type
>   come first;
> - the schema JSON of a relation of a NumPy array or of data of no rows was 
> the core's
>   unparsed(...) text, where PySpark sends the schema's json(), which writes 
> "atomic" or the
>   field's JSON: the server's PARSE_SYNTAX_ERROR echoed another JSON.
> Repro (master 4f0d0fa9, against a Spark 4.3.0 server)
>   >>> schema = StructType([StructField("a", AtomicType())])
>   >>> spark.read.schema(schema).csv(spark.range(1))
>   PySparkValueError: [INVALID_DATATYPE_STRING] ...
>   >>> class U(UserDefinedType):
>   ...     @classmethod
>   ...     def sqlType(cls): return StructType([StructField("v", 
> AtomicType())])
>   ...     @classmethod
>   ...     def module(cls): return "__main__"
>   ...     def serialize(self, obj): return (obj,)
>   >>> spark.range(1).select(col("id").cast(U())).schema
>   InvalidPlanInput: 
> [CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_PROTO_TO_CATALYST] ...
>   >>> spark.createDataFrame([(1,)], StructType([StructField("a", U())]))
>   TypeError: unsupported value type for createDataFrame: int
>   >>> spark.createDataFrame([(None,)], StructType([StructField("a", 
> AtomicType(), False)]))
>   PySparkTypeError: [UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION] AtomicType() 
> is not supported in conversion to Arrow.
> Expected (as with pyspark-client 4.3.0)
>   The reader's DataFrame is made, and each request raises
>     [CANNOT_PARSE_DATATYPE] Unable to parse datatype. atomic.
>   The cast raises [UNSUPPORTED_OPERATION] data type AtomicType() is not 
> supported.
>   createDataFrame of U raises
>     [UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION] AtomicType() is not 
> supported in conversion to Arrow.
>   The non-nullable None raises PySparkValueError: input for AtomicType() must 
> not be None
>   A pipeline's @dp.materialized_view(schema=schema) raises
>     [UNSUPPORTED_OPERATION] data type AtomicType() is not supported.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to