[
https://issues.apache.org/jira/browse/SPARK-61980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61980:
------------------------------------
Assignee: Hyukjin Kwon
> Refuse the types that have no proto or Arrow type where PySpark refuses them:
> pipeline schemas, the reader's parse of a JSON schema, the SQL type of a
> user-defined type and local rows
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61980
> URL: https://issues.apache.org/jira/browse/SPARK-61980
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark 4.3.0 has no proto and no Arrow type of a type of its class only,
> such as AtomicType(),
> DataType() or NumericType(), nor of a StructField used as a type.
> pyspark_types_to_proto_types
> raises UNSUPPORTED_OPERATION, to_arrow_type raises
> UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION, and
> StructType.fromJson raises CANNOT_PARSE_DATATYPE or UNSUPPORTED_DATA_TYPE of
> their JSON. The
> drop-in lowers such a type into an unparsed core type of its JSON. It refused
> it in most places,
> but not in these:
> - the schema of a pipeline's table or materialized view (register_output
> converts it with
> pyspark_types_to_proto_types): the definition was sent, and the server
> answered;
> - the reader's csv/json/xml of a DataFrame of strings, whose Parse plan reads
> the JSON schema with
> StructType.fromJson when it is built: the call raised the core's
> INVALID_DATATYPE_STRING or
> INVALID_DATATYPE_FORMAT, or the plan was sent, as for the names "null" and
> "calendarinterval",
> which PySpark refuses too;
> - such a type in the sqlType() of a user-defined type, which
> pyspark_types_to_proto_types and
> to_arrow_type convert too: a cast, a non-Arrow UDF, registerJavaFunction,
> createTable, a UDTF
> and DataFrame.to sent their plans for the server to refuse, and
> createDataFrame of local rows
> raised TypeError: unsupported value type for createDataFrame;
> - local rows of such a type raised UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION
> before their values
> were converted. PySpark's LocalDataToArrowConversion.convert runs the
> converters first, then
> to_arrow_schema (python/pyspark/sql/conversion.py:976-1006), so the errors
> of a UDT's
> serialize, of a decimal, binary or non-nullable value, and a CharType field
> before the type
> come first;
> - the schema JSON of a relation of a NumPy array or of data of no rows was
> the core's
> unparsed(...) text, where PySpark sends the schema's json(), which writes
> "atomic" or the
> field's JSON: the server's PARSE_SYNTAX_ERROR echoed another JSON.
> Repro (master 4f0d0fa9, against a Spark 4.3.0 server)
> >>> schema = StructType([StructField("a", AtomicType())])
> >>> spark.read.schema(schema).csv(spark.range(1))
> PySparkValueError: [INVALID_DATATYPE_STRING] ...
> >>> class U(UserDefinedType):
> ... @classmethod
> ... def sqlType(cls): return StructType([StructField("v",
> AtomicType())])
> ... @classmethod
> ... def module(cls): return "__main__"
> ... def serialize(self, obj): return (obj,)
> >>> spark.range(1).select(col("id").cast(U())).schema
> InvalidPlanInput:
> [CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_PROTO_TO_CATALYST] ...
> >>> spark.createDataFrame([(1,)], StructType([StructField("a", U())]))
> TypeError: unsupported value type for createDataFrame: int
> >>> spark.createDataFrame([(None,)], StructType([StructField("a",
> AtomicType(), False)]))
> PySparkTypeError: [UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION] AtomicType()
> is not supported in conversion to Arrow.
> Expected (as with pyspark-client 4.3.0)
> The reader's DataFrame is made, and each request raises
> [CANNOT_PARSE_DATATYPE] Unable to parse datatype. atomic.
> The cast raises [UNSUPPORTED_OPERATION] data type AtomicType() is not
> supported.
> createDataFrame of U raises
> [UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION] AtomicType() is not
> supported in conversion to Arrow.
> The non-nullable None raises PySparkValueError: input for AtomicType() must
> not be None
> A pipeline's @dp.materialized_view(schema=schema) raises
> [UNSUPPORTED_OPERATION] data type AtomicType() is not supported.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]