Hyukjin Kwon created SPARK-61980:
------------------------------------
Summary: Refuse the types that have no proto or Arrow type where
PySpark refuses them: pipeline schemas, the reader's parse of a JSON schema,
the SQL type of a user-defined type and local rows
Key: SPARK-61980
URL: https://issues.apache.org/jira/browse/SPARK-61980
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Problem
PySpark 4.3.0 has no proto and no Arrow type of a type of its class only, such
as AtomicType(),
DataType() or NumericType(), nor of a StructField used as a type.
pyspark_types_to_proto_types
raises UNSUPPORTED_OPERATION, to_arrow_type raises
UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION, and
StructType.fromJson raises CANNOT_PARSE_DATATYPE or UNSUPPORTED_DATA_TYPE of
their JSON. The
drop-in lowers such a type into an unparsed core type of its JSON. It refused
it in most places,
but not in these:
- the schema of a pipeline's table or materialized view (register_output
converts it with
pyspark_types_to_proto_types): the definition was sent, and the server
answered;
- the reader's csv/json/xml of a DataFrame of strings, whose Parse plan reads
the JSON schema with
StructType.fromJson when it is built: the call raised the core's
INVALID_DATATYPE_STRING or
INVALID_DATATYPE_FORMAT, or the plan was sent, as for the names "null" and
"calendarinterval",
which PySpark refuses too;
- such a type in the sqlType() of a user-defined type, which
pyspark_types_to_proto_types and
to_arrow_type convert too: a cast, a non-Arrow UDF, registerJavaFunction,
createTable, a UDTF
and DataFrame.to sent their plans for the server to refuse, and
createDataFrame of local rows
raised TypeError: unsupported value type for createDataFrame;
- local rows of such a type raised UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION
before their values
were converted. PySpark's LocalDataToArrowConversion.convert runs the
converters first, then
to_arrow_schema (python/pyspark/sql/conversion.py:976-1006), so the errors of
a UDT's
serialize, of a decimal, binary or non-nullable value, and a CharType field
before the type
come first;
- the schema JSON of a relation of a NumPy array or of data of no rows was the
core's
unparsed(...) text, where PySpark sends the schema's json(), which writes
"atomic" or the
field's JSON: the server's PARSE_SYNTAX_ERROR echoed another JSON.
Repro (master 4f0d0fa9, against a Spark 4.3.0 server)
>>> schema = StructType([StructField("a", AtomicType())])
>>> spark.read.schema(schema).csv(spark.range(1))
PySparkValueError: [INVALID_DATATYPE_STRING] ...
>>> class U(UserDefinedType):
... @classmethod
... def sqlType(cls): return StructType([StructField("v", AtomicType())])
... @classmethod
... def module(cls): return "__main__"
... def serialize(self, obj): return (obj,)
>>> spark.range(1).select(col("id").cast(U())).schema
InvalidPlanInput:
[CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_PROTO_TO_CATALYST] ...
>>> spark.createDataFrame([(1,)], StructType([StructField("a", U())]))
TypeError: unsupported value type for createDataFrame: int
>>> spark.createDataFrame([(None,)], StructType([StructField("a",
AtomicType(), False)]))
PySparkTypeError: [UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION] AtomicType()
is not supported in conversion to Arrow.
Expected (as with pyspark-client 4.3.0)
The reader's DataFrame is made, and each request raises
[CANNOT_PARSE_DATATYPE] Unable to parse datatype. atomic.
The cast raises [UNSUPPORTED_OPERATION] data type AtomicType() is not
supported.
createDataFrame of U raises
[UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION] AtomicType() is not supported
in conversion to Arrow.
The non-nullable None raises PySparkValueError: input for AtomicType() must
not be None
A pipeline's @dp.materialized_view(schema=schema) raises
[UNSUPPORTED_OPERATION] data type AtomicType() is not supported.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]