Hyukjin Kwon created SPARK-61980:
------------------------------------

             Summary: Refuse the types that have no proto or Arrow type where 
PySpark refuses them: pipeline schemas, the reader's parse of a JSON schema, 
the SQL type of a user-defined type and local rows
                 Key: SPARK-61980
                 URL: https://issues.apache.org/jira/browse/SPARK-61980
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Problem
PySpark 4.3.0 has no proto and no Arrow type of a type of its class only, such 
as AtomicType(),
DataType() or NumericType(), nor of a StructField used as a type. 
pyspark_types_to_proto_types
raises UNSUPPORTED_OPERATION, to_arrow_type raises 
UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION, and
StructType.fromJson raises CANNOT_PARSE_DATATYPE or UNSUPPORTED_DATA_TYPE of 
their JSON. The
drop-in lowers such a type into an unparsed core type of its JSON. It refused 
it in most places,
but not in these:
- the schema of a pipeline's table or materialized view (register_output 
converts it with
  pyspark_types_to_proto_types): the definition was sent, and the server 
answered;
- the reader's csv/json/xml of a DataFrame of strings, whose Parse plan reads 
the JSON schema with
  StructType.fromJson when it is built: the call raised the core's 
INVALID_DATATYPE_STRING or
  INVALID_DATATYPE_FORMAT, or the plan was sent, as for the names "null" and 
"calendarinterval",
  which PySpark refuses too;
- such a type in the sqlType() of a user-defined type, which 
pyspark_types_to_proto_types and
  to_arrow_type convert too: a cast, a non-Arrow UDF, registerJavaFunction, 
createTable, a UDTF
  and DataFrame.to sent their plans for the server to refuse, and 
createDataFrame of local rows
  raised TypeError: unsupported value type for createDataFrame;
- local rows of such a type raised UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION 
before their values
  were converted. PySpark's LocalDataToArrowConversion.convert runs the 
converters first, then
  to_arrow_schema (python/pyspark/sql/conversion.py:976-1006), so the errors of 
a UDT's
  serialize, of a decimal, binary or non-nullable value, and a CharType field 
before the type
  come first;
- the schema JSON of a relation of a NumPy array or of data of no rows was the 
core's
  unparsed(...) text, where PySpark sends the schema's json(), which writes 
"atomic" or the
  field's JSON: the server's PARSE_SYNTAX_ERROR echoed another JSON.

Repro (master 4f0d0fa9, against a Spark 4.3.0 server)
  >>> schema = StructType([StructField("a", AtomicType())])
  >>> spark.read.schema(schema).csv(spark.range(1))
  PySparkValueError: [INVALID_DATATYPE_STRING] ...
  >>> class U(UserDefinedType):
  ...     @classmethod
  ...     def sqlType(cls): return StructType([StructField("v", AtomicType())])
  ...     @classmethod
  ...     def module(cls): return "__main__"
  ...     def serialize(self, obj): return (obj,)
  >>> spark.range(1).select(col("id").cast(U())).schema
  InvalidPlanInput: 
[CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_PROTO_TO_CATALYST] ...
  >>> spark.createDataFrame([(1,)], StructType([StructField("a", U())]))
  TypeError: unsupported value type for createDataFrame: int
  >>> spark.createDataFrame([(None,)], StructType([StructField("a", 
AtomicType(), False)]))
  PySparkTypeError: [UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION] AtomicType() 
is not supported in conversion to Arrow.

Expected (as with pyspark-client 4.3.0)
  The reader's DataFrame is made, and each request raises
    [CANNOT_PARSE_DATATYPE] Unable to parse datatype. atomic.
  The cast raises [UNSUPPORTED_OPERATION] data type AtomicType() is not 
supported.
  createDataFrame of U raises
    [UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION] AtomicType() is not supported 
in conversion to Arrow.
  The non-nullable None raises PySparkValueError: input for AtomicType() must 
not be None
  A pipeline's @dp.materialized_view(schema=schema) raises
    [UNSUPPORTED_OPERATION] data type AtomicType() is not supported.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to