Hyukjin Kwon created SPARK-61242:
------------------------------------
Summary: DataType.from_ddl rejects NOT NULL and COMMENT of a
field, and type_name of intervals is "interval"
Key: SPARK-61242
URL: https://issues.apache.org/jira/browse/SPARK-61242
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
DataType::from_ddl rejects the NOT NULL and COMMENT clauses of a field, at top
level ("a INT NOT NULL, b STRING COMMENT 'x'") and inside struct<...>, with
INVALID_DATATYPE_FORMAT. So DataType::to_ddl output with a non-nullable or
commented field does not parse back, while Spark's parser accepts it (nullable
false, the comment in the field metadata under "comment").
DataType::type_name returns "interval" for day-time and year-month intervals,
where PySpark's typeName gives "daytimeinterval" and "yearmonthinterval". The
UDT name is now the lower-cased class name, as UserDefinedType.typeName() gives
it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
