[
https://issues.apache.org/jira/browse/SPARK-61242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61242:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> DataType.from_ddl rejects NOT NULL and COMMENT of a field, and type_name of
> intervals is "interval"
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-61242
> URL: https://issues.apache.org/jira/browse/SPARK-61242
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> DataType::from_ddl rejects the NOT NULL and COMMENT clauses of a field, at
> top level ("a INT NOT NULL, b STRING COMMENT 'x'") and inside struct<...>,
> with INVALID_DATATYPE_FORMAT. So DataType::to_ddl output with a non-nullable
> or commented field does not parse back, while Spark's parser accepts it
> (nullable false, the comment in the field metadata under "comment").
> DataType::type_name returns "interval" for day-time and year-month intervals,
> where PySpark's typeName gives "daytimeinterval" and "yearmonthinterval". The
> UDT name is now the lower-cased class name, as UserDefinedType.typeName()
> gives it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
