Hyukjin Kwon created SPARK-60947:
------------------------------------
Summary: Send TIME values and parse the TIME type of DDL schemas
in Spark Connect Rust as PySpark does
Key: SPARK-60947
URL: https://issues.apache.org/jira/browse/SPARK-60947
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect's {{createDataFrame}} infers {{TimeType()}} for a
{{datetime.time}} value and sends it as an Arrow time, and it parses a DDL
schema on the server. Spark 4.3.0 does not support the TIME type in a local
relation, so the server then raises {{UNSUPPORTED_TIME_TYPE}}. The Rust-backed
Python client differs:
* {{createDataFrame([(datetime.time(1, 2, 3),)])}} raises {{TypeError:
unsupported value type for createDataFrame: time}}, rather than the server's
{{AnalysisException}} {{UNSUPPORTED_TIME_TYPE}}.
* {{createDataFrame([(datetime.time(1, 2, 3),)], "t time")}} raises
{{ValueError: invalid type string 't time'}}: the client parses a DDL string
itself, and does not know the {{time}} or {{time(p)}} type.
Send TIME values in Arrow, and parse {{time}} and {{time(p)}} in DDL strings,
as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
