[
https://issues.apache.org/jira/browse/SPARK-60947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60947:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Send TIME values and parse the TIME type of DDL schemas in Spark Connect Rust
> as PySpark does
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-60947
> URL: https://issues.apache.org/jira/browse/SPARK-60947
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect's {{createDataFrame}} infers {{TimeType()}} for a
> {{datetime.time}} value and sends it as an Arrow time, and it parses a DDL
> schema on the server. Spark 4.3.0 does not support the TIME type in a local
> relation, so the server then raises {{UNSUPPORTED_TIME_TYPE}}. The
> Rust-backed Python client differs:
> * {{createDataFrame([(datetime.time(1, 2, 3),)])}} raises {{TypeError:
> unsupported value type for createDataFrame: time}}, rather than the server's
> {{AnalysisException}} {{UNSUPPORTED_TIME_TYPE}}.
> * {{createDataFrame([(datetime.time(1, 2, 3),)], "t time")}} raises
> {{ValueError: invalid type string 't time'}}: the client parses a DDL string
> itself, and does not know the {{time}} or {{time(p)}} type.
> Send TIME values in Arrow, and parse {{time}} and {{time(p)}} in DDL strings,
> as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
