[
https://issues.apache.org/jira/browse/SPARK-61317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61317:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Raise for nanosecond timestamps on every schema surface, as PySpark does
> ------------------------------------------------------------------------
>
> Key: SPARK-61317
> URL: https://issues.apache.org/jira/browse/SPARK-61317
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The drop-in raises UNSUPPORTED_OPERATION for a nanosecond-precision timestamp
> (TIMESTAMP_NTZ(p) or TIMESTAMP_LTZ(p), p > 6) on the collecting APIs, as
> PySpark does, but it still returns the schema of such a frame and parses such
> a DDL string. Official PySpark 4.3.0 cannot convert these types from the
> proto, so the schema, columns, dtypes, printSchema, repr, schema.json(),
> DataType/StructType.fromDDL, createDataFrame with a DDL schema, from_json
> with a DDL schema and the StructType.add of the type all raise.
> Repro: spark.sql("SELECT CAST('2020-01-01 00:00:00.123456789' AS
> TIMESTAMP_NTZ(9)) c").schema returns struct<c:timestamp_ntz(7)> (official:
> PySparkValueError UNSUPPORTED_OPERATION), and spark.createDataFrame([(1,)],
> "a timestamp_ntz(9)") raises a bare AssertionError.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
