Hyukjin Kwon created SPARK-61317:
------------------------------------
Summary: Raise for nanosecond timestamps on every schema surface,
as PySpark does
Key: SPARK-61317
URL: https://issues.apache.org/jira/browse/SPARK-61317
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The drop-in raises UNSUPPORTED_OPERATION for a nanosecond-precision timestamp
(TIMESTAMP_NTZ(p) or TIMESTAMP_LTZ(p), p > 6) on the collecting APIs, as
PySpark does, but it still returns the schema of such a frame and parses such a
DDL string. Official PySpark 4.3.0 cannot convert these types from the proto,
so the schema, columns, dtypes, printSchema, repr, schema.json(),
DataType/StructType.fromDDL, createDataFrame with a DDL schema, from_json with
a DDL schema and the StructType.add of the type all raise.
Repro: spark.sql("SELECT CAST('2020-01-01 00:00:00.123456789' AS
TIMESTAMP_NTZ(9)) c").schema returns struct<c:timestamp_ntz(7)> (official:
PySparkValueError UNSUPPORTED_OPERATION), and spark.createDataFrame([(1,)], "a
timestamp_ntz(9)") raises a bare AssertionError.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
