Hyukjin Kwon created SPARK-61249:
------------------------------------
Summary: Raise for out-of-range createDataFrame integers and
nanosecond timestamps in collect, as PySpark does
Key: SPARK-61249
URL: https://issues.apache.org/jira/browse/SPARK-61249
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
createDataFrame of an integer that does not fit its declared type wrapped it
silently in the Rust core: 300 into TINYINT gave 44, 4294967297 into INT gave
1. Collect of TIMESTAMP_NTZ(p) and TIMESTAMP_LTZ(p) with p > 6 dropped the
sub-microsecond digits and truncated pre-1970 values toward zero: '1969-12-31
23:59:59.1234567' gave -876543 us instead of -876544 us.
Repro: spark.createDataFrame([(300,)], "b TINYINT").collect() returned
Row(b=44) on master; spark.sql("SELECT CAST('1969-12-31 23:59:59.1234567' AS
TIMESTAMP_NTZ(7))").collect() returned the wrong microsecond value.
Official PySpark 4.3.0 raises ArrowInvalid ("Value 300 too large to fit in C
integer type") for the first, and PySparkValueError UNSUPPORTED_OPERATION
("data type timestamp_ntz_nanos {precision: 9} is not supported") for collect,
first, head, take, tail, toLocalIterator, toPandas and toArrow of the second.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
