[
https://issues.apache.org/jira/browse/SPARK-61249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61249:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Raise for out-of-range createDataFrame integers and nanosecond timestamps in
> collect, as PySpark does
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-61249
> URL: https://issues.apache.org/jira/browse/SPARK-61249
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> createDataFrame of an integer that does not fit its declared type wrapped it
> silently in the Rust core: 300 into TINYINT gave 44, 4294967297 into INT gave
> 1. Collect of TIMESTAMP_NTZ(p) and TIMESTAMP_LTZ(p) with p > 6 dropped the
> sub-microsecond digits and truncated pre-1970 values toward zero: '1969-12-31
> 23:59:59.1234567' gave -876543 us instead of -876544 us.
> Repro: spark.createDataFrame([(300,)], "b TINYINT").collect() returned
> Row(b=44) on master; spark.sql("SELECT CAST('1969-12-31 23:59:59.1234567' AS
> TIMESTAMP_NTZ(7))").collect() returned the wrong microsecond value.
> Official PySpark 4.3.0 raises ArrowInvalid ("Value 300 too large to fit in C
> integer type") for the first, and PySparkValueError UNSUPPORTED_OPERATION
> ("data type timestamp_ntz_nanos {precision: 9} is not supported") for
> collect, first, head, take, tail, toLocalIterator, toPandas and toArrow of
> the second.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
