[
https://issues.apache.org/jira/browse/SPARK-60910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138893#comment-18138893
]
sepuri sai krishna commented on SPARK-60910:
--------------------------------------------
The PR fixes {{datediff}} / {{date_diff}} (silent wrap-around) and makes date
subtraction ({{d1 - d2}}) fail with {{DATETIME_OVERFLOW}} instead of a raw
{{java.lang.ArithmeticException}}.
{{timestampdiff}} and {{months_between}} are left out: on these dates they fail
while implicitly casting the {{DATE}} to {{TIMESTAMP}} ({{daysToMicros}}),
which happens for any date outside the timestamp range, so that is a separate
issue in the cast.
> datediff silently wraps around on integer overflow
> --------------------------------------------------
>
> Key: SPARK-60910
> URL: https://issues.apache.org/jira/browse/SPARK-60910
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.3, 5.0.0, 4.4.0
> Reporter: sepuri sai krishna
> Priority: Major
> Labels: Correctness, pull-request-available
>
> {{datediff}} / {{date_diff}} subtract the two day counts as an {{INT}}, so a
> difference that does
> not fit in an {{INT}} silently wraps around, with ANSI mode on or off and in
> both interpreted and
> generated code. On 4.0.3 and master, on values read from a table:
> {code:sql}
> CREATE TABLE dd USING parquet AS
> SELECT date_from_unix_date(2147483647) AS e, date_from_unix_date(-1) AS s,
> date_from_unix_date(-2147483648) AS mn;
> SELECT datediff(e, s) FROM dd; -- -2147483648, should be 2147483648
> SELECT datediff(e, mn) FROM dd; -- -1, should be 4294967295
> {code}
> On the same values, the related operations fail with a raw
> {{java.lang.ArithmeticException}}
> instead of an error class:
> {code:sql}
> SELECT e - s FROM dd; -- java.lang.ArithmeticException:
> integer overflow
> SELECT timestampdiff(DAY, e, s) FROM dd; -- java.lang.ArithmeticException:
> long overflow
> SELECT months_between(e, s) FROM dd; -- java.lang.ArithmeticException:
> long overflow
> {code}
> {{DateDiff.nullSafeEval}} is {{end - start}} on {{Int}} (same in codegen).
> This is the same class
> of bug as SPARK-59859 ({{date_add}} / {{date_sub}}), and was raised in the
> review of
> https://github.com/apache/spark/pull/59135.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
