MaxGekk opened a new pull request, #59955: URL: https://github.com/apache/spark/pull/59955
### What changes were proposed in this pull request? `make_timestamp`, `make_timestamp_ltz` and `make_timestamp_ntz` (the `MakeTimestamp` expression) build their result in `DateTimeExpressionUtils.makeTimestampMicros`. That method narrowed the unscaled seconds value, in microseconds, to an `int`: ```java // 8 digits cannot overflow Int. int totalMicros = (int) secAndMicros.toUnscaledLong(); ``` The comment dates from when the `sec` input was `DecimalType(8, 6)`. SPARK-36428 widened it to `DecimalType(16, 6)` so that integer seconds are accepted, and from then on the unscaled value can reach about 10^16, so the cast drops the high bits. This PR keeps the microseconds and the whole seconds as `long`s. Seconds that do not fit an `int` cannot be valid, so they are rejected with `ChronoField.SECOND_OF_MINUTE.checkValidValue` before narrowing. Every other value goes to `LocalDateTime.of` exactly as before, so an out-of-range value such as 4324 seconds takes the same path, and gets the same error, as 61 seconds. The PostgreSQL-compatible case of exactly 60 seconds, which rolls over to the next minute, is unchanged. Both the interpreted and the code-generated paths call this method, so both are fixed. ### Why are the changes needed? A seconds value 2^32 microseconds (4294.967296 seconds) away from a valid one wraps into that valid value and returns a wrong timestamp with no error, even with ANSI mode on: ```sql SELECT make_timestamp_ntz(2020, 1, 1, 0, 0, 4324.967296); -- 2020-01-01 00:00:30 (4324967296 - 2^32 = 30000000 micros) SELECT make_timestamp_ntz(2019, 6, 30, 23, 59, 4354.967296); -- 2019-07-01 00:00:00 (wraps to exactly 60 seconds and rolls over) SELECT make_timestamp_ntz(2020, 1, 1, 0, 0, -4264.967296); -- 2020-01-01 00:00:30 ``` A value of 61 seconds gives NULL with ANSI off and a `DATETIME_FIELD_OUT_OF_BOUNDS` error with ANSI on, and these values should too. Values whose wrapped result is still invalid, for example 3000 seconds, already fail, but the error reports the wrapped number (`Invalid value for SecondOfMinute (valid values 0 - 59): -1295`) rather than the input. ### Does this PR introduce _any_ user-facing change? Yes, a bug fix. The six- and seven-argument forms of `make_timestamp`, `make_timestamp_ltz` and `make_timestamp_ntz`, with seconds above 2147.483647 or below -2147.483648, no longer return a wrapped timestamp. They now return NULL when ANSI mode is off and raise `DATETIME_FIELD_OUT_OF_BOUNDS.WITHOUT_SUGGESTION` when it is on, as 61 seconds does. `try_make_timestamp*` returns NULL for them. The error message now shows the seconds actually given. Results for valid seconds, including the 60-second rollover, do not change. The bug has been present since Spark 3.2.0. ### How was this patch tested? Added a test to `DateExpressionsSuite`. It checks six seconds values against `TimestampType` and `TimestampNTZType`, with ANSI on and off: 4324.967296 (wraps to 30 s), 4354.967296 (wraps to 60 s and rolls over), 3000 (wraps to a negative int), -4264.967296 (a negative value that wraps to 30 s), 9999999999.999999 (the largest `DecimalType(16, 6)` value, whose whole seconds do not fit an int) and 61 as the reference. With ANSI on it expects `DATETIME_FIELD_OUT_OF_BOUNDS.WITHOUT_SUGGESTION` with the input seconds in the message, and with ANSI off it expects NULL. `checkErrorInExpression` and `checkEvaluation` run both the interpreted and the code-generated path. Before the fix the test failed: the first case returned `2020-01-01 00:00:30` with ANSI on. ``` build/sbt 'catalyst/testOnly *DateExpressionsSuite' build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z timestamp -z datetime-special' ``` Both pass. The golden files are unchanged. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
