MaxGekk opened a new pull request #27612: [SPARK-30857][SQL][2.4] Fix truncations of timestamps before the epoch to minutes and seconds URL: https://github.com/apache/spark/pull/27612 ### What changes were proposed in this pull request? In the PR, I propose to replace `%` by `Math.floorMod` in `DateTimeUtils.truncTimestamp` for the `HOUR` and `DAY` levels. ### Why are the changes needed? This fixes the issue of incorrect truncation of timestamps before the epoch `1970-01-01T00:00:00.000000Z` to the `HOUR` and `DAY` levels. For example, timestamps after the epoch are truncated by cutting off the rest part of the timestamp: ```sql spark-sql> select date_trunc('HOUR', '2020-02-11 00:01:02.123'), date_trunc('HOUR', '2020-02-11 00:01:02.789'); 2020-02-11 00:00:00 2020-02-11 00:00:00 ``` but hours in the truncated timestamp before the epoch are increased by 1: ```sql spark-sql> select date_trunc('HOUR', '1960-02-11 00:01:02.123'), date_trunc('HOUR', '1960-02-11 00:01:02.789'); 1960-02-11 01:00:00 1960-02-11 01:00:00 ``` ### Does this PR introduce any user-facing change? Yes. After the changes, the example above outputs correct result: ```sql spark-sql> select date_trunc('HOUR', '1960-02-11 00:01:02.123'); 1960-02-11 00:00:00 ``` ### How was this patch tested? Added new tests to `DateFunctionsSuite`.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
