thepinetree opened a new pull request, #56908:
URL: https://github.com/apache/spark/pull/56908
### What changes were proposed in this pull request?
`date_trunc` to a date-level unit (`WEEK` / `MONTH` / `QUARTER` / `YEAR`)
uses an
offset-arithmetic fast path (added in SPARK-56769, extending SPARK-56663).
The fast
path resolves the truncated local-midnight boundary back to UTC using the
offset of
the *source* timestamp. At a daylight-saving fall-back transition, that
local midnight
occurs twice (once before and once after the clocks go back), so the source
offset is
not necessarily the offset the slow-path reference would pick: the slow path
(`daysToMicros`) resolves the midnight with the *earliest* valid offset.
Both instants are valid representations of the overlapped midnight, so this
is not a
wrong result. But it does make `date_trunc` depend on the source timestamp's
offset:
two timestamps in the same period can truncate to different instants when
one of them
sits on the overlap, so `GROUP BY date_trunc(...)` may place them in
different groups.
Some workloads prefer the offset-independent (earliest) resolution, which is
deterministic per period.
Concrete example (session time zone `Europe/Berlin`; Germany's first DST
ended
1916-10-01 01:00 CEST -> 00:00 CET, so local `1916-10-01 00:00` exists at
both +02:00
and +01:00):
- `date_trunc('MONTH', TIMESTAMP '1916-10-15 12:00:00')` (source offset
+01:00 CET):
fast path resolves to `1916-09-30 23:00:00Z`, while the slow-path
reference resolves
to `1916-09-30 22:00:00Z` (earliest, +02:00 CEST).
- A source sitting in the overlap, `1916-10-01 00:30 +02:00`, truncates to
`1916-09-30 22:00:00Z` under both paths, so the two October timestamps
disagree.
`Atlantic/Azores` 1912-01-01 (LMT -01:54:32 -> -02:00) is another instance,
a 328s
difference.
This PR adds a kill-switch
`spark.sql.legacy.timestampTruncateToOverlapEarliestOffset`
(default `false`). When `false` (the default), the behavior is unchanged:
the fast
path reuses the source offset. When `true`, date-level truncations are
routed through
the slow path so the earliest valid offset is always chosen, which is
independent of
the source offset and therefore deterministic per period. The flag only
affects
date-level truncations whose midnight boundary coincides with a fall-back
overlap; all
other truncations (and `MIN`/`HOUR`/`DAY`, whose slow path already keeps the
source
offset) are unchanged.
The fast path remains the default because it is the existing
post-SPARK-56769 behavior
and is significantly faster; the determinism trade-off is documented in the
config and
in the `truncTimestamp` scaladoc so users hitting the overlap edge case can
opt into
the earliest-offset (deterministic) semantics.
### Why are the changes needed?
The fast path silently diverges from its slow-path reference at a
daylight-saving
fall-back overlap, making `date_trunc` non-deterministic per `(period,
zone)` and
breaking grouped aggregations. The flag gives a documented escape hatch to
the
offset-independent (deterministic) result.
### Does this PR introduce _any_ user-facing change?
Yes. A new internal SQL config
`spark.sql.legacy.timestampTruncateToOverlapEarliestOffset` (default
`false`) is added.
With the default value, behavior is unchanged. Setting it to `true` changes
the result
of `date_trunc` for date-level units only when the truncated boundary lands
exactly on
a daylight-saving fall-back overlap, selecting the earliest valid offset (the
pre-SPARK-56769 / slow-path result) so that the result no longer depends on
the source
timestamp's offset.
### How was this patch tested?
New unit test in `DateTimeUtilsSuite` (`truncTimestamp date-level units at a
fall-back
overlap`) that asserts both config modes for the `Europe/Berlin` 1916 and
`Atlantic/Azores` 1912 overlaps, and asserts the determinism property (two
sources in
the same period truncate to the same instant under the legacy mode and to
different
instants under the default). The full `DateTimeUtilsSuite` passes.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
