MaxGekk commented on PR #56908:
URL: https://github.com/apache/spark/pull/56908#issuecomment-4853147553
Agreed with @Licht-T. date_trunc to a date-level unit should be a property
of the period - every timestamp in the same period must truncate to the same
instant, deterministic and offset-independent. A config flag keeps the
non-deterministic behavior as the default, and that default is exactly what
silently corrupts persisted partition/bucket/merge keys, so I'd rather fix it
than gate it. This shouldn't be opt-in.
The per-zone detection is the right shape: detect once whether a zone has
a midnight fall-back overlap and route only those zones' date-level truncations
through the slow path - deterministic everywhere, no config, and every normal
zone stays on the fast path at zero per-row cost. #56848's ZoneOffsetCache
(per-zone, per-task, already walking the zone's transitions) is the natural
home, so folding it in there as the follow-up sounds good to me.
@thepinetree would you be up for rebasing this onto #56848 once it lands, or
handing it to @Licht-T as the follow-up he offered?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
