dongjoon-hyun commented on PR #57681:
URL: https://github.com/apache/spark/pull/57681#issuecomment-5170245400
The three-zone e2e matrix (UTC / LA / JST under `withDefaultTimeZone`)
validates the LTZ literal
frame for timestamps away from DST transitions — including UTC-ahead JST,
which would have failed
`checkAnswer` if the literal were in the wrong frame wholesale. However,
there is one untested
~1-hour-per-year window where the mapping is non-monotonic:
`Timestamp.valueOf` shifts any wall
clock that falls into the JVM zone's spring-forward gap one hour ahead,
colliding with (and
inverting against) the hour that follows. With the JVM in
`America/Los_Angeles`, instants whose
UTC wall clock lies in `2020-03-08 02:00..03:00` hit that gap when
re-interpreted in LA:
`2020-03-08T02:30Z` yields a literal epoch of local `03:30`, while the later
instant
`2020-03-08T03:15Z` yields local `03:15`, so instant order and literal order
disagree. If ORC's
stored-value shift for `TIMESTAMP_INSTANT` statistics does not reproduce
exactly the same gap
behavior, a `ts < lit` boundary inside that window could prune a stripe
containing rows the
residual filter would keep. (NTZ is not affected this way: its literal
mapping is identical to the
write-path mapping, so pruning stays consistent with what the scan returns.)
Could you either confirm the ORC-side shift matches `Timestamp.valueOf`
gap-for-gap, or extend the
e2e test with a case under `withDefaultTimeZone(LA)` whose data and boundary
straddle such a window
(e.g. wall clocks `2020-03-08T01:30..03:30` with the boundary at `02:45`)?
Two smaller notes:
- An `In` pushdown case for the nanos types would cover the per-value
`castLiteralValue` path in
`buildLeafSearchArgument` (`values.map(...)`); it is a one-line addition
to the new
`OrcFilterSuite` test.
- The existing micros `TimestampNTZType` path pushes a `LONG` leaf against
ORC timestamp
statistics, which likely never prunes (type mismatch evaluates to
`YES_NO_NULL`). The
`TIMESTAMP`-leaf approach taken here looks like the right template to
align it with in a
follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
