[
https://issues.apache.org/jira/browse/ORC-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raviston Thanasekar updated ORC-2236:
-------------------------------------
Attachment: OrcDstRepro.java
> Predicate push down skips row groups with TIMESTAMP values in the repeated
> hour at the end of daylight saving time
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ORC-2236
> URL: https://issues.apache.org/jira/browse/ORC-2236
> Project: ORC
> Issue Type: Bug
> Components: Java
> Affects Versions: 2.3.1
> Reporter: Raviston Thanasekar
> Priority: Major
> Attachments: OrcDstRepro.java
>
>
> With useUTCTimestamp=false (the default), a SearchArgument on a TIMESTAMP
> column skips a row group whose values match, when a value is in the first
> occurrence of the hour that repeats at the end of daylight saving time in the
> JVM default time zone. With orc-core 2.3.1, the attached OrcDstRepro.java
> writes one value per file with the Java API and reads it back with and
> without {{t = value}}:
> {noformat}
> JVM time zone America/Los_Angeles
> 2020-11-01T01:30:00-07:00: value 2020-11-01 01:30:00.0 (1604219400000), file
> min 2020-11-01 01:30:00.0 (1604223000000), rows 1, rows with t = value: 0
> <-- row group skipped
> 2020-11-01T01:30:00-08:00: value 2020-11-01 01:30:00.0 (1604223000000), file
> min 2020-11-01 01:30:00.0 (1604223000000), rows 1, rows with t = value: 1
> 2020-07-01T12:00:00-07:00: value 2020-07-01 12:00:00.0 (1593630000000), file
> min 2020-07-01 12:00:00.0 (1593630000000), rows 1, rows with t = value: 1
> JVM time zone Europe/Berlin
> 2024-10-27T02:30:00+02:00: value 2024-10-27 02:30:00.0 (1729989000000), file
> min 2024-10-27 02:30:00.0 (1729992600000), rows 1, rows with t = value: 0
> <-- row group skipped
> 2024-10-27T02:30:00+01:00: value 2024-10-27 02:30:00.0 (1729992600000), file
> min 2024-10-27 02:30:00.0 (1729992600000), rows 1, rows with t = value: 1
> JVM time zone Europe/Madrid
> 1900-01-01T00:00:00-00:14:44: value 1900-01-01 00:00:00.0 (-2208987916000),
> file min 1900-01-01 00:00:00.0 (-2208992400000), rows 1, rows with t = value:
> 0 <-- row group skipped
> 1900-01-02T00:00:00-00:14:44: value 1900-01-02 00:00:00.0 (-2208901516000),
> file min 1900-01-02 00:00:00.0 (-2208901516000), rows 1, rows with t = value:
> 1
> {noformat}
> The writer stores the statistic as the wall clock of the value, millis +
> offset ({{SerializationUtils.convertToUtc}} in {{TimestampTreeWriter}}).
> {{TimestampStatisticsImpl.getMinimum()}} and {{getMaximum()}} convert it back
> with {{SerializationUtils.convertFromUtc}}, which looks up the offset at
> {{time - rawOffset}}:
> {code:java}
> public static long convertFromUtc(TimeZone local, long time) {
> int offset = local.getOffset(time - local.getRawOffset());
> return time - offset;
> }
> {code}
> That isn't the offset of the value when the offset at that instant differs
> from the current raw offset:
> * A wall clock in the repeated hour belongs to two instants.
> {{convertFromUtc}} always returns the second one, so the minimum and maximum
> of a row group with a value in the first one are one hour later than the
> value.
> * Before 1900, {{java.util.TimeZone}} returns the current raw offset instead
> of the historical one. A wall clock near 1900-01-01 00:00 is then converted
> with a different offset than it was written with, in time zones such as
> Europe/Madrid, CET, Europe/Brussels, Africa/Ceuta, Antarctica/Casey and
> Pacific/Kiritimati.
> As the wall clock alone can't tell the two instants apart, a fix probably
> needs the minimum to use the earliest instant with that wall clock and the
> maximum the latest one, so that the range still covers the values.
> This was found through Spark, where it makes queries on ORC tables return
> fewer rows with filter push down on: SPARK-61463. ORC-451 looks related.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
