This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c69bbf4 [SPARK-31959][SQL][3.0] Fix Gregorian-Julian micros rebasing
while switching standard time zone offset
c69bbf4 is described below
commit c69bbf4652f8ded5108826f38ae2316f31eb10e7
Author: Max Gekk <[email protected]>
AuthorDate: Tue Jun 16 04:56:30 2020 +0000
[SPARK-31959][SQL][3.0] Fix Gregorian-Julian micros rebasing while
switching standard time zone offset
### What changes were proposed in this pull request?
Fix the bug in microseconds rebasing during transitions from one standard
time zone offset to another one. In the PR, I propose to change the
implementation of `rebaseGregorianToJulianMicros` which performs rebasing via
local timestamps. In the case of overlapping:
1. Check that the original instant belongs to earlier or later instant of
overlapped local timestamp.
2. If it is an earlier instant, take zone and DST offsets from the previous
day otherwise
3. Set time zone offsets to Julian timestamp from the next day.
Note: The fix assumes that transitions cannot happen more often than once
per 2 days.
Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to
outdated tzdb. Old JDK can have outdated time zone database in which
Asia/Hong_Kong doesn't have timestamp overlapping in 1945 at all.
### Why are the changes needed?
1. Current implementation handles timestamps overlapping only during
daylight saving time but overlapping can happen also during transition from one
standard time zone to another one. For example in the case of `Asia/Hong_Kong`,
the time zone switched from `Japan Standard Time` (UTC+9) to `Hong Kong Time`
(UTC+8) on _Sunday, 18 November, 1945 01:59:59 AM_. The changes allow to handle
the special case as well.
2. To fix the test failures on old JDK w/ outdated tzdb like on Jenkins
machine `research-jenkins-worker-09`.
### Does this PR introduce _any_ user-facing change?
It might affect micros rebasing in before common era when not-optimised
version of `rebaseGregorianToJulianMicros()` is used directly.
### How was this patch tested?
1. By existing tests in `DateTimeUtilsSuite`, `RebaseDateTimeSuite`,
`DateFunctionsSuite`, `DateExpressionsSuite` and `TimestampFormatterSuite`.
2. Added new test to `RebaseDateTimeSuite`
3. Regenerated `gregorian-julian-rebase-micros.json` with the step of 30
minutes, and got the same JSON file. The JSON file isn't affected because
previously it was generated with the step of 1 week. And the spike in
diffs/switch points during 1 hour of timestamp overlapping wasn't detected.
Authored-by: Max Gekk <max.gekkgmail.com>
Signed-off-by: Wenchen Fan <wenchendatabricks.com>
(cherry picked from commit c259844)
Signed-off-by: Dongjoon Hyun <dongjoonapache.org>
(cherry picked from commit eae1747)
Signed-off-by: Max Gekk <max.gekkgmail.com>
Closes #28809 from MaxGekk/HongKong-tz-1945-3.0.
Authored-by: Max Gekk <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
.../spark/sql/catalyst/util/RebaseDateTime.scala | 26 +++++++++++----
.../sql/catalyst/util/RebaseDateTimeSuite.scala | 38 ++++++++++++++++++++++
2 files changed, 58 insertions(+), 6 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala
index e29fa4b..92d76c8 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala
@@ -326,20 +326,34 @@ object RebaseDateTime {
*/
private[sql] def rebaseGregorianToJulianMicros(zoneId: ZoneId, micros:
Long): Long = {
val instant = microsToInstant(micros)
- var ldt = instant.atZone(zoneId).toLocalDateTime
+ val zonedDateTime = instant.atZone(zoneId)
+ var ldt = zonedDateTime.toLocalDateTime
if (ldt.isAfter(julianEndTs) && ldt.isBefore(gregorianStartTs)) {
ldt = LocalDateTime.of(gregorianStartDate, ldt.toLocalTime)
}
val cal = new Calendar.Builder()
- // `gregory` is a hybrid calendar that supports both
- // the Julian and Gregorian calendar systems
+ // `gregory` is a hybrid calendar that supports both the Julian and
Gregorian calendar systems
.setCalendarType("gregory")
.setDate(ldt.getYear, ldt.getMonthValue - 1, ldt.getDayOfMonth)
.setTimeOfDay(ldt.getHour, ldt.getMinute, ldt.getSecond)
- // Local time-line can overlaps, such as at an autumn daylight savings
cutover.
- // This setting selects the original local timestamp mapped to the given
`micros`.
- .set(Calendar.DST_OFFSET,
zoneId.getRules.getDaylightSavings(instant).toMillis.toInt)
.build()
+ // A local timestamp can have 2 instants in the cases of switching from:
+ // 1. Summer to winter time.
+ // 2. One standard time zone to another one. For example, Asia/Hong_Kong
switched from JST
+ // to HKT on 18 November, 1945 01:59:59 AM.
+ // Below we check that the original `instant` is earlier or later instant.
If it is an earlier
+ // instant, we take the standard and DST offsets of the previous day
otherwise of the next one.
+ val trans = zoneId.getRules.getTransition(ldt)
+ if (trans != null && trans.isOverlap) {
+ val cloned = cal.clone().asInstanceOf[Calendar]
+ // Does the current offset belong to the offset before the transition.
+ // If so, we will take zone offsets from the previous day otherwise from
the next day.
+ // This assumes that transitions cannot happen often than once per 2
days.
+ val shift = if (trans.getOffsetBefore == zonedDateTime.getOffset) -1
else 1
+ cloned.add(Calendar.DAY_OF_MONTH, shift)
+ cal.set(Calendar.ZONE_OFFSET, cloned.get(Calendar.ZONE_OFFSET))
+ cal.set(Calendar.DST_OFFSET, cloned.get(Calendar.DST_OFFSET))
+ }
fromMillis(cal.getTimeInMillis) + ldt.get(ChronoField.MICRO_OF_SECOND)
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/RebaseDateTimeSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/RebaseDateTimeSuite.scala
index 0111fa0..6ecdb05 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/RebaseDateTimeSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/RebaseDateTimeSuite.scala
@@ -409,4 +409,42 @@ class RebaseDateTimeSuite extends SparkFunSuite with
Matchers with SQLHelper {
}
}
}
+
+ test("SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945") {
+ // The 'Asia/Hong_Kong' time zone switched from 'Japan Standard Time' (JST
= UTC+9)
+ // to 'Hong Kong Time' (HKT = UTC+8). After Sunday, 18 November, 1945
01:59:59 AM,
+ // clocks were moved backward to become Sunday, 18 November, 1945 01:00:00
AM.
+ // In this way, the overlap happened w/o Daylight Saving Time.
+ val hkZid = getZoneId("Asia/Hong_Kong")
+ withDefaultTimeZone(hkZid) {
+ var expected = "1945-11-18 01:30:00.0"
+ var ldt = LocalDateTime.of(1945, 11, 18, 1, 30, 0)
+ var earlierMicros =
instantToMicros(ldt.atZone(hkZid).withEarlierOffsetAtOverlap().toInstant)
+ var laterMicros =
instantToMicros(ldt.atZone(hkZid).withLaterOffsetAtOverlap().toInstant)
+ var overlapInterval = MICROS_PER_HOUR
+ if (earlierMicros + overlapInterval != laterMicros) {
+ // Old JDK might have an outdated time zone database.
+ // See https://bugs.openjdk.java.net/browse/JDK-8228469: "Hong Kong
... Its 1945 transition
+ // from JST to HKT was on 11-18 at 02:00, not 09-15 at 00:00"
+ expected = "1945-09-14 23:30:00.0"
+ ldt = LocalDateTime.of(1945, 9, 14, 23, 30, 0)
+ earlierMicros =
instantToMicros(ldt.atZone(hkZid).withEarlierOffsetAtOverlap().toInstant)
+ laterMicros =
instantToMicros(ldt.atZone(hkZid).withLaterOffsetAtOverlap().toInstant)
+ // If time zone db doesn't have overlapping at all, set the overlap
interval to zero.
+ overlapInterval = laterMicros - earlierMicros
+ }
+ val rebasedEarlierMicros = rebaseGregorianToJulianMicros(hkZid,
earlierMicros)
+ val rebasedLaterMicros = rebaseGregorianToJulianMicros(hkZid,
laterMicros)
+ def toTsStr(micros: Long): String = toJavaTimestamp(micros).toString
+ assert(toTsStr(rebasedEarlierMicros) === expected)
+ assert(toTsStr(rebasedLaterMicros) === expected)
+ assert(rebasedEarlierMicros + overlapInterval === rebasedLaterMicros)
+ // Check optimized rebasing
+ assert(rebaseGregorianToJulianMicros(earlierMicros) ===
rebasedEarlierMicros)
+ assert(rebaseGregorianToJulianMicros(laterMicros) === rebasedLaterMicros)
+ // Check reverse rebasing
+ assert(rebaseJulianToGregorianMicros(rebasedEarlierMicros) ===
earlierMicros)
+ assert(rebaseJulianToGregorianMicros(rebasedLaterMicros) === laterMicros)
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
