[
https://issues.apache.org/jira/browse/SPARK-57737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rito Takeuchi updated SPARK-57737:
----------------------------------
Description:
h2. Background
{{date_trunc}} (expression {{{}TruncTimestamp{}}}) truncates a timestamp in the
session time zone. For every level above {{SECOND}} it resolves the zone offset
for the row via {{{}ZoneRules.getOffset(Instant){}}}, a binary search over the
zone's transition array. For non-fixed-offset zones this runs {*}twice per
row{*}: once for the input instant and once for the candidate truncated instant
(the DST-equality guard from SPARK-56663 / SPARK-56769).
h2. Proposal
Add a per-task {{ZoneOffsetCache}} that memoizes the resolved offset over the
half-open epoch-second interval {{[lo, hi)}} on which it is provably constant,
derived from the surrounding zone transitions
({{{}nextTransition{}}}/{{{}previousTransition{}}}). A lookup that falls inside
the cached interval reduces to two comparisons instead of a binary search.
h2. Benchmark
{{DateTimeBenchmark}} Truncation, whole-stage codegen on, session zone
{{{}America/Los_Angeles{}}}, OpenJDK 17 on a 12th Gen Intel i7-1260P, ns/row
(lower is better):
||level||without cache||with cache||speedup||
|date_trunc YEAR|98.2|56.8|1.73x|
|date_trunc QUARTER|109.3|71.7|1.52x|
|date_trunc MONTH|90.8|53.7|1.69x|
|date_trunc WEEK|77.8|40.6|1.92x|
|date_trunc DAY|64.8|33.0|1.96x|
|date_trunc SECOND (control)|28.7|27.7|~1.0x|
was:
h2. Background
{{date_trunc}} (expression {{{}TruncTimestamp{}}}) truncates a timestamp in the
session
time zone. For every level above {{SECOND}} it resolves the zone offset for the
row via
{{{}ZoneRules.getOffset(Instant){}}}, a binary search over the zone's
transition array. For
non-fixed-offset zones this runs {*}twice per row{*}: once for the input
instant and once for
the candidate truncated instant (the DST-equality guard from SPARK-56663 /
SPARK-56769).
The session time zone is constant for the whole query, and a zone's offset is
piecewise-constant between DST/historical transitions. Analytic data is
typically
temporally clustered (time series, date-partitioned tables, post-sort), so
consecutive
rows almost always fall in the same constant-offset window. Re-running the
binary search
per row is therefore redundant work on the hot path.
h2. Proposal
Add a per-task {{ZoneOffsetCache}} that memoizes the resolved offset over the
half-open
epoch-second interval {{[lo, hi)}} on which it is provably constant, derived
from the
surrounding zone transitions
({{{}nextTransition{}}}/{{{}previousTransition{}}}). A lookup that
falls inside the cached interval reduces to two comparisons instead of a binary
search.
h2. Benchmark
{{DateTimeBenchmark}} Truncation, whole-stage codegen on, session zone
{{{}America/Los_Angeles{}}}, OpenJDK 17 on a 12th Gen Intel i7-1260P, ns/row
(lower is better):
||level||without cache||with cache||speedup||
|date_trunc YEAR|98.2|56.8|1.73x|
|date_trunc QUARTER|109.3|71.7|1.52x|
|date_trunc MONTH|90.8|53.7|1.69x|
|date_trunc WEEK|77.8|40.6|1.92x|
|date_trunc DAY|64.8|33.0|1.96x|
|date_trunc SECOND (control)|28.7|27.7|~1.0x|
> Cache zone offset per task in date_trunc
> ----------------------------------------
>
> Key: SPARK-57737
> URL: https://issues.apache.org/jira/browse/SPARK-57737
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Rito Takeuchi
> Priority: Major
>
> h2. Background
> {{date_trunc}} (expression {{{}TruncTimestamp{}}}) truncates a timestamp in
> the session time zone. For every level above {{SECOND}} it resolves the zone
> offset for the row via {{{}ZoneRules.getOffset(Instant){}}}, a binary search
> over the zone's transition array. For non-fixed-offset zones this runs
> {*}twice per row{*}: once for the input instant and once for the candidate
> truncated instant (the DST-equality guard from SPARK-56663 / SPARK-56769).
> h2. Proposal
> Add a per-task {{ZoneOffsetCache}} that memoizes the resolved offset over the
> half-open epoch-second interval {{[lo, hi)}} on which it is provably
> constant, derived from the surrounding zone transitions
> ({{{}nextTransition{}}}/{{{}previousTransition{}}}). A lookup that falls
> inside the cached interval reduces to two comparisons instead of a binary
> search.
> h2. Benchmark
> {{DateTimeBenchmark}} Truncation, whole-stage codegen on, session zone
> {{{}America/Los_Angeles{}}}, OpenJDK 17 on a 12th Gen Intel i7-1260P, ns/row
> (lower is better):
> ||level||without cache||with cache||speedup||
> |date_trunc YEAR|98.2|56.8|1.73x|
> |date_trunc QUARTER|109.3|71.7|1.52x|
> |date_trunc MONTH|90.8|53.7|1.69x|
> |date_trunc WEEK|77.8|40.6|1.92x|
> |date_trunc DAY|64.8|33.0|1.96x|
> |date_trunc SECOND (control)|28.7|27.7|~1.0x|
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
