[
https://issues.apache.org/jira/browse/SPARK-57737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rito Takeuchi updated SPARK-57737:
----------------------------------
Description:
h2. Background
{{date_trunc}} (expression {{{}TruncTimestamp{}}}) truncates a timestamp in the
session
time zone. For every level above {{SECOND}} it resolves the zone offset for the
row via
{{{}ZoneRules.getOffset(Instant){}}}, a binary search over the zone's
transition array. For
non-fixed-offset zones this runs {*}twice per row{*}: once for the input
instant and once for
the candidate truncated instant (the DST-equality guard from SPARK-56663 /
SPARK-56769).
The session time zone is constant for the whole query, and a zone's offset is
piecewise-constant between DST/historical transitions. Analytic data is
typically
temporally clustered (time series, date-partitioned tables, post-sort), so
consecutive
rows almost always fall in the same constant-offset window. Re-running the
binary search
per row is therefore redundant work on the hot path.
h2. Proposal
Add a per-task {{ZoneOffsetCache}} that memoizes the resolved offset over the
half-open
epoch-second interval {{[lo, hi)}} on which it is provably constant, derived
from the
surrounding zone transitions
({{{}nextTransition{}}}/{{{}previousTransition{}}}). A lookup that
falls inside the cached interval reduces to two comparisons instead of a binary
search.
h2. Benchmark
{{DateTimeBenchmark}} Truncation, whole-stage codegen on, session zone
{{{}America/Los_Angeles{}}}, OpenJDK 17 on a 12th Gen Intel i7-1260P, ns/row
(lower is better):
||level||without cache||with cache||speedup||
|date_trunc YEAR|98.2|56.8|1.73x|
|date_trunc QUARTER|109.3|71.7|1.52x|
|date_trunc MONTH|90.8|53.7|1.69x|
|date_trunc WEEK|77.8|40.6|1.92x|
|date_trunc DAY|64.8|33.0|1.96x|
|date_trunc SECOND (control)|28.7|27.7|~1.0x|
> Cache zone offset per task in date_trunc
> ----------------------------------------
>
> Key: SPARK-57737
> URL: https://issues.apache.org/jira/browse/SPARK-57737
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Rito Takeuchi
> Priority: Major
>
> h2. Background
> {{date_trunc}} (expression {{{}TruncTimestamp{}}}) truncates a timestamp in
> the session
> time zone. For every level above {{SECOND}} it resolves the zone offset for
> the row via
> {{{}ZoneRules.getOffset(Instant){}}}, a binary search over the zone's
> transition array. For
> non-fixed-offset zones this runs {*}twice per row{*}: once for the input
> instant and once for
> the candidate truncated instant (the DST-equality guard from SPARK-56663 /
> SPARK-56769).
> The session time zone is constant for the whole query, and a zone's offset is
> piecewise-constant between DST/historical transitions. Analytic data is
> typically
> temporally clustered (time series, date-partitioned tables, post-sort), so
> consecutive
> rows almost always fall in the same constant-offset window. Re-running the
> binary search
> per row is therefore redundant work on the hot path.
> h2. Proposal
> Add a per-task {{ZoneOffsetCache}} that memoizes the resolved offset over the
> half-open
> epoch-second interval {{[lo, hi)}} on which it is provably constant, derived
> from the
> surrounding zone transitions
> ({{{}nextTransition{}}}/{{{}previousTransition{}}}). A lookup that
> falls inside the cached interval reduces to two comparisons instead of a
> binary search.
> h2. Benchmark
> {{DateTimeBenchmark}} Truncation, whole-stage codegen on, session zone
> {{{}America/Los_Angeles{}}}, OpenJDK 17 on a 12th Gen Intel i7-1260P, ns/row
> (lower is better):
> ||level||without cache||with cache||speedup||
> |date_trunc YEAR|98.2|56.8|1.73x|
> |date_trunc QUARTER|109.3|71.7|1.52x|
> |date_trunc MONTH|90.8|53.7|1.69x|
> |date_trunc WEEK|77.8|40.6|1.92x|
> |date_trunc DAY|64.8|33.0|1.96x|
> |date_trunc SECOND (control)|28.7|27.7|~1.0x|
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
