neilconway opened a new pull request, #11187:
URL: https://github.com/apache/arrow-rs/pull/11187
# Which issue does this PR close?
- N/A
# Rationale for this change
`date_part` is implemented by converting the timestamp to a Chrono value and
then extracting the requested date/time component using Chrono. For timestamp
values without a timestamp, we can often do better: "hour" through "nanosecond"
values can be extracted by doing arithmetic directly on the timestamp value,
without needing to consider the calendar date represented by the timestamp.
This is substantially faster than going through Chrono.
This is partly motivated by improving the performance of `date_part` and
`extract` in DataFusion. For example, ClickBench Q18 improves by about 2% using
this optimization.
Benchmarks: (ARM64, elapsed times in microseconds)
- timestamp_s/Hour/no_nulls: 33.964 → 5.903, −82.6%
- timestamp_s/Hour/mixed_nulls: 31.586 → 5.909, −81.3%
- timestamp_s/Minute/no_nulls: 37.616 → 7.172, −80.9%
- timestamp_s/Minute/mixed_nulls: 33.373 → 7.153, −78.6%
- timestamp_s/Second/no_nulls: 35.562 → 2.568, −92.8%
- timestamp_s/Second/mixed_nulls: 32.921 → 2.581, −92.2%
- timestamp_s/Millisecond/no_nulls: 0.344 → 0.353, +2.7%
- timestamp_s/Millisecond/mixed_nulls: 0.342 → 0.354, +3.4%
- timestamp_s/Microsecond/no_nulls: 0.339 → 0.344, +1.6%
- timestamp_s/Microsecond/mixed_nulls: 0.345 → 0.346, +0.4%
- timestamp_s/Nanosecond/no_nulls: 0.348 → 0.349, +0.2%
- timestamp_s/Nanosecond/mixed_nulls: 0.351 → 0.350, −0.3%
- timestamp_s/Minute/timezone_control: 60.107 → 60.102, 0.0%
- timestamp_s/Nanosecond/timezone_control: 0.344 → 0.346, +0.6%
- timestamp_s/Year/control: 32.778 → 33.179, +1.2%
- timestamp_ms/Hour/no_nulls: 45.956 → 5.911, −87.1%
- timestamp_ms/Hour/mixed_nulls: 40.266 → 5.919, −85.3%
- timestamp_ms/Minute/no_nulls: 49.837 → 6.305, −87.3%
- timestamp_ms/Minute/mixed_nulls: 41.962 → 6.265, −85.1%
- timestamp_ms/Second/no_nulls: 48.070 → 6.271, −87.0%
- timestamp_ms/Second/mixed_nulls: 41.210 → 6.302, −84.7%
- timestamp_ms/Millisecond/no_nulls: 45.229 → 2.333, −94.8%
- timestamp_ms/Millisecond/mixed_nulls: 39.643 → 2.345, −94.1%
- timestamp_ms/Microsecond/no_nulls: 45.364 → 2.490, −94.5%
- timestamp_ms/Microsecond/mixed_nulls: 40.163 → 2.492, −93.8%
- timestamp_ms/Nanosecond/no_nulls: 43.847 → 2.390, −94.5%
- timestamp_ms/Nanosecond/mixed_nulls: 39.255 → 2.394, −93.9%
- timestamp_ms/Minute/timezone_control: 74.756 → 74.833, +0.1%
- timestamp_ms/Nanosecond/timezone_control: 67.016 → 66.269, −1.1%
- timestamp_ms/Year/control: 45.080 → 45.166, +0.2%
- timestamp_us/Hour/no_nulls: 41.860 → 6.539, −84.4%
- timestamp_us/Hour/mixed_nulls: 38.822 → 6.408, −83.5%
- timestamp_us/Minute/no_nulls: 45.357 → 6.853, −84.9%
- timestamp_us/Minute/mixed_nulls: 40.589 → 6.743, −83.4%
- timestamp_us/Second/no_nulls: 42.589 → 6.246, −85.3%
- timestamp_us/Second/mixed_nulls: 39.346 → 6.205, −84.2%
- timestamp_us/Millisecond/no_nulls: 39.906 → 2.650, −93.4%
- timestamp_us/Millisecond/mixed_nulls: 38.902 → 2.631, −93.2%
- timestamp_us/Microsecond/no_nulls: 39.104 → 2.324, −94.1%
- timestamp_us/Microsecond/mixed_nulls: 38.804 → 2.329, −94.0%
- timestamp_us/Nanosecond/no_nulls: 38.297 → 2.380, −93.8%
- timestamp_us/Nanosecond/mixed_nulls: 37.745 → 2.392, −93.7%
- timestamp_us/Minute/timezone_control: 67.390 → 65.911, −2.2%
- timestamp_us/Nanosecond/timezone_control: 60.115 → 59.787, −0.5%
- timestamp_us/Year/control: 38.823 → 38.556, −0.7%
- timestamp_ns/Hour/no_nulls: 39.858 → 3.565, −91.1%
- timestamp_ns/Hour/mixed_nulls: 38.831 → 3.578, −90.8%
- timestamp_ns/Minute/no_nulls: 43.810 → 3.411, −92.2%
- timestamp_ns/Minute/mixed_nulls: 40.649 → 3.418, −91.6%
- timestamp_ns/Second/no_nulls: 42.057 → 6.246, −85.1%
- timestamp_ns/Second/mixed_nulls: 40.125 → 6.280, −84.3%
- timestamp_ns/Millisecond/no_nulls: 39.049 → 2.645, −93.2%
- timestamp_ns/Millisecond/mixed_nulls: 38.840 → 2.650, −93.2%
- timestamp_ns/Microsecond/no_nulls: 39.130 → 2.660, −93.2%
- timestamp_ns/Microsecond/mixed_nulls: 38.776 → 2.652, −93.2%
- timestamp_ns/Nanosecond/no_nulls: 38.303 → 2.326, −93.9%
- timestamp_ns/Nanosecond/mixed_nulls: 37.715 → 2.323, −93.8%
- timestamp_ns/Minute/timezone_control: 67.940 → 67.854, −0.1%
- timestamp_ns/Nanosecond/timezone_control: 61.504 → 60.833, −1.1%
- timestamp_ns/Year/control: 39.344 → 39.668, +0.8%
# What changes are included in this PR?
* Optimize `date_part` as described above
* Add unit tests
* Add benchmark for `date_part`
# Are these changes tested?
Existing tests pass. Two new tests were added: one checks the consistency of
this code path with the results produced by Chrono; the second checks the
results of this code path for extreme values (outside of Chrono's supported
calendar range).
# Are there any user-facing changes?
There is one user-visible behavior change. Chrono has a limited calendar
range (roughly +/- 262,000 years). Calling `date_part` on a timestamp outside
that range returned null. In this implementation, `date_part` will succeed for
the entire range of timestamp values. The returned value is correct, but this
behavior is slightly inconsistent with `date_part` for units that are still
implemented via Chrono, and for timestamp values with time zones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
