neilconway opened a new pull request, #11226:
URL: https://github.com/apache/arrow-rs/pull/11226
# Which issue does this PR close?
- N/A
# Rationale for this change
Casting integers to decimals currently uses `unary_opt` (in safe mode) or
`try_unary` (otherwise). This is relatively slow, because neither kernel
vectorizes well.
In practice, `int` -> `decimal` casts are frequently infallible, either
because the target type can accommodate any integer, or because the target type
is chosen to be large enough to accommodate all inputs in the source data. We
can exploit this: we do a branch-free (vectorized) scan of the source data to
check if all of the values will fit into the target type. If so, we can use
`unary` to do the cast. Empirically, the cost of doing the branch-free scan is
much smaller than the win from using the faster kernel (see benchmarks below).
We make sure to use chunking and early-stopping when scanning for
out-of-range inputs: for many input distributions, if there is an out of range
value it will likely appear in the first few chunks, so we don't need to scan
the entire input anyway.
Benchmarks: (M4 Max)
- cast int64 to decimal128(38, 10) 512: 994.360 → 300.580 ns, −69.8%
- cast int64 within 1e6 to decimal128(18, 2) 512: 989.400 → 315.160 ns,
−68.1%
- cast int64 within 1e6 late overflow to decimal128(18, 2) 512: 992.870 →
1060.300 ns, +6.8%
- cast int64 to decimal32(9, 0) 512: 738.400 → 781.610 ns, +5.9%
- cast int64 to decimal32(9, -1) 512: 957.480 → 991.820 ns, +3.6%
In the first case, no input scan is needed because the target type is large
enough to hold any `int64`. In the second case, we need to scan the inputs to
establish that no out-of-range value is found. The third benchmark is the
worst-case, where an out-of-range value is found at the end of the input. The
last two cases are existing benchmarks where an out-of-range value is found
early in the scan.
# What changes are included in this PR?
* Add `integers_fit_digits`, which does a chunked, branch-free scan to check
for out-of-range values
* Use `unary` in `cast_integer_to_decimal`, when the cast is infallible
* Add unit test
* Extend benchmarks
# Are these changes tested?
Yes; existing tests pass and new test added.
# Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
