comphead opened a new pull request, #12342:
URL: https://github.com/apache/arrow-rs/pull/12342

   # Which issue does this PR close?
   
   - Closes #12326.
   
   # Rationale for this change
   
   The `DecimalType::is_valid_decimal_precision` implementations are not 
`#[inline]`, so a caller in another crate calls them out of line for every 
value unless it builds with cross-crate LTO. The arrow-rs workspace does not, 
and arrow-cast's decimal casts reach them through 262 out-of-line calls in the 
`cast_kernels` bench binary. #12326 has the disassembly and the DataFusion 
Comet case.
   
   Adding `#[inline]` alone makes eight decimal casts 5% to 37% faster but 
three slower. Profiles of the slower ones with 
[samply](https://github.com/mstange/samply) show two causes, both in the 
arrow-cast kernels rather than in the check:
   
   - `cast decimal256 to decimal128 512` (+88%). With the check inlined into 
the per-row closure, LLVM compiles the rounding step of `make_downscaler` (`d + 
1` when `r >= half`) as a branch on the remainder instead of a `csel`. The two 
targets of that branch take 51.5% of all samples, which is what mispredictions 
look like. The bench values are random, so about half the rows round up.
   - `cast int64 to decimal32(9, 0) 512` (+24%) and `cast decimal256 to 
decimal256 512` (+8%). The larger closure is no longer inlined into the two 
loops of `try_for_each_valid_idx`, so every row calls it out of line and 
reloads its captures through the closure environment. Both closures also handle 
an `ArrowError` from `mul_checked` and then discard it with `.ok()`. #11529 
describes the same mechanism in arrow-arith.
   
   # What changes are included in this PR?
   
   - `#[inline]` on the four `is_valid_decimal_precision` implementations.
   - `make_downscaler` keeps the predictable sign branch and rounds with 
`std::hint::select_unpredictable`, stable since Rust 1.88, the MSRV.
   - A provided method, `DecimalCast::mul_checked_opt`, returns `None` on 
overflow without building an error. The four implementations use `checked_mul`. 
`make_upscaler` and the safe integer-to-decimal cast use it instead of 
`mul_checked(..).ok()`, which brings their closures back under LLVM's inline 
threshold.
   
   I did not change `ArrowNativeTypeOp` itself. #11529 found that moving its 
error construction into a `#[cold]` function slows integer division with nulls 
by 6% to 10%.
   
   # Are these changes tested?
   
   The behavior does not change, and existing tests cover the rewritten paths: 
`test_rescale_decimal_downscale_rounds_half_away_from_zero` for rounding of 
positive and negative values, `test_cast_numeric_to_decimal128_overflow` for 
overflow of the integer-to-decimal multiply with `safe: true`, and the 
`test_cast_decimal*_to_decimal*_overflow` tests for overflow of the upscale 
multiply.
   
   `cast_kernels` on an Apple M5 Max with rustc 1.99.1 and the default bench 
profile, criterion medians with `--warm-up-time 2 --measurement-time 4`. `main` 
at eb62e55 and this change ran in two interleaved rounds. The last column is 
from a separate run with the same setup and `#[inline]` only.
   
   | Benchmark | `main` | This PR | Change | Round 1 / 2 | `#[inline]` only |
   |---|---:|---:|---:|---:|---:|
   | `cast int64 within 1e6 late overflow to decimal128(18, 2) 512` | 0.97 µs | 
0.60 µs | -38% | -37% / -39% | -28% |
   | `cast float32 to decimal32(7, 2) 8192 valid strict` | 9.93 µs | 6.66 µs | 
-33% | -35% / -30% | -29% |
   | `cast float32 to decimal32(7, 2) 8192 valid safe` | 8.22 µs | 5.83 µs | 
-29% | -33% / -25% | -32% |
   | `cast float64 to decimal64(3, 0) 8192 precision overflow safe` | 14.3 µs | 
10.8 µs | -24% | -25% / -23% | -37% |
   | `cast decimal128 to decimal128 512 lower precision` | 18.1 µs | 14.2 µs | 
-22% | -22% / -22% | -16% |
   | `cast int32 to decimal32(8, 0) 8192 precision overflow safe` | 10.4 µs | 
8.73 µs | -16% | -16% / -16% | -5% |
   | `cast decimal256 to decimal256 512` | 39.1 µs | 34.7 µs | -11% | -11% / 
-12% | +8% |
   | `cast float64 to decimal128(32, 3)` | 16.6 µs | 15.2 µs | -8% | -8% / -9% 
| -12% |
   | `cast int64 to decimal32(9, 0) 512` | 0.67 µs | 0.63 µs | -5% | -4% / -7% 
| +24% |
   | `cast decimal256 to decimal128 512` | 47.6 µs | 45.6 µs | -4% | -5% / -4% 
| +88% |
   | `cast string to decimal128(38, 3)` | 50.0 µs | 53.0 µs | +6% | +8% / +4% | 
0% |
   
   The other 22 decimal benchmarks moved by -6% to +3%. Two sub-microsecond 
ones, `cast int64 to decimal128(38, 10) 512` and `cast int64 within 1e6 to 
decimal128(18, 2) 512`, measured +8% and +4% in this run but -0.2% and +0.1% in 
a four-round ABBA recheck.
   
   `cast string to decimal128(38, 3)` measured +3% to +6% in three runs. Its 
two hot functions, `parse_decimal_checked` and `cast_to_decimal`, compile to 
the same instructions as with `#[inline]` only, which measured -0.3%, apart 
from the addresses of constants. I think this is a code layout effect, but I 
have not confirmed it.
   
   <details><summary>All 35 decimal benchmarks</summary>
   
   | Benchmark | `main` | This PR | Change | Round 1 / 2 | `#[inline]` only |
   |---|---:|---:|---:|---:|---:|
   | `cast int64 to decimal32(9, 0) 512` | 0.67 µs | 0.63 µs | -5% | -4% / -7% 
| +24% |
   | `cast int64 to decimal32(9, -1) 512` | 0.68 µs | 0.66 µs | -3% | -1% / -6% 
| -7% |
   | `cast int64 to decimal128(38, 10) 512` | 0.27 µs | 0.29 µs | +8% | +15% / 
+1% | 0% |
   | `cast int64 within 1e6 to decimal128(18, 2) 512` | 0.31 µs | 0.32 µs | +4% 
| +9% / 0% | -2% |
   | `cast int64 within 1e6 late overflow to decimal128(18, 2) 512` | 0.97 µs | 
0.60 µs | -38% | -37% / -39% | -28% |
   | `cast decimal128 to decimal128 512` | 2.70 µs | 2.75 µs | +2% | +2% / +2% 
| 0% |
   | `cast decimal128 to decimal128 512 lower precision` | 18.1 µs | 14.2 µs | 
-22% | -22% / -22% | -16% |
   | `cast decimal128 to decimal256 512` | 11.0 µs | 11.2 µs | +2% | +2% / +2% 
| 0% |
   | `cast decimal256 to decimal128 512` | 47.6 µs | 45.6 µs | -4% | -5% / -4% 
| +88% |
   | `cast decimal256 to decimal256 512` | 39.1 µs | 34.7 µs | -11% | -11% / 
-12% | +8% |
   | `cast decimal128 to decimal128 512 with same scale` | 0.02 µs | 0.02 µs | 
+1% | 0% / +1% | -3% |
   | `cast decimal128 to decimal128 512 with lower scale (infallible)` | 18.8 
µs | 18.7 µs | -1% | -1% / 0% | -3% |
   | `cast decimal256 to decimal256 512 with same scale` | 0.02 µs | 0.02 µs | 
+1% | +2% / -1% | -1% |
   | `cast string to decimal128(38, 3)` | 50.0 µs | 53.0 µs | +6% | +8% / +4% | 
0% |
   | `cast string to decimal256(76, 3)` | 76.3 µs | 76.4 µs | 0% | +2% / -2% | 
-3% |
   | `cast float64 to decimal128(32, 3)` | 16.6 µs | 15.2 µs | -8% | -8% / -9% 
| -12% |
   | `cast invalid float64 to to decimal128(32, 3)` | 9.34 µs | 8.83 µs | -5% | 
-3% / -8% | -9% |
   | `cast decimal128 to float64` | 12.7 µs | 13.0 µs | +3% | +3% / +2% | -3% |
   | `cast decimal128 to int8` | 30.5 µs | 29.8 µs | -2% | -3% / -1% | +1% |
   | `cast decimal128 to int64` | 24.5 µs | 24.9 µs | +2% | +3% / +1% | -1% |
   | `cast decimal256 to float64` | 24.3 µs | 24.6 µs | +2% | +2% / +2% | -1% |
   | `cast decimal256 to int64` | 31.8 µs | 32.1 µs | +1% | +1% / +1% | +1% |
   | `cast int32 to decimal32(9, 0) 8192 valid safe` | 0.86 µs | 0.87 µs | +1% 
| +1% / +1% | -3% |
   | `cast int32 to decimal32(9, 0) 8192 valid strict` | 0.91 µs | 0.87 µs | 
-5% | -2% / -7% | -3% |
   | `cast int64 to decimal128(38, 2) 8192 valid safe` | 3.11 µs | 3.15 µs | 
+1% | +4% / -2% | +2% |
   | `cast int64 to decimal128(38, 2) 8192 valid strict` | 3.10 µs | 3.12 µs | 
0% | +2% / -1% | -1% |
   | `cast uint64 to decimal256(76, -1) 8192 scale down safe` | 3.99 µs | 4.04 
µs | +1% | +3% / -1% | -1% |
   | `cast int64 to decimal32(9, -20) 8192 all zero safe` | 0.44 µs | 0.43 µs | 
-1% | +2% / -3% | +1% |
   | `cast int32 to decimal32(8, 0) 8192 precision overflow safe` | 10.4 µs | 
8.73 µs | -16% | -16% / -16% | -5% |
   | `cast int64 to decimal32(9, -1) 8192 scale before narrowing safe` | 5.26 
µs | 5.31 µs | +1% | +1% / +1% | +1% |
   | `cast float32 to decimal32(7, 2) 8192 valid safe` | 8.22 µs | 5.83 µs | 
-29% | -33% / -25% | -32% |
   | `cast float32 to decimal32(7, 2) 8192 valid strict` | 9.93 µs | 6.66 µs | 
-33% | -35% / -30% | -29% |
   | `cast float64 to decimal128(20, 3) 8192 valid strict` | 17.2 µs | 16.9 µs 
| -1% | +2% / -5% | -6% |
   | `cast float64 to decimal256(40, -2) 8192 scale down safe` | 180.1 µs | 
175.6 µs | -2% | -7% / +3% | +1% |
   | `cast float64 to decimal64(3, 0) 8192 precision overflow safe` | 14.3 µs | 
10.8 µs | -24% | -25% / -23% | -37% |
   
   </details>
   
   All measurements are on aarch64. x86-64 may make different branch and 
inlining decisions.
   
   # Are there any significant user-facing changes?
   
   No. `DecimalCast` gains a provided method, so existing implementations keep 
compiling.
   
   The changes, measurements and this description were prepared with Claude 
Code.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to