comphead opened a new pull request, #12342: URL: https://github.com/apache/arrow-rs/pull/12342
# Which issue does this PR close? - Closes #12326. # Rationale for this change The `DecimalType::is_valid_decimal_precision` implementations are not `#[inline]`, so a caller in another crate calls them out of line for every value unless it builds with cross-crate LTO. The arrow-rs workspace does not, and arrow-cast's decimal casts reach them through 262 out-of-line calls in the `cast_kernels` bench binary. #12326 has the disassembly and the DataFusion Comet case. Adding `#[inline]` alone makes eight decimal casts 5% to 37% faster but three slower. Profiles of the slower ones with [samply](https://github.com/mstange/samply) show two causes, both in the arrow-cast kernels rather than in the check: - `cast decimal256 to decimal128 512` (+88%). With the check inlined into the per-row closure, LLVM compiles the rounding step of `make_downscaler` (`d + 1` when `r >= half`) as a branch on the remainder instead of a `csel`. The two targets of that branch take 51.5% of all samples, which is what mispredictions look like. The bench values are random, so about half the rows round up. - `cast int64 to decimal32(9, 0) 512` (+24%) and `cast decimal256 to decimal256 512` (+8%). The larger closure is no longer inlined into the two loops of `try_for_each_valid_idx`, so every row calls it out of line and reloads its captures through the closure environment. Both closures also handle an `ArrowError` from `mul_checked` and then discard it with `.ok()`. #11529 describes the same mechanism in arrow-arith. # What changes are included in this PR? - `#[inline]` on the four `is_valid_decimal_precision` implementations. - `make_downscaler` keeps the predictable sign branch and rounds with `std::hint::select_unpredictable`, stable since Rust 1.88, the MSRV. - A provided method, `DecimalCast::mul_checked_opt`, returns `None` on overflow without building an error. The four implementations use `checked_mul`. `make_upscaler` and the safe integer-to-decimal cast use it instead of `mul_checked(..).ok()`, which brings their closures back under LLVM's inline threshold. I did not change `ArrowNativeTypeOp` itself. #11529 found that moving its error construction into a `#[cold]` function slows integer division with nulls by 6% to 10%. # Are these changes tested? The behavior does not change, and existing tests cover the rewritten paths: `test_rescale_decimal_downscale_rounds_half_away_from_zero` for rounding of positive and negative values, `test_cast_numeric_to_decimal128_overflow` for overflow of the integer-to-decimal multiply with `safe: true`, and the `test_cast_decimal*_to_decimal*_overflow` tests for overflow of the upscale multiply. `cast_kernels` on an Apple M5 Max with rustc 1.99.1 and the default bench profile, criterion medians with `--warm-up-time 2 --measurement-time 4`. `main` at eb62e55 and this change ran in two interleaved rounds. The last column is from a separate run with the same setup and `#[inline]` only. | Benchmark | `main` | This PR | Change | Round 1 / 2 | `#[inline]` only | |---|---:|---:|---:|---:|---:| | `cast int64 within 1e6 late overflow to decimal128(18, 2) 512` | 0.97 µs | 0.60 µs | -38% | -37% / -39% | -28% | | `cast float32 to decimal32(7, 2) 8192 valid strict` | 9.93 µs | 6.66 µs | -33% | -35% / -30% | -29% | | `cast float32 to decimal32(7, 2) 8192 valid safe` | 8.22 µs | 5.83 µs | -29% | -33% / -25% | -32% | | `cast float64 to decimal64(3, 0) 8192 precision overflow safe` | 14.3 µs | 10.8 µs | -24% | -25% / -23% | -37% | | `cast decimal128 to decimal128 512 lower precision` | 18.1 µs | 14.2 µs | -22% | -22% / -22% | -16% | | `cast int32 to decimal32(8, 0) 8192 precision overflow safe` | 10.4 µs | 8.73 µs | -16% | -16% / -16% | -5% | | `cast decimal256 to decimal256 512` | 39.1 µs | 34.7 µs | -11% | -11% / -12% | +8% | | `cast float64 to decimal128(32, 3)` | 16.6 µs | 15.2 µs | -8% | -8% / -9% | -12% | | `cast int64 to decimal32(9, 0) 512` | 0.67 µs | 0.63 µs | -5% | -4% / -7% | +24% | | `cast decimal256 to decimal128 512` | 47.6 µs | 45.6 µs | -4% | -5% / -4% | +88% | | `cast string to decimal128(38, 3)` | 50.0 µs | 53.0 µs | +6% | +8% / +4% | 0% | The other 22 decimal benchmarks moved by -6% to +3%. Two sub-microsecond ones, `cast int64 to decimal128(38, 10) 512` and `cast int64 within 1e6 to decimal128(18, 2) 512`, measured +8% and +4% in this run but -0.2% and +0.1% in a four-round ABBA recheck. `cast string to decimal128(38, 3)` measured +3% to +6% in three runs. Its two hot functions, `parse_decimal_checked` and `cast_to_decimal`, compile to the same instructions as with `#[inline]` only, which measured -0.3%, apart from the addresses of constants. I think this is a code layout effect, but I have not confirmed it. <details><summary>All 35 decimal benchmarks</summary> | Benchmark | `main` | This PR | Change | Round 1 / 2 | `#[inline]` only | |---|---:|---:|---:|---:|---:| | `cast int64 to decimal32(9, 0) 512` | 0.67 µs | 0.63 µs | -5% | -4% / -7% | +24% | | `cast int64 to decimal32(9, -1) 512` | 0.68 µs | 0.66 µs | -3% | -1% / -6% | -7% | | `cast int64 to decimal128(38, 10) 512` | 0.27 µs | 0.29 µs | +8% | +15% / +1% | 0% | | `cast int64 within 1e6 to decimal128(18, 2) 512` | 0.31 µs | 0.32 µs | +4% | +9% / 0% | -2% | | `cast int64 within 1e6 late overflow to decimal128(18, 2) 512` | 0.97 µs | 0.60 µs | -38% | -37% / -39% | -28% | | `cast decimal128 to decimal128 512` | 2.70 µs | 2.75 µs | +2% | +2% / +2% | 0% | | `cast decimal128 to decimal128 512 lower precision` | 18.1 µs | 14.2 µs | -22% | -22% / -22% | -16% | | `cast decimal128 to decimal256 512` | 11.0 µs | 11.2 µs | +2% | +2% / +2% | 0% | | `cast decimal256 to decimal128 512` | 47.6 µs | 45.6 µs | -4% | -5% / -4% | +88% | | `cast decimal256 to decimal256 512` | 39.1 µs | 34.7 µs | -11% | -11% / -12% | +8% | | `cast decimal128 to decimal128 512 with same scale` | 0.02 µs | 0.02 µs | +1% | 0% / +1% | -3% | | `cast decimal128 to decimal128 512 with lower scale (infallible)` | 18.8 µs | 18.7 µs | -1% | -1% / 0% | -3% | | `cast decimal256 to decimal256 512 with same scale` | 0.02 µs | 0.02 µs | +1% | +2% / -1% | -1% | | `cast string to decimal128(38, 3)` | 50.0 µs | 53.0 µs | +6% | +8% / +4% | 0% | | `cast string to decimal256(76, 3)` | 76.3 µs | 76.4 µs | 0% | +2% / -2% | -3% | | `cast float64 to decimal128(32, 3)` | 16.6 µs | 15.2 µs | -8% | -8% / -9% | -12% | | `cast invalid float64 to to decimal128(32, 3)` | 9.34 µs | 8.83 µs | -5% | -3% / -8% | -9% | | `cast decimal128 to float64` | 12.7 µs | 13.0 µs | +3% | +3% / +2% | -3% | | `cast decimal128 to int8` | 30.5 µs | 29.8 µs | -2% | -3% / -1% | +1% | | `cast decimal128 to int64` | 24.5 µs | 24.9 µs | +2% | +3% / +1% | -1% | | `cast decimal256 to float64` | 24.3 µs | 24.6 µs | +2% | +2% / +2% | -1% | | `cast decimal256 to int64` | 31.8 µs | 32.1 µs | +1% | +1% / +1% | +1% | | `cast int32 to decimal32(9, 0) 8192 valid safe` | 0.86 µs | 0.87 µs | +1% | +1% / +1% | -3% | | `cast int32 to decimal32(9, 0) 8192 valid strict` | 0.91 µs | 0.87 µs | -5% | -2% / -7% | -3% | | `cast int64 to decimal128(38, 2) 8192 valid safe` | 3.11 µs | 3.15 µs | +1% | +4% / -2% | +2% | | `cast int64 to decimal128(38, 2) 8192 valid strict` | 3.10 µs | 3.12 µs | 0% | +2% / -1% | -1% | | `cast uint64 to decimal256(76, -1) 8192 scale down safe` | 3.99 µs | 4.04 µs | +1% | +3% / -1% | -1% | | `cast int64 to decimal32(9, -20) 8192 all zero safe` | 0.44 µs | 0.43 µs | -1% | +2% / -3% | +1% | | `cast int32 to decimal32(8, 0) 8192 precision overflow safe` | 10.4 µs | 8.73 µs | -16% | -16% / -16% | -5% | | `cast int64 to decimal32(9, -1) 8192 scale before narrowing safe` | 5.26 µs | 5.31 µs | +1% | +1% / +1% | +1% | | `cast float32 to decimal32(7, 2) 8192 valid safe` | 8.22 µs | 5.83 µs | -29% | -33% / -25% | -32% | | `cast float32 to decimal32(7, 2) 8192 valid strict` | 9.93 µs | 6.66 µs | -33% | -35% / -30% | -29% | | `cast float64 to decimal128(20, 3) 8192 valid strict` | 17.2 µs | 16.9 µs | -1% | +2% / -5% | -6% | | `cast float64 to decimal256(40, -2) 8192 scale down safe` | 180.1 µs | 175.6 µs | -2% | -7% / +3% | +1% | | `cast float64 to decimal64(3, 0) 8192 precision overflow safe` | 14.3 µs | 10.8 µs | -24% | -25% / -23% | -37% | </details> All measurements are on aarch64. x86-64 may make different branch and inlining decisions. # Are there any significant user-facing changes? No. `DecimalCast` gains a provided method, so existing implementations keep compiling. The changes, measurements and this description were prepared with Claude Code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
