yinli-systems opened a new pull request, #11062: URL: https://github.com/apache/arrow-rs/pull/11062
# Which issue does this PR close? - Closes #10891. # Rationale for this change Numeric kernels currently reject run-end encoded inputs. This prevents consumers such as DataFusion from applying arithmetic to compressed ordering columns without decoding them first. # What changes are included in this PR? - Route the checked, wrapping, division, remainder, and negation numeric entry points through a shared REE implementation. - Preserve run boundaries for scalar operations and use the union of boundaries for two REE operands. Aligned inputs avoid gathering their physical values. - Normalize sliced run ends, preserve primitive null/overflow/decimal/temporal semantics, and select the wider run-end integer type for two differently encoded operands. - Return an explicit error for a REE array combined with a non-scalar unencoded array. Like dictionary arithmetic, that case would require decoding to produce an unencoded result and is outside this compression-preserving path. # Are these changes tested? Yes. The new differential tests compare decoded REE results with the existing primitive kernels across all public numeric operations, all run-end integer widths, integer/float/decimal/timestamp values, nulls, checked and wrapping overflow, division errors, unequal run boundaries, every slice offset in a deterministic matrix, empty inputs, and both scalar directions. A billion-row logical array test verifies work and result storage remain proportional to physical runs. Local validation: - `cargo test -p arrow-arith --all-features`: 239 unit tests, 13 integration tests, and 16 doctests passed. - `cargo test --lib --tests --release -p arrow-arith`: 239 unit tests and 13 integration tests passed. - `cargo test -p arrow --features=force_validate,prettyprint,ipc_compression,ffi,chrono-tz`: passed. - `cargo clippy -p arrow-arith --all-targets --all-features -- -D warnings`: passed. - `cargo +stable fmt --all -- --check`: passed. A local one-million-row release-mode timing harness showed the intended compression-dependent tradeoff. With runs of 128 rows, REE was 2.4x-13x faster than arithmetic over already-resident dense arrays; with runs of 4096 rows it was 64x-275x faster. Runs of 8 rows were 15%-6.5x slower, and run length 1 was substantially slower, so this PR does not claim REE is beneficial for low-compression data. The billion-row test covers the non-expansion property deterministically. The same integration test file was also run against the unmodified base: 12 of 13 tests failed because REE numeric operations were unsupported; only the invalid-input test passed. # Are there any user-facing changes? Yes. Numeric arithmetic and negation can now operate directly on REE arrays when the other operand is a scalar or REE array. The result remains run-end encoded. There are no public API signature changes. AI disclosure: Codex assisted with repository research, implementation, test generation, performance analysis, and self-review. I reviewed the resulting behavior through differential tests, negative controls against the base revision, release-mode tests, strict Clippy, and the compression benchmark. No independent human review is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
