nat-openai opened a new pull request, #6833: URL: https://github.com/apache/datafusion-comet/pull/6833
## Which issue does this PR close? Partially addresses #5712. This splits the integral-sequence optimisation work out of #6819. ## Rationale for this change Generating long integral sequences spends substantial time on per-element buffer-capacity checks and wide length arithmetic. Scalar arguments are also expanded into temporary arrays before the kernel starts. This change reduces that overhead while preserving Spark's sequence values, null handling and errors. Native Criterion benchmarks on an Apple M3 Max show approximately 5× faster generation for 365-element and 10,000-element column sequences. ## What changes are included in this PR? - Borrow scalar arguments directly instead of expanding them into temporary columns. - Calculate ordinary sequence lengths with checked `i64` arithmetic, retaining the wide-arithmetic fallback for Spark's exact overflow errors. Keep validated row lengths so generation does not repeat that work. - Reserve the output once and write through fixed-size slices. Removing per-element capacity checks lets LLVM vectorise the element loop; ARM SIMD was confirmed in the compiled benchmark. - Combine input validity bitmaps with Arrow's bulk operations and reuse the result. Entirely null batches return an empty-child list array without per-row generation. - Extend behavioural tests across all four integer widths, scalar/column combinations, sliced null bitmaps, arithmetic boundaries and error precedence. Expand the native benchmark to 36 shapes, restore JVM benchmark coverage for 2-, 1,000- and 50,000-element sequences, and record the performance audit. Scalar results and constant folding retain their existing behaviour. This PR does not introduce sequence byte limits, memory accounting, cancellation polling, or changes to native eligibility. An LLM assisted with implementation and independent correctness, performance and scope reviews. ## How are these changes tested? - All 10 focused native sequence tests pass in release mode, including the existing tests unchanged. - Rust formatting, Scalafix, Spotless and the changed Markdown file's Prettier check pass. - The existing `CometSqlFileTestSuite sequence` fixture passes on the default Spark 4.1 profile using the freshly built release native library. - The full native release build, JVM package build and workspace all-target release Clippy check (`-D warnings`) pass. - Independent correctness, performance and scope reviews found no actionable issues. - Native Criterion comparisons use unmodified production source from `main` at `bfacc43a50aba13da4fa8a828db52309541191b4` and the same expanded harness on both sides. Flagged regressions were independently repeated. - All 36 native Criterion cases execute successfully in test mode, including the restored scalar 2- and 10,000-element cases for both row counts. - The expanded `CometSequenceBenchmark` completes all 10 cases with the standard session configuration and no sequence byte-allowance override. Scala test compilation, Spotless, native benchmark compilation and Markdown formatting checks pass. - Before merge, enable `run-spark-4.1-tests` and `run-benchmark-check` on the PR. These broader CI gates have not been run locally. ### Performance Native kernel mean times. Difference is `(this branch / main − 1) × 100`; negative is faster. | Shape | Main | This branch | Difference (%) | |---|---:|---:|---:| | short_1_elems | 39.525 µs | 35.564 µs | -10.0 | | short_2_elems | 70.255 µs | 40.167 µs | -42.8 | | short_5_elems | 78.662 µs | 51.535 µs | -34.5 | | long_365_elems | 1.699 ms | 0.320 ms | -81.1 | | long_10000_elems | 44.777 ms | 8.580 ms | -80.8 | | descending_365_elems | 1.695 ms | 0.586 ms | -65.5 | | zero_step_start_eq_stop | 46.238 µs | 40.785 µs | -11.8 | | sparse_nulls_365_elems | 1.585 ms | 0.302 ms | -81.0 | | dense_nulls_365_elems | 887.294 µs | 171.972 µs | -80.6 | | explicit_step_one_element | 48.044 µs | 36.866 µs | -23.3 | | mixed_non_unit_steps | 127.146 µs | 58.756 µs | -53.8 | | error_illegal_boundaries | 0.258 µs | 0.221 µs | -14.2 | | scalar_step_1_elems | 47.353 µs | 33.971 µs | -28.3 | | scalar_step_2_elems | 79.512 µs | 40.798 µs | -48.7 | | scalar_step_5_elems | 86.406 µs | 43.366 µs | -49.8 | | scalar_step_365_elems | 1.732 ms | 0.316 ms | -81.8 | | scalar_step_10000_elems | 48.757 ms | 8.279 ms | -83.0 | | all_scalar_1_rows_1_elems | 0.515 µs | 0.375 µs | -27.2 | | all_scalar_1_rows_2_elems | 0.508 µs | 0.349 µs | -31.4 | | all_scalar_1_rows_5_elems | 0.544 µs | 0.381 µs | -30.0 | | all_scalar_1_rows_365_elems | 0.679 µs | 0.402 µs | -40.8 | | all_scalar_1_rows_10000_elems | 4.621 µs | 1.542 µs | -66.6 | | all_scalar_8192_rows_1_elems | 46.123 µs | 45.679 µs | -1.0 | | all_scalar_8192_rows_2_elems | 50.057 µs | 49.896 µs | -0.3 | | all_scalar_8192_rows_5_elems | 60.518 µs | 60.369 µs | -0.2 | | all_scalar_8192_rows_365_elems | 440.385 µs | 428.354 µs | -2.7 | | all_scalar_8192_rows_10000_elems | 5.934 ms | 6.022 ms | +1.5 | | sparse_nulls_1_elems | 44.191 µs | 35.941 µs | -18.7 | | sparse_nulls_2_elems | 82.586 µs | 41.809 µs | -49.4 | | sparse_nulls_5_elems | 79.315 µs | 52.218 µs | -34.2 | | dense_nulls_1_elems | 33.867 µs | 26.184 µs | -22.7 | | dense_nulls_2_elems | 49.061 µs | 29.620 µs | -39.6 | | dense_nulls_5_elems | 54.080 µs | 36.874 µs | -31.8 | | all_nulls_1_elems | 19.792 µs | 0.631 µs | -96.8 | | all_nulls_2_elems | 19.796 µs | 0.623 µs | -96.9 | | all_nulls_5_elems | 19.860 µs | 0.613 µs | -96.9 | End-to-end best times. | Shape | Spark best (ms) | Comet best (ms) | |---|---:|---:| | seq_short_2_elems | 18 | 8 | | seq_short_5_elems | 12 | 7 | | seq_spine_365_elems | 14 | 8 | | seq_long_1000_elems | 18 | 10 | | seq_long_10000_elems | 68 | 39 | | seq_long_50000_elems | 275 | 164 | | seq_descending_default_step | 13 | 7 | | seq_explicit_step_7 | 10 | 6 | | seq_sparse_nulls_365_elems | 12 | 6 | | seq_date_spine_dispatcher | 12 | 17 | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
