peterxcli opened a new pull request, #5046: URL: https://github.com/apache/datafusion-comet/pull/5046
## Which issue does this PR close? Closes #888. ## Rationale for this change `ArrowWriter` currently uses Arrow's resizing-safe setters for every value. This performs a capacity check on every write and can reallocate fixed-width vectors while converting Spark columnar batches. `CometSparkToColumnarExec` knows the output batch size before conversion, so fixed-width vectors can be allocated to the required capacity upfront and written without repeated resize checks. ## What changes are included in this PR? - Add an `ArrowWriter.create` overload that preallocates fixed-width vectors using the known batch row count. - Add capacity-asserted column write paths that use non-resizing setters for fixed-width values. - Clear fixed-width validity bits directly after checking capacity. - Keep resizing-safe setters for variable-width and nested vectors. - Update `SparkColumnarArrowReader` to allocate for `rowsToProduce` and use the new write paths. - Add a regression test covering nullable and non-null fixed-width columns beyond Arrow's default initial capacity and verifying that no reallocation occurs. ## How are these changes tested? - Built the native core with `make core`. - Ran `CometArrowStreamSuite` with Spark 4.1 / Scala 2.13: 6/6 tests passed. - Ran `CometArrowStreamSuite` with Spark 3.4 / Scala 2.12: 6/6 tests passed. - Spotless checks passed. A targeted paired A/B benchmark compared the previous `allocateNew` plus resizing-safe write path with the new exact-capacity, non-resizing path. Benchmark configuration: - Two `Long` columns - 4,096 input rows per batch - 10,000 batches per measured round - Five warmup rounds and 15 measured rounds - Median across three fresh JVM forks - macOS arm64, JDK 17, Spark 3.4, Arrow 18.3.0 | Input | Previous path | New path | Time reduction | Speedup | |---|---:|---:|---:|---:| | No nulls | 11.026 ns/row | 8.450 ns/row | 23.4% | 1.31x | | 50% nulls | 10.765 ns/row | 7.732 ns/row | 28.2% | 1.39x | All three JVM forks showed an improvement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
