peterxcli opened a new pull request, #5442: URL: https://github.com/apache/datafusion-comet/pull/5442
## Which issue does this PR close? Closes #5299. ## Rationale for this change The fixed-width no-null path introduced in #5046 still calls `setValueUnsafe` once per value. Spark's on-heap and off-heap column vectors expose public bulk getters, so sufficiently large fixed-width slices can be copied into Arrow with one bulk getter call and one memory copy instead. The implementation uses the writer-level extension point described in #5317. This keeps vector ownership and slice offsets explicit without reflection or access to private `ColumnarArray` internals. ## What changes are included in this PR? - Add `ArrowWriter.writeColumns` and `ArrowFieldWriter.writeColumnSlice` so field writers receive the source `ColumnVector`, row offset, and slice length. - Bulk-copy supported fixed-width, no-null `OnHeapColumnVector` and `OffHeapColumnVector` slices using Spark's public bulk getters and `Platform.copyMemory`. - Enable bulk copy only for initial writes of at least 32 rows on little-endian hosts. Nullable, dictionary-backed, short, big-endian, Boolean, Decimal, CalendarInterval, and other unsupported inputs retain the existing scalar path. - Update `SparkColumnarArrowReader` to use the shared batch API while preserving split-batch offsets and zero-column row counts. - Extend focused tests and the existing Arrow writer benchmark. ### Benchmark results Apple M4, macOS 26.5.2, OpenJDK 21.0.6, Spark 4.1 profile; three columns (`Int`, `Long`, and `Double`). Values are best time in nanoseconds per row. | Rows | On-heap bulk | On-heap scalar | Speedup | Off-heap bulk | Off-heap scalar | Speedup | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 32 | 27.3 | 35.2 | 1.29x | 28.6 | 36.4 | 1.27x | | 512 | 2.9 | 12.9 | 4.45x | 2.9 | 14.7 | 5.07x | | 8192 | 1.6 | 11.6 | 7.25x | 1.5 | 14.3 | 9.53x | The 1-row and 8-row cases remain on the scalar path; the measured crossover starts at 32 rows. ## How are these changes tested? - `make core` - `./mvnw test -Dtest=none -Dsuites=org.apache.spark.sql.comet.execution.arrow.CometArrowStreamSuite` (Spark 4.1 / JDK 21: 12 tests passed) - `./mvnw test -Pspark-3.4 -Dtest=none -Dsuites=org.apache.spark.sql.comet.execution.arrow.CometArrowStreamSuite` (Spark 3.4 / JDK 17: 12 tests passed) - `./mvnw spotless:apply`; Scalastyle and Spotless checks also passed in both focused Maven runs - `CometArrowWriterBenchmark` via Maven Exec on the Spark 4.1 profile - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
