andygrove opened a new pull request, #3221: URL: https://github.com/apache/datafusion-comet/pull/3221
## Summary This PR adds an experimental native (Rust-based) implementation of ColumnarToRowExec that converts Arrow columnar data to Spark UnsafeRow format. **Benefits over the current Scala implementation:** - **Zero-copy for variable-length types**: String and Binary data is written directly to the output buffer without intermediate Java object allocation - **Vectorized processing**: The native implementation processes data in a columnar fashion, improving CPU cache utilization - **Reduced GC pressure**: All conversion happens in native memory, avoiding the creation of temporary Java objects that would need garbage collection - **Buffer reuse**: The output buffer is allocated once and reused across batches, minimizing memory allocation overhead **Configuration:** The feature is disabled by default and can be enabled by setting: ``` spark.comet.exec.columnarToRow.native.enabled=true ``` **Supported data types:** - Primitive types: Boolean, Byte, Short, Int, Long, Float, Double - Date and Timestamp (microseconds) - Decimal (both inline precision<=18 and variable-length precision>18) - String and Binary - Complex types: Struct, Array, Map (nested) ## Test plan - [ ] Unit tests for Rust conversion logic (6 tests passing) - [ ] Integration tests with existing Comet test suite - [ ] Performance benchmarks comparing native vs JVM implementation ⚠️ **This is an experimental feature for evaluation and benchmarking purposes.** 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
