andygrove opened a new pull request, #3814: URL: https://github.com/apache/datafusion-comet/pull/3814
## Which issue does this PR close? Closes #875. ## Rationale for this change The `spark.comet.parquet.enable.directBuffer` config causes a null pointer exception when enabled because Parquet's `toByteBuffer()` often returns heap buffers (via `ByteBuffer.wrap(toByteArray())`), not direct buffers. The native JNI code calls `GetDirectBufferAddress` which returns NULL for heap buffers, triggering `CometNativeException: Null pointer in get_direct_buffer_address return value`. Rather than adding a fallback check, the entire feature is removed because: - It was disabled by default, undocumented, and untested - Parquet's `toByteBuffer()` returns heap buffers for compressed pages (the common case), so the "fast path" rarely triggers - Even when it works, the benefit is minimal — it only avoids one memcpy compared to the byte array path - The fallback would make the code worse than not having the feature at all (calling both `toByteBuffer()` and `toByteArray()`) ## What changes are included in this PR? Removes the `spark.comet.parquet.enable.directBuffer` config and all associated code across 5 files: - `ColumnReader.java` — removed direct buffer branch, now always uses byte array path - `Native.java` — removed `setPageBufferV1` native method declaration - `CometConf.scala` — removed `COMET_PARQUET_ENABLE_DIRECT_BUFFER` config entry - `mod.rs` — removed `setPageBufferV1` JNI function, `last_data_page` field, and unused imports - `CometParquetFileFormat.scala` — removed hadoop config propagation ## How are these changes tested? This is a pure removal of dead/broken code. The default behavior (byte array path) is already extensively tested by the existing `ParquetReadSuite` tests. Clippy passes cleanly after the removal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
