andygrove commented on PR #6943: URL: https://github.com/apache/datafusion-comet/pull/6943#issuecomment-6156044743
Thanks for the review. I added the codec-tag flag to the Encoding step and to the block header description in `native_shuffle.md`, and the config to the tables in `native_shuffle.md` and `jvm_shuffle.md` (1e0879139b). I also pushed a simplification of the writer (3194f1aaa2): it now decides when it is built instead of per block. The fallback you mentioned still holds, just per writer. It encodes an empty batch once and keeps plain offsets if arrow-ipc's buffers don't match the mapped roles, which is enough because a type's buffer count doesn't depend on its data. The push restarts the Lint Java job and the Spark 4.1 SQL run. Agreed on waiting for that run before this is queued. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
