andygrove commented on PR #1511: URL: https://github.com/apache/datafusion-comet/pull/1511#issuecomment-2718221569
> The failure in java-native_iceberg_compat test is caused by a bug in the [fast shuffle writer](https://github.com/apache/datafusion-comet/blob/58a36b62004c1e300d7f3113d9a64713a94c0db9/native/core/src/execution/shuffle/codec.rs#L302-L304). The null buffers written for sliced arrays are not correct. I'll submit a dedicated patch to fix it later. There is now an option in the Arrow IPC reader to disable re-validating the data and we may be able to remove our custom encoding if that solves the performance issues we saw previously. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
