viirya commented on PR #239: URL: https://github.com/apache/arrow-datafusion-comet/pull/239#issuecomment-2030132659
> I tried to follow all the discussions of the related issues/PRs, and want to make sure I understand the issues/situations correctly. Per my understanding, please correct me if I'm wrong: > > 1. Java Arrow's C data interface is exporting empty(/null) arraybuf for empty offset buffer, which will be fixed in [GH-40038: [Java] Export non empty offset buffer for variable-size layout through C Data Interface arrow#40043](https://github.com/apache/arrow/pull/40043). Correct. > 2. arrow-rs's implementation would allow empty offset buffer per discussion and its follow-up PRs: [Ensure there is a single zero in the offsets buffer for an empty ListArray. arrow-rs#1620](https://github.com/apache/arrow-rs/issues/1620) and https://github.com/apache/arrow-rs/pull/2836/files > > * But the arrow-rs ffi interface doesn't allow empty offset yet? So we have to patch arrow-rs in https://github.com/viirya/arrow-rs/commits/workaround_java_offset_bufs/ ? > * Will the upstream arrow-rs to pick that change/fix? > > If the upstream arrow-rs is going to pick-up the fix, is it possible for us to wait a bit and use arrow-rs upstream directly? I'm a bit of worried about the stability/quality of using the main branch of arrow-rs directly. No. As empty offset buffer is invalid, we are not going to merge the fix into arrow-rs. It is a hacky fix temporarily only before we have the fix at Java Arrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
