alamb commented on issue #50172: URL: https://github.com/apache/arrow/issues/50172#issuecomment-4710742639
We recently went through something related to this in arrow-rs (for Utf8/Binary, not Utf8View and BInaryView) with @atwam and polars. Apparently their reading of the spec expects a non null (empty) buffer even when there is are no rows - https://github.com/apache/arrow-rs/issues/9716 We chose to follow the C/C++ implementation (see here https://github.com/apache/arrow-rs/issues/9716#issuecomment-4261755578) which writes non null buffers in this case Following what @paleolimbot says, it seems like we have existing implementations (and we may not know of them all) that can write null buffers; This mean that in practice, readers will likely have to handle null buffers even if we change the spec to not officially allow it, to be tolerant on read Therefore I would personally suggest we clarify the spec to say that null buffers are allowed in IPC/C Data Interface and that they are semantically equivalent to a zero length, non null buffer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
