ivirshup opened a new issue, #37756: URL: https://github.com/apache/arrow/issues/37756
### Describe the enhancement requested As discussed with @jorisvandenbossche at the NumFOCUS summit (thanks for pointing me to `Message.fbs`!). At the moment, there is no documentation of how compression works with the IPC format in the format docs (e.g. https://arrow.apache.org/docs/format/Columnar.html). It would be quite helpful if this was documented. There is some documentation of this in the [Message.fbs](https://github.com/apache/arrow/blob/main/format/Message.fbs) file, but it should probably be part of the formal docs as well. Some questions I have that I think prose documentation could answer: * Is it correct that each buffer (in the flattened tree) is compressed separately, but the exact same compression must be used in each case? * Can we randomly access a compressed `Buffer` within a `RecordBatch` because flatbuffers should provide us with information about the lengths of the compressed `Buffer`s? ### Component(s) Documentation, Format -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
