andrewthad opened a new issue, #34560: URL: https://github.com/apache/arrow/issues/34560
### Describe the bug, including details regarding any error messages, version, and platform. Contrasting arrow with parquet, the arrow FAQ claims: > Conversely, Arrow is an in-memory format meant for direct and efficient use for computational purposes. Arrow data is not compressed (or only lightly so, when using dictionary encoding) but laid out in natural format for the CPU, so that data can be accessed at arbitrary places at full speed. This is not accurate. According to `Message.fbs`, buffers may be compressed with LZ4 or ZSTD. Pyarrow supports even more: brotli, gzip, lz4, lz4_raw, snappy, zstd. I've not followed the history of Apache Arrow, but I assume that support for compression was added later and that the FAQ is out of date. If the FAQ is brought up to date, it would also be good to justify why support for compression was added. My best guess is that a lot of analytics workloads just needed a single beginning-to-end scan of certain columns, and for these workloads, there is no advantage of memory mapping. (Memory mapping would be most helpful if you repeatedly accessed the same data.) In these cases, compression would reduce disk (or network) latency, and since the buffers are compressed individually, you only decompress what you need (unlike parquet). But all of that is just my guess. It would be good to have a more authoritative source make this claim. ### Component(s) Documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
