pdet opened a new pull request, #935: URL: https://github.com/apache/arrow-nanoarrow/pull/935
This PR adds LZ4 and ZSTD compression to nanoarrow’s IPC encoder and writer, allowing applications to write compressed Arrow streams and files. The motivation comes from the duckdb-arrow extension, where we want nanoarrow to handle compression while the extension takes care of the SQL options. To support this, callers can select a codec and compression level, query the supported level range, and convert between codec names and enum values without needing to include the compression libraries’ headers themselves. Compression happens per buffer within each record batch, following the Arrow IPC format. If compressing a buffer does not make it smaller, we store it uncompressed, and empty buffers remain empty. The implementation provides a serial compressor by default, with an interface for applications to supply their own. Unsupported codecs and out-of-range levels are rejected when configuring the built-in compressor, so callers can report these errors before writing data. The added tests cover compression and decompression roundtrips, compressed streams and files, compression levels, and builds with either codec unavailable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
