Hi, There seems to be a new (if I’m not mistaken it was published yesterday) codec/compression framework called OpenZL [1][2][3]. I haven’t looked at it thoroughly yet, but it somewhat reminds me of BtrBlocks. Even if we don’t consider more advanced features of a framework like this, we could offload the various codec implementations to another project.
Krisztian [1]: https://openzl.org/ [2]: https://github.com/facebook/openzl/tree/dev/src/openzl/codecs [3]: https://engineering.fb.com/2025/10/06/developer-tools/openzl-open-source-format-aware-compression-framework/ > On 2025. Oct 1., at 20:11, Andrew Lamb <[email protected]> wrote: > > I would like to start a discussion to help organize and rally anyone > interested in adding new encodings to Parquet. > > I am pretty sure there are many people interested in adding new encodings, > but there are only a few mentions on the mailing list, such as pcode [1] > and FSST/ALP/FastLanes [2]. Prateek mentioned on the sync call today > that he is working on evaluating some potential encodings and hopes to have > some information to share soon, and Julien mentioned he had spoken to > someone else who might be doing something similar. > > Now that Julien has defined a process to extend the spec[3] I think the > steps are much clearer. > > So, I would like to invite anyone interested in adding new encodings to > respond and let us know if you are willing to help evaluate new encodings > and prototype integrations into Parquet implementations? > > Andrew > > > [1]: https://lists.apache.org/thread/bdmfcj4g6y1ccd3mfgrp7d43d73s6zf6 > [2]: https://lists.apache.org/thread/s3o9jk0hr942pv6ono4ymnvvj6pfdsdw > [3]: > https://github.com/apache/parquet-format/blob/master/proposals/README.md
