Hello! On 7/13/26 13:15, wenhui qiu wrote: > HI hackers > > LZ4 and Zstandard are both excellent and well-maintained open-source > projects. However, they remain optional dependencies maintained outside > the PostgreSQL project, while PGLZ is an implementation that PostgreSQL > ships and controls itself. > > Given the known performance gap between PGLZ and modern compression > algorithms, I wanted to explore a simple question: rather than relying > only on external libraries for better compression performance, can we > also improve PostgreSQL's own built-in compression algorithm? > > This patch grew out of that idea. It is not intended to replace LZ4 or > Zstandard, but to make PostgreSQL's self-contained compression option > substantially more competitive. > > > The attached RFC patch explores a self-contained modernization of PGLZ. > > For inputs of at least 32KB, the encoder first tries a new byte-aligned > stream format. It uses a single-candidate LZ77 parser, a 65535-byte > history window, literal runs, fixed-width offsets, and ULEB128 length > extensions. This format is designed primarily for fast decoding. > > If that representation cannot satisfy the compression strategy, a second > format can encode literals with canonical Huffman codes. Smaller inputs > and unsuccessful modern-format attempts continue to use the legacy PGLZ > representation. >
So it effectively implements a new compression algorithm/format, incompatible with PGLZ? Considering there are great external compression libraries, why should we try to compete with them? I'm sure inventing new compression algorithm is fun, but ISTM we're unlikely to beat the external libraries. I suppose most packages are built with these external libraries, right? So people who care about compression would already benefit from that. We've even switched TOAST to lz4 when possible in PG 19. > The patch also optimizes the legacy path with a rolling hash, indexed > history rings, bounded candidate searches, word/SIMD match extension, > and an AVX2 implementation selected by PostgreSQL's runtime CPU check. > Other platforms use PostgreSQL's existing SIMD abstraction. > This, on the other hand would be a *huge* improvement for all existing systems with a lot of data compressed with pglz (and there's fair amount of those). If we could just make them faster by optimizing the code, that seems very desirable. > The new streams begin with a sequence that is invalid in the legacy > format, allowing the decoder to distinguish them without changing the > external TOAST compression identifier. The new decoder continues to > read existing legacy PGLZ values. Compatibility is one-way: an > unpatched server cannot decode values written in either new format. > > The implementation is self-contained. It does not link to or incorporate > source code from LZ4 or Zstandard. > > I benchmarked the 51,975-byte example-large.json input, repeated 4,100 > times per run. The following are medians of five runs on ARM64 macOS, > compiled with Apple clang -O3, based on master at 1ae87df63ff: > > master patched LZ4 > compression MB/s 325.9 2137.3 2811.2 > decompression MB/s 2831.1 7582.5 11777.8 > compressed bytes 14370 12445 12802 > > On this input, the patch made PGLZ compression about 6.6 times faster > and decompression about 2.7 times faster than master. The resulting > stream was 13.4% smaller. It remains slower than LZ4, especially during > decompression. > Those are nice improvements, but once again - why wouldn't I just use lz4 for new system? The new algorithm still doesn't beat it, right? > Testing performed: > > * clean out-of-tree build with --with-lz4 and --with-zstd > * make check: all 245 tests passed > * 24,820 full and partial round-trip cases under ASan and UBSan > * tests for partial decompression and malformed version-3 streams > * git diff --check > > This is intentionally an RFC because it changes the PGLZ on-disk stream > and is currently a large patch. I would especially appreciate feedback > on: > > * whether the one-way compatibility model is acceptable > * whether having both the byte-aligned and Huffman fallback formats is > worth the implementation complexity > * whether the 32KB selection threshold is reasonable I think one-way compatibility would be fine, it'd require a major version upgrade anyway. But I'm not convinced we want to extend the pglz like this. > * whether the legacy hot-path changes should be submitted separately > Yes. I think the cost/benefit is much clearer there, and I expect this part to have much higher chance of succeeding. > For transparency, I used an AI coding assistant while prototyping parts > of this patch. I reviewed and tested the resulting implementation and > took responsibility for maintaining and revising it. > > Thanks regards -- Tomas Vondra
