On Tue, Oct 27, 2026 at 12:53 PM Greg Burd <[email protected]> wrote: > > > On Monday, October 26th, 2026 at 3:02 PM, Greg Burd <[email protected]> wrote: > > > > On Sunday, October 25th, 2026 at 12:52 PM, Hannu Krosing > > <[email protected]> wrote: > > > Hi Hannu, > > I read the Generic Datum Encoding and Compression Contexts patch. It is a > bigger idea than this one, and worth pulling apart where the two meet and > where they don't.
Thanks for a thoughtful review! > Inline below. > > > Please take a look at a generalized design for the compression > > dictionary [...] I have used this myself in dictionary compression for > > zstd and lz4 and for keeping Apache Avro schemas for JSON and JSONB > > encodings and defining cascaded Avro+zstd > > Where we agree: a trained dictionary is just one case of "a value needs a > side artifact to be decoded." Your pg_encoding_context is that artifact > in the general form, and a zstd/lz4 dictionary is one kind of content for > it. The two designs even store nearly the same thing: an id, owning > state, and a bytea blob. > > Where they differ is scope, and it is the scope that drove this design. Yes, I left all the scope work to the user, so very little beyond basig support for retiring is in the patch. Because we lack reference-counted pg_depend (the possible need for which was also revealed in recent PGQ retraction discussions) I considered this something that should be tied to specific use, not the context table itself. > This series is about TOAST, specifically. The hard problem it solves is > not "store a dictionary," it is "know when a dictionary can be dropped > without a full table scan to prove no value still needs it." In some of my designs I just llet this to the admin, so the sequence would be - lock the context - run migration off the context - call drop_context_if_safe(ditionary, VARIADIC user_columns[]) that was done to allow using the same contexts across tables without making the lifetime tracking frameworks too complex. > The answer > here is column-local ownership: a dictionary belongs to one (relid, > attnum) like a pg_statistic row, values compressed with it never leave > that column in compressed form (heap_fill_tuple marks them external so > every self-contained copy detoasts them), and a rewrite is what retires > the old one. That containment rule is the whole point, and it is TOAST > machinery, not general datum machinery; I don't see how to express it in a > facility that doesn't know about attcompression, detoast, and the rewrite > path. I think the high level lock-migrate-drop pattern still works, but the details can be quite complex > We'll have to consider this, and other use cases if your generalized > method is going to work. > > > And also I used the same for storing quantization tables for pgvector > > int8 and other quantizations [...] like having a public key for writes > > and a more restricted private key for reads, etc. The main point is > > that having a generic facility for these kinds of tasks would benefit > > many use cases. > > This is the part I'd be careful with. Quantization tables, Avro schemas, > and key management each have their own lifecycle and their own "when can > this be dropped" answer, and those answers are not the same as a TOAST > dictionary's. A facility general enough to hold all of them tends to push > the lifecycle decision back onto each caller, which is exactly the part > that was hard to get right here. Yes, I started from the assumption that this belongs to the feature's user. And I decided that until we have more capable dependency tracking, I'll just fall back to manually managing this. > I'm not against the generic facility; I > am wary of making the TOAST feature wait on, or inherit the constraints > of, a design that also has to satisfy pgvector and key rotation. Agreed. For me this is agood use case to test the design :) > > [from the follow-up] At least the underlying storage should allow > > multiple dictionaries for the same relation and moving to newer > > dictionaries or some heuristic for dictionary selection. > > That part we already do, and it's the strongest point of overlap. > pg_toast_dictionary holds many dictionaries per column, each value carries > the 4-byte OID of the one it was compressed with, exactly one is current > (tdcurrent) and new writes use it, and 0005 adds the heuristic you > describe: ANALYZE trains a candidate and adopts it only when a held-out > test says it beats the incumbent. So the "multiple dictionaries, move to > newer ones, select by heuristic" shape is in the series today. Thanks for confirming also other see this popping up in multiple places. > > It also would be nice to have an option for "retiring" a dictionary by > > - setting it read-only, so no new rows can use it > > - having some process re-encode old values (or [...] wait until > > partitions with such values are dropped) > > - then have a verify and drop function > > This is close to what happens, under different names. Marking a > dictionary non-current is your "read-only": no new value uses it, old ones > still read. The re-encode step is a table rewrite (VACUUM FULL, CLUSTER, > REPACK), which recompresses every value with the current dictionary and > then drops the non-current ones. What the series does not have, and your > framing makes a fair case for, is a rewrite-free retirement: a verify-then- > drop that proves no live value references a dictionary without a full > rewrite. I left that out on purpose because proving it needs a full > attribute scan, but a dedicated "verify and drop" function the way you > describe is a reasonable way to expose it, and I'd take it as a follow-up. And I was approaching the same issue from exactly the opposite direction - considering the (VACUUM FULL, CLUSTER, REPACK) to be the prohibitively expensive option for live databases :) If there is a way to query for current dictionary one can do the retiring of the locked ones as a lazy background process instead of one-shot reorg. > So, concretely, where I'd land: > > - The two efforts aren't competitors; this one is the TOAST-specific > consumer, yours is the general facility. If pg_encoding_context lands, > the TOAST dictionary catalog is a natural thing to re-express on top of > it, with the column-local lifecycle kept as a TOAST-side policy. Indeed. I will take a close look at the differences and similarities of the dictionary stores and see how they would map to each other. > - I'd rather not block this series on that re-expression. The containment > and retirement rules are the reviewed-and-tested core here, and they're > TOAST-shaped; Yup, I sent a few more findings, but in general it looks good. > rebasing them onto a generic store is a mechanical change > that can happen once the generic store exists and is agreed on. > - I'm happy to review the generic patch with the TOAST case as a concrete > first consumer, and to see whether the multiple-dictionary and > verify-and-drop pieces are better in the common layer than in TOAST. > > Does that division make sense to you, or do you see the containment rule > fitting into the generic context after all? That's the piece I can't yet > see clearly. Absolutely makes sense. Thanks for taking the time to think about this :) My main takeaway is that we have been spending our focus on different parts of the same issue and, at least I hoe, there is a large overlap that can be used at some point to reduce the complexity of similar features, -- Cheers Hannu
