Hi Greg,
On Mon, Nov 2, 2026 at 7:46 PM Greg Burd <[email protected]> wrote:
> Manu, the v13 containment fixes are unchanged in v14, so your repros
> for them should still hold. The new parts are the header validation
> in 0002 and the UPDATE fast path in 0004, if you want to point your
> harness at them.
They hold: v14 gives the same results as v13, and 4(a) still fails
without Muzzammil's v1 and passes with it.
Header validation: of 1044 corrupt inline headers (4 to 12 bytes, both
forms, va_cmid 0-255), with five readers each under cassert+ASan, none
reads past the datum or fails an assertion. Hannu's toast_save_datum()
point reproduces: va_cmid 3 or >= 6 fails its Assert through UPDATE,
INSERT ... SELECT and CTAS, and his change makes all of them report
"compressed data is corrupt".
UPDATE fast path: 1M unrelated-column UPDATEs of values on a
non-current dictionary stay HOT and take 0.9 s and 128 MB of WAL, where
v13 takes 5.4-5.7 s and 460 MB; VACUUM FULL, CLUSTER, REPACK and REPACK
(CONCURRENTLY) then recompress them and retire the old dictionary.
Among 141 uses of the TOAST predicates and 24 other container paths,
Hannu's check_domain_for_new_field() case is the only one missing; his
one-line change fixes it in the repro. The tables are attached.
Manu
ZSTD TOAST compression, v14: containment, header validation, UPDATE fast path
=============================================================================
Base: master fa5ecff759b + v14-0001..0005 (git am, clean). Builds, all with
lz4 and zstd:
A v14, --enable-cassert, -fsanitize=address
AM A plus Muzzammil Sarwar's v1 for the PL/pgSQL R/W record fast path
N v14, -O2, no asserts
F A plus the two changes Hannu suggested on 02-11 (VARATT_IS_COMPRESSED_DICT
in check_domain_for_new_field(); toast_inline_compression_id() and an
ERRCODE_DATA_CORRUPTED error in toast_save_datum())
v13 v13-0001..0006 on the same base, cassert+ASan and -O2, for comparison
make check (243 tests) and contrib/amcheck pass on A, AM and F; 0 compiler
warnings, 0 ASan reports.
1. The v13 containment repros on v14
------------------------------------
Copies of dictionary-compressed values made while dictionary #1 is current;
then #2 is trained and the owning table rewritten (VACUUM FULL), which deletes
#1; every copy is read back against its md5. On A the output is the same as
on v13, line for line.
A AM
t itself (two zstd_dict, one lz4_dict column) 1003 ok 1003 ok
C1 INSERT ... SELECT into a plain text column 1003 ok 1003 ok
C2 into another table's zstd_dict column 1003 ok 1003 ok
C3 CREATE TABLE AS 1003 ok 1003 ok
C4 whole-row composite stored in a table 1003 ok 1003 ok
C5 array_agg stored 500 ok 500 ok
C6 UPDATE t SET b = a (other dictionary column) 100 ok 100 ok
C7 materialized view, and REFRESH ok ok
C8 partition row movement (300 rows p1 -> p2) 1200 ok 1200 ok
C9 WITH HOLD cursor fetched after the rewrite 1003 ok 1003 ok
C10 procedure, text variables across COMMIT ok ok
C10b procedure, records returned by a PL/pgSQL
function, across COMMIT (4(a)) ERROR (1) ok
C11 logical decoding peeked after the rewrite 1003 ok 1003 ok
C12 pg_statistic MCV and extended MCV ok ok
C13 verify_heapam(check_toast) on the tables 0 rows 0 rows
4(a) A1 DO + DROP TABLE, A2 PROCEDURE + DROP TABLE,
A3/A3r no DROP, other session trains #2 and runs
VACUUM FULL / REPACK ERROR (1) ok
A4 control (column into a text variable) ok ok
4(b) indirect pointer: trigger NEW := make_tuple_
indirect(NEW), test function, WITH HOLD cursor ok (7) ok (7)
4(c) C procedure, SPI_datumTransfer() + SPI_commit() ok (2) ok (2)
REPACK (CONCURRENTLY) with DML committed during the
copy (gdb holds it in copy_table_data()) 2008 ok -
(1) "compression dictionary N does not exist", as on v12 and v13 without v1.
2. Header validation (0002): toast_inline_compression_id()
----------------------------------------------------------
2a. Enumeration, in memory. Each header is handed to the readers by a test C
function that returns the bytes, unchanged, as a text datum in a malloc() block
of exactly its size, so ASan sees any read past the datum's end. va_tcinfo
claims 100 raw bytes; every byte after the header is 0x00.
VARSIZE 4-7 1 case each (the method bits lie past the end) 4
VARSIZE 8 method bits 0, 1, 2, 3 (va_cmid past the end) 4
VARSIZE 9-12 plain form, bits 0, 1, 2 12
long form, va_cmid 0-255 1024
total 1044
Readers: pg_column_compression(), length(), substr(v, 1, 5), octet_length(),
v::text, each in its own session on a UTF8 database.
On A: 0 reads past the datum, 0 failed assertions. Outcomes:
pg_column_ length substr octet_
compression ::text length
VARSIZE 4-7 4 corrupt corrupt corrupt corrupt
VARSIZE 8, plain pglz/lz4 2 method corrupt (2) size
plain form claiming zstd 5 corrupt corrupt corrupt size
VARSIZE 8, long bits 1 corrupt corrupt corrupt size
VARSIZE 9-12, plain pglz/lz4 8 method (2) (2) size
long form, va_cmid 0/1 8 corrupt corrupt corrupt size
long form, va_cmid 2 (zstd) 4 zstd corrupt corrupt size
long form, va_cmid 3 4 corrupt corrupt corrupt size
long form, va_cmid 4/5, < 13 B 8 zstd_dict / corrupt corrupt size
lz4_dict (3) (3)
long form, va_cmid 6-255 1000 (4) (4) (4) size
corrupt = "compressed data is corrupt" or "compressed {pglz,lz4,zstd} data
is corrupt"; size = the raw size in the header (100), as for any complete
header.
(2) pglz/lz4 decode the zero bytes: an error, or "" for a slice and for
the 9-byte lz4 one (a one-byte zero block is valid lz4).
(3) "compressed dictionary data is corrupt".
(4) elog "invalid compression method id N" (XX000).
(3) and (4) are the two items Hannu listed as optional polish; that is what
they give today. On v13 the same run reports a read past the datum for the
8-byte header with the long-form bits. F gives the same table as A.
2b. On disk, and toast_save_datum(). A table (id, w text STORAGE MAIN, v text),
v the last column, ~1.2 kB zstd or ~1.9 kB zstd_dict compressed inline.
Headers patched in the heap file with the server stopped. Then reads, and
three statements that make the toaster push the untouched datum out of line:
ins INSERT INTO h2 SELECT id, <1 kB incompressible>, v FROM h
ctas CREATE TABLE AS SELECT id, <1 kB incompressible> AS w, v FROM h
upd UPDATE h SET w = <1 kB incompressible>
"stored" is what the copy then holds.
zstd source, patch A (cassert) N (no asserts) F
none (control) ok, out of line same same
va_cmid 0 stored as a pglz same ERROR (5)
pointer, reads
"corrupt"
va_cmid 1 as an lz4 same ERROR (5)
pointer, "corrupt"
va_cmid 3 TRAP (6) out of line, method ERROR (5)
NULL, reads "corrupt"
va_cmid 6, 99, 255 TRAP (6) out of line, reads ERROR (5)
"invalid compression
method id N"
VARSIZE 8, long bits stays inline (7) same same
zstd_dict source
none (control) ins/ctas: lz4 (the target's method), upd: kept
VARSIZE 10 ins/ctas: ERROR (3); upd: kept inline as it was
va_cmid 4 -> 5 ins/ctas: "compressed lz4 data is corrupt";
upd: kept, out of line, reads the same error
(5) "compressed data is corrupt" at write time, in all three statements.
(6) failed Assert at toast_internals.c:342, in all three statements (12).
(7) a 4- to 8-byte value is never chosen to go out of line, so truncated
headers do not reach toast_save_datum().
All reads of the patched rows on A and N: an error, no ASan report.
3. UPDATE fast path (0004)
--------------------------
3a. Function, A against v13 (cassert). u: 2000 rows, 3000-byte JSON, inline
zstd_dict (338 B on average), fillfactor 50, all on dictionary #1; then #2
trained. The dictionary of each value is read from the raw datum.
v14 v13
UPDATE u SET n = n + 1, all rows kept on #1 recompressed
2000 HOT to #2, 1999 HOT
then VACUUM FULL / CLUSTER / REPACK / all on the current dictionary, one
REPACK (CONCURRENTLY), each after a dictionary left, md5 of every row ok
new training and unrelated UPDATE (both)
SET n = n + 1, d = d kept on the old one recompressed
SET d = d || '' current current
SET d = <another row's d> current current
BEFORE trigger NEW.d := OLD.d kept on the old one recompressed
BEFORE trigger NEW.d := NEW.d || 'x' current current
UPDATE ... RETURNING d, stored into
another table through a CTE not dictionary-compressed, readable
after the retire (both)
partitions: UPDATE inside p1 kept on p1's old recompressed
UPDATE of the key, 200 rows p1 -> p2 on p2's current dictionary, none on
a p1 dictionary (both)
copies made right after the unrelated
UPDATE (CTAS, whole-row, array_agg,
another zstd_dict column) none dictionary-compressed, all read
back after VACUUM FULL retired #1
log WARNING/ERROR/TRAP, ASan none
3b. Logical replication (A). Publisher: u (primary key) and uf (REPLICA
IDENTITY FULL), 500 rows each on a non-current dictionary. Subscriber: same
tables, zstd_dict, never trained. With the subscription disabled the
publisher runs the unrelated UPDATEs (values kept on the old dictionary),
trains again, runs VACUUM FULL on both tables (both old dictionaries deleted)
and VACUUM pg_toast_dictionary; then the subscription is enabled. The
walsender decodes the UPDATEs without error, both tables match row by row
(id, n, md5), and the subscriber stores the values with plain zstd.
3c. 1M unrelated-column UPDATEs, v14 (N) against v13, both -O2 without
asserts, same base, one dedicated 8-CPU lane, rounds alternating the build
order, lane calibrated clean before and after. u: 1,000,000 rows, inline
zstd_dict (331 B on average), fillfactor 50, 742 MB; each run starts from a
fresh VACUUM FULL, then CHECKPOINT, then one UPDATE u SET n = n + 1.
wal_level = replica, full_page_writes = off (WAL is record bytes only).
values on the current dictionary v13 v14
time, 3 rounds (ms) 984, 943, 988 1029, 885, 902
WAL 128.4 MB each 128.4 MB each
HOT 1,000,000 1,000,000
values on a non-current one
time, 3 rounds (ms) 5701, 5399, 5743 876, 868, 940
WAL (MB) 459.7, 460.0, 460.9 128.4 each
HOT 999,076 998,093 1,000,000 each
998,992
With the current dictionary the two are the same within the spread. With a
non-current one, v13 recompresses every value: 6.5x the time, 3.6x the WAL.
ZSTD TOAST compression, v14: audit of the containment checks
============================================================
Tree: master fa5ecff759b + v14-0001..0005. File:line below are in that tree.
The table was written from reading the code, before any of the tests in the
last section were run.
What is audited. 0004 keeps a dictionary datum inside its column with two
mechanisms: fill_val() sets HEAP_HASEXTERNAL for a tuple that holds one
(heaptuple.c:372-374), so everything that already flattens a tuple with TOAST
pointers also reaches it, and toast_tuple_init() decompresses it when it is
stored anywhere but its own column (toast_helper.c:205-221). The toaster does
not look inside composite, array or range values, so a dictionary datum can
escape where code decides "is this value self-contained?" by testing ONE datum
with VARATT_IS_EXTERNAL() instead of the tuple flag, and then keeps or copies
it. Hannu's check_domain_for_new_field() finding is of that kind.
Scope: every line in src/ and contrib/ (without src/test) that uses
VARATT_IS_EXTERNAL[_ONDISK|_INDIRECT|_EXPANDED|_NON_EXPANDED](),
VARATT_IS_EXTENDED(), VARATT_IS_COMPRESSED() or HeapTuple[Header]HasExternal():
141 lines in 47 files, grouped below by condition or function, plus 24 places
that decide self-containment without those macros.
OK = handles dictionary datums (how). NA = not a containment decision (why).
MISSING = takes the wrong branch for a dictionary datum.
A. Composite values and expanded records
MISSING expandedrecord.c:1553 check_domain_for_new_field: the new value
is tested with VARATT_IS_EXTERNAL() only, so dummy_erh is flattened
for the domain check without inlining it (Hannu, 02-11)
OK expandedrecord.c:472, 523 expanded_record_set_tuple: tuple flag
OK expandedrecord.c:707 ER_get_flat_size: VARATT_IS_COMPRESSED_DICT
OK expandedrecord.c:1160, 1191 expanded_record_set_field_internal: same
OK expandedrecord.c:1307, 1320 expanded_record_set_fields: same
OK expandedrecord.c:1523 check_domain_for_new_field, old fields: copies
erh's flag
OK expandedrecord.c:1617 check_domain_for_new_tuple: tuple flag
OK execTuples.c:2517 HeapTupleHeaderGetDatum: flag -> flatten
OK heaptuple.c:1009 heap_copy_tuple_as_datum: flag -> flatten
OK heaptuple.c:328-330 fill_val: sets HEAP_HASEXTERNAL (:372-374)
OK heaptoast.c:378 toast_flatten_tuple: VARATT_IS_COMPRESSED_DICT
OK heaptoast.c:493-494 toast_flatten_tuple_to_datum: external or
compressed -> detoast_attr
OK heaptoast.c:594 toast_build_flattened_tuple: as :378
NA expandedrecord.c:634 Assert (composite datums are flat)
NA execExprInterp.c:3757 FieldSelect on an expanded record (a read)
NA heaptuple.c:249 heap_compute_data_size: expanded only
NA pl_exec.c:836, 1092, 7698 expanded-object checks
B. Values that outlive the transaction or the lock
OK pl_exec.c:8920 assign_simple_var (non-atomic)
OK tstoreReceiver.c:163 WITH HOLD cursors
OK datum.c:201 datumTransfer / SPI_datumTransfer (v13)
OK detoast.c:51-59, 67-91, 114 detoast_external_attr: on disk, indirect
(v13) and inline dictionary datums
OK catcache.c:2198 tuple flag (and catalogs cannot use these
methods)
NA detoast.c:101 expanded object, flattened
NA datum.c:144 datumCopy: verbatim, as a TOAST pointer;
in-transaction copies
NA datum.c:474, 523 datumSerialize: workers of the same query
NA execTuples.c:199, 235 slot materialization: expanded only
NA expandeddatum.c:34; arrayfuncs.c:1852, 2273; array_expanded.c:76, 410
expanded-form dispatch (elements: section E)
C. Storing into a table, an index, or another table's TOAST
OK toast_helper.c:143 toast_tuple_init: a foreign external value is
fetched with detoast_external_attr, which
decompresses it
OK toast_helper.c:168-174 toast_dictionary_keep(); the UPDATE fast path
needs an old tuple of the same relation, no
relrewrite, and byte-identical datums
OK heapam.c:2263-2266, 3848-3854 insert / update: tuple flag -> toaster
OK heapam.c:9381, 9434 ExtractReplicaIdentity: flag -> flatten
OK rewriteheap.c:616-619 raw_heap_insert: flag -> toaster (relrewrite)
OK indextuple.c:108 index_form_tuple: decompresses (0004)
OK indextuple.c:121 recompression without a rel: plain zstd
(toast_internals.c:89-93), although
index.c:360 copies attcompression
OK brin_tuple.c:209, 221 as indextuple.c
OK repack.c:3101; pgrepack.c:214, 256-275 REPACK (CONCURRENTLY): the
dictionary datum travels inside the tuple and
reaches toast_tuple_init with relrewrite set
NA toast_helper.c:79-82, 249-252, 383 unchanged external value of the same
relation; size heuristics; deletion
NA heapam.c:3183-3185, 4639, 6526 deletion; has_external for replica
identity (a dictionary datum is inline, so in
WAL already); speculative abort
NA toast_internals.c:332 toast_save_datum: keep() ran before; header
validation is Hannu's point 2 (matrix, 2b)
NA toast_internals.c:53, 103-104, 241, 295, 569; heaptoast.c:720;
repack.c:3047 Asserts, deletion, slice fetch, indirect
D. Reading, reporting, decoding
NA detoast.c:149-199, 257-325, 388, 445, 519, 560, 617-640, 678-701
detoast_attr[_slice], sizes, preconditions
NA fmgr.c:1800, 1809, 1832 pg_detoast_datum*: decompress everything
NA varlena.c:716-717, 4292 text_substring; a reporting function
NA toast_compression.c:525, 534 toast_get_compression_id (reporting)
NA reorderbuffer.c:5187, 5306, 5345; proto.c:815; pgoutput.c:1405-1406;
test_decoding.c:594 decoding: the datum is inline in WAL, and
plugins decompress under the historic
snapshot, which still sees a retired
dictionary while the slot holds catalog_xmin
NA inv_api.c:142; ts_selfuncs.c:238 pg_largeobject is a catalog; Assert
NA verify_heapam.c:1621, 1751-1755, 1794; verify_nbtree.c:2873-2892;
pageinspect heapfuncs.c:371-373 checks; index tuples never hold one
NA toast_compression.h:93, 123; detoast.h:28, 69; htup_details.h:576, 753;
pl_exec.c:8726; heaptoast.c:346 definitions, comments
E. Places that decide without the macros
OK arrayfuncs.c:327, 1526, 2271, 3296, 3538, 6216, 6471-6473
array_in, array_recv, array_set_element,
array_map, construct_md_array, array_fill,
array_replace: PG_DETOAST_DATUM
OK arrayfuncs.c:5400 accumArrayResult: PG_DETOAST_DATUM_COPY
OK arrayfuncs.c:2501 array_set_element_expanded: only caller is
:2276, after :2271
OK rangetypes.c:2033, 2043 range_serialize: PG_DETOAST_DATUM_PACKED, so
the :2960 guard (toast pointers only) is never
reached by a dictionary datum; multiranges
store serialized ranges
OK spgdoinsert.c:1959, 1973 SP-GiST leaf and INCLUDE values
OK analyze.c:2243, 2590; mcv.c:764; extended_stats.c:1148 statistics
OK makefuncs.c:366; clauses.c:5995 makeConst, evaluate_expr
OK spi.c:1097; execTuples.c:2017 SPI_returntuple,
ExecFetchSlotHeapTupleDatum: tuple flag
NA tuplestore.c, tuplesort.c verbatim within the query; WITH HOLD goes
through tstoreReceiver.c:163
NA postgres_fdw.c:4591, 4622, 5198; dblink; copyto.c:526; logical
replication values sent through output/send functions
Result from reading: 1 MISSING (expandedrecord.c:1553), none other.
Tests (run after the table was written)
---------------------------------------
One inline zstd_dict value (326 B) taken through each path while its
dictionary is current; then (A) a second dictionary trained and VACUUM FULL on
the source, which deletes the first, and (B) DROP TABLE on the source; every
copy read back against the value's md5. v14 = cassert+ASan; v14+F = the same
with Hannu's two suggested changes. "stored" is how the copied value, or the
field / element inside it, is stored.
path stored v14 after A, B v14+F
ROW(d, 1)::comp_t plain ok, ok ok, ok
whole-row Var plain ok, - (1) ok, - (1)
UPDATE ... SET v.a = d plain ok, ok ok, ok
SQL function returning a row plain ok, ok ok, ok
PL/pgSQL RETURN record plain ok, ok ok, ok
record field := d plain ok, ok ok, ok
SELECT * INTO record plain ok, - (1) ok, - (1)
domain over composite,
field := d, CHECK stores
VALUE (Hannu's case) zstd_dict ERROR (2), ERROR plain; ok, ok
same, SELECT ... INTO plain ok, ok ok, ok
same, whole ROW() assigned plain ok, ok ok, ok
domain over text, CHECK
stores VALUE lz4 ok, ok ok, ok
ARRAY[d], array_agg,
a[1] := d, array_append,
array_fill, array_replace plain ok, ok (each) ok, ok (each)
range and multirange over
text plain ok, ok ok, ok
ADD COLUMN ... DEFAULT f()
(attmissingval) plain ok, ok ok, ok
COPY BINARY out and in lz4 ok, ok ok, ok
postgres_fdw INSERT
(loopback) lz4 ok, ok ok, ok
dblink_exec INSERT lz4 ok, ok ok, ok
btree index on the column,
index-only scan zstd (3) - -
(1) the column has the source's row type, so DROP TABLE ... CASCADE drops it.
(2) "compression dictionary N does not exist".
(3) index_form_tuple() decompresses it and recompresses without a rel.
No failed assertion and no ASan report in either build.
The rest of the paths in the table are covered by the containment matrix
(holdable cursors, procedure variables, SPI_datumTransfer, indirect pointers,
logical decoding, statistics, partitions, REPACK (CONCURRENTLY)).
