Hi Greg,
On Sat, Nov 14, 2026 at 1:50 PM Greg Burd <[email protected]> wrote:
> Most of the changes are in 0001 and 0002, since those are the ones
> closest to being committed.
I found nothing broken in 0001 or 0002, and one check in 0002 that no
test exercises.
0001: reading pglz and lz4 values through text, substr(),
pg_column_*(), verify_heapam() and test_decoding gives byte-identical
output (58,990 lines) on master and master+0001, and each of four wrong
conversions I planted changes that output.
0002: on 0001-0003 under cassert, reverting any one check in
toast_compression_id_resolve() fails t/007, except method_byte ==
TOAST_INVALID_COMPRESSION_ID. From 0004 on, with the last ID at 5, that
check alone keeps a long pointer with byte 3 away from the Assert in
toast_compression_id_needs_cmid_byte(), and t/007 has no byte-3 case.
A byte-3 row in the long-form part of t/007 would cover it.
Old data reads the same with a master data directory started under
0001-0003, and after pg_upgrade from REL_18 and REL_19 to 0001-0003.
The scripts and the results, forged pointers included, are attached.
Manu
Checks of v17-0001 and v17-0002 (with 0003-0005 where noted)
on master 77bba5454f9, --with-lz4 --with-zstd
Builds: cassert -O2, and cassert + ASan -O1. With the cassert -O2
build each prefix (0001, 0001-0002, ... 0001-0005) builds without
warnings and passes make check, and 0001-0004 passes without 0005.
1. 0001 changes no behavior
toast-oracle.sh.txt fills a cluster with pglz and lz4 values (out of
line compressed and uncompressed, in line compressed; text and bytea;
sizes 1992 and 1996 bytes +-2 times 1 to 3, and up to 1 MB) and reads
them with md5(text), length(), octet_length(), substr() and substring()
at offsets and lengths around the chunk size, left(), pg_column_size(),
pg_column_compression(), pg_column_toast_chunk_id(),
verify_heapam(check_toast => true) and test_decoding (INSERT, UPDATE,
DELETE with REPLICA IDENTITY FULL). 58,990 output lines.
master vs master+0001, cassert build: identical
master vs master+0001, ASan build: identical, no ASan report
Each of these wrong conversions, put on top of 0001, changes the output
(number of lines that differ from master):
toast_datum_size() returns rawsize - VARHDRSZ, not extsize 57
amcheck keeps rawsize - VARHDRSZ as the saved size 192
toast_fetch_datum_slice() treats every value as uncompressed 12033
toast_external_info_get() keeps the method bits of an
uncompressed value instead of TOAST_INVALID_COMPRESSION_ID 113
2. 0002: which test fails when one check is reverted
On 0001-0003, cassert. t/007 is amcheck/t/007_verify_toast_pointer.pl.
toast_compression_id_resolve(), plain form:
method_bits == VARLENA_COMPRESS_METHOD_LONG t/007 (the
Assert in toast_compression_id_needs_cmid_byte() fires)
needs_cmid_byte(method_bits) t/007, 2 tests
toast_compression_id_resolve(), long form:
method_bits != VARLENA_COMPRESS_METHOD_LONG t/007, 2 tests
method_byte == TOAST_INVALID_COMPRESSION_ID none
method_byte > TOAST_LAST_COMPRESSION_ID t/007, 2 tests
!needs_cmid_byte(method_byte) t/007, 2 tests
toast_external_info_get(): not compressed -> no method
make check
(4 tests) and
amcheck check_heap
toast_get_compression_id(): ERROR for a compressed
pointer with no method t/007, 5 tests
The byte == TOAST_INVALID_COMPRESSION_ID test cannot matter on
0001-0003, where 3 is above the last ID (2). From 0004 on the last ID
is 5, and it is the only thing that rejects a long pointer with byte 3
before toast_compression_id_needs_cmid_byte() asserts on it. On
0001-0005 with that test removed, t/007 still passes, and the byte-3
row of forge-pointers.sh.txt gives (section 3):
TRAP: failed Assert("cmid != TOAST_INVALID_COMPRESSION_ID"),
File: "src/include/access/toast_compression.h", Line: 76
(8 TRAP lines: one per reader of that row, and one for verify_heapam()
on the zstd table, which then returns nothing; the readers of the
other 268 rows give the same results as in section 3)
Without asserts the function would return 3, which is
TOAST_INVALID_COMPRESSION_ID anyway, so only assert-enabled builds see
it. A byte-3 case in the long-form part of t/007 would cover it.
3. Forged pointers
forge-pointers.sh.txt rewrites, with the server stopped, the method
bits, the method byte or the saved size of one pointer per row, and
then reads every row. On 0001-0003 and on 0001-0005, cassert + ASan:
269 rows, no crash, no ASan report, no TRAP. On 0001-0005:
zstd, method byte 0,1,3,6..255 (253 rows)
amcheck: has a long TOAST pointer with invalid compression method
id <the byte>
pg_column_compression(): ERROR: compressed data is corrupt
text, length(), substr(), left(): the value, intact
zstd, method byte 4 or 5
amcheck: nothing
pg_column_compression(): zstd_dict / lz4_dict
text, length(), substr(), left(): the value, intact
zstd, method bits 0, 1, 2
amcheck: has a long TOAST pointer but compression method bits <N>
pg_column_compression(): ERROR: compressed data is corrupt
text, length(), substr(), left(): the value, intact
zstd, saved size = rawsize - 4, or = rawsize ("not compressed")
amcheck: ... is not compressed but has a long TOAST pointer
(plus the chunk size messages)
pg_column_compression(): NULL
reads: ERROR: unexpected chunk size
zstd, saved size -1 or +1
amcheck and reads: unexpected chunk size
pglz and lz4, method bits 2 or 3
amcheck: has invalid compression method id 2 / 3
pg_column_compression(): ERROR: compressed data is corrupt
text, length(), substr(), left(): the value, intact
On 0001-0003 the same, except that bytes 4 and 5 are rejected like
the other unknown bytes.
So the pointer checks are seen by amcheck and pg_column_compression()
only: the data readers take the method from the header stored with the
data, and return the value whatever the pointer says. The one case no
check catches is a long pointer naming the other long-form methods
(byte 4 or 5 on a plain zstd value); amcheck already reads every chunk
of the value and could compare the pointer's method with the stored
header's. I have not tried that.
4. Old data
upgrade-check.sh.txt, with toast-oracle.sh.txt:
master data directory, read by master, then by 0001-0002 and by
0001-0003 (same catalog version): identical, 58,990 lines
REL_18_STABLE (18.7) and REL_19_STABLE (19rc2), release builds,
pg_upgrade --copy to 0001-0003: identical, 58,752 lines
(no logical slot here: pg_upgrade refuses one with pending changes)
then zstd values written next to the old ones: verify_heapam() finds
nothing in any table, and the new values read back.
Both for the cassert and the ASan builds of 0001-0002 / 0001-0003.
#!/bin/bash
# toast-oracle.sh BINDIR_PARENT OUTFILE [fill|read|both] [DATADIR]
# BINDIR_PARENT is an install prefix (the directory holding bin/).
# Differential oracle for "0001 changes no behavior": builds a corpus of
# TOAST values and reads every one of them through every reader we know of,
# writing one line per (value, reader) to OUTFILE. Two builds are compared
# with cmp on their OUTFILEs, e.g.
# bash toast-oracle.sh.txt /path/to/master out-master.txt
# bash toast-oracle.sh.txt /path/to/master+0001 out-0001.txt
# cmp out-master.txt out-0001.txt
# both : initdb, fill, read, stop (default)
# fill : initdb + fill only, leave the cluster stopped in DATADIR
# read : start an existing DATADIR (e.g. after pg_upgrade), read, stop
# Corpus: pglz and lz4, external compressed, external uncompressed
# (STORAGE EXTERNAL), inline compressed (STORAGE MAIN), text and bytea,
# sizes around the TOAST chunk size (1996 and 1992 bytes, +-2, times 1..3).
# NOSLOT=1 skips the logical slot (pg_upgrade refuses a slot with pending WAL).
set -eu
I=$1; OUT=$2; MODE=${3:-both}
D=${4:-$(mktemp -d ${TMPDIR:-/tmp}/oracle.XXXXXX)}
P=${P:-$((61000 + RANDOM % 900))}
export ASAN_OPTIONS=detect_leaks=0:abort_on_error=1:log_path=$D/asan
Q() { "$I/bin/psql" -X -qAt -h $D -p $P -U postgres -d postgres -v
ON_ERROR_STOP=1 "$@"; }
start() { "$I/bin/pg_ctl" -D $D/data -l $D/log -w -t 120 -o "-p $P -k $D -c
listen_addresses=''" start >/dev/null; }
stop() { "$I/bin/pg_ctl" -D $D/data -m fast -w stop >/dev/null; }
fill() {
"$I/bin/initdb" -D $D/data -A trust --no-sync --no-data-checksums -U postgres
-E UTF8 --locale=C >/dev/null
cat >> $D/data/postgresql.conf <<CONF
shared_buffers = 64MB
max_connections = 20
autovacuum = off
jit = off
wal_level = logical
max_replication_slots = 4
fsync = off
CONF
start
Q ${NOSLOT:+-v noslot=1} <<'SQL'
CREATE EXTENSION amcheck;
-- deterministic, poorly compressible filler of exactly n bytes
CREATE FUNCTION noise(seed int, n int) RETURNS text LANGUAGE sql IMMUTABLE AS $$
SELECT substr(string_agg(md5((seed::bigint * 100003 + i)::text), ''), 1, n)
FROM generate_series(0, n / 32 + 1) i $$;
-- compressible: a short alphabet, repeated with drift
CREATE FUNCTION soft(seed int, n int) RETURNS text LANGUAGE sql IMMUTABLE AS $$
SELECT substr(string_agg(repeat(chr(97 + (seed + i) % 7), 1 + i % 5) || (i %
97)::text, ''), 1, n)
FROM generate_series(0, n) i $$;
CREATE TABLE sizes(n int);
INSERT INTO sizes SELECT k * c + d FROM generate_series(1, 3) k,
(VALUES (1996), (1992)) cs(c), generate_series(-2, 2) d;
INSERT INTO sizes VALUES (1), (100), (2000), (8000), (8192), (65536), (200000),
(1048576);
CREATE TABLE c (id serial, kind text, n int,
tp text COMPRESSION pglz, tl text COMPRESSION lz4,
bp bytea COMPRESSION pglz, bl bytea COMPRESSION lz4);
ALTER TABLE c ALTER tp SET STORAGE EXTENDED, ALTER tl SET STORAGE EXTENDED;
CREATE TABLE u (id serial, kind text, n int, t text, b bytea);
ALTER TABLE u ALTER t SET STORAGE EXTERNAL, ALTER b SET STORAGE EXTERNAL;
CREATE TABLE m (id serial, kind text, n int, tp text COMPRESSION pglz, tl text
COMPRESSION lz4);
ALTER TABLE m ALTER tp SET STORAGE MAIN, ALTER tl SET STORAGE MAIN;
\if :{?noslot}
\else
SELECT 'slot' FROM pg_create_logical_replication_slot('o', 'test_decoding');
\endif
-- external compressed (large soft values), and poorly compressible ones
INSERT INTO c (kind, n, tp, tl, bp, bl)
SELECT 'soft', n, soft(n, n), soft(n + 1, n), convert_to(soft(n + 2, n),
'UTF8'),
convert_to(soft(n + 3, n), 'UTF8') FROM sizes;
INSERT INTO c (kind, n, tp, tl, bp, bl)
SELECT 'noise', n, noise(n, n), noise(n + 1, n), convert_to(noise(n + 2, n),
'UTF8'),
convert_to(noise(n + 3, n), 'UTF8') FROM sizes;
-- external uncompressed
INSERT INTO u (kind, n, t, b)
SELECT 'noise', n, noise(n + 4, n), convert_to(noise(n + 5, n), 'UTF8') FROM
sizes;
INSERT INTO u (kind, n, t, b)
SELECT 'soft', n, soft(n + 4, n), convert_to(soft(n + 5, n), 'UTF8') FROM
sizes;
-- inline compressed (MAIN keeps them in line while they fit)
INSERT INTO m (kind, n, tp, tl)
SELECT 'soft', n, soft(n + 6, n), soft(n + 7, n) FROM sizes WHERE n <= 8192;
ALTER TABLE c REPLICA IDENTITY FULL;
UPDATE c SET n = n WHERE id % 3 = 0;
DELETE FROM c WHERE id % 7 = 0;
CHECKPOINT;
SQL
}
read_all() {
local
offs="0,1,1990,1991,1992,1995,1996,1997,3983,3984,3991,3992,3993,5987,5988,5989,100000"
local lens="1,2,4,100,1996,1997,4000"
Q <<SQL > $OUT
\\pset tuples_only on
-- every column of every table: text-level readers
WITH v AS (
SELECT 'c' t, id, kind, n, 'tp' col, tp::bytea x, tp::text s FROM c UNION ALL
SELECT 'c', id, kind, n, 'tl', tl::bytea, tl FROM c UNION ALL
SELECT 'u', id, kind, n, 't', t::bytea, t FROM u UNION ALL
SELECT 'm', id, kind, n, 'tp', tp::bytea, tp FROM m UNION ALL
SELECT 'm', id, kind, n, 'tl', tl::bytea, tl FROM m)
SELECT format('%s.%s id=%s %s n=%s md5=%s len=%s oct=%s', t, col, id, kind, n,
md5(s),
length(s), octet_length(s)) FROM v ORDER BY t, col, id;
SELECT format('size c id=%s tp=%s/%s/%s tl=%s/%s/%s bp=%s/%s/%s bl=%s/%s/%s',
id,
pg_column_size(tp), pg_column_compression(tp), pg_column_toast_chunk_id(tp)
IS NOT NULL,
pg_column_size(tl), pg_column_compression(tl), pg_column_toast_chunk_id(tl)
IS NOT NULL,
pg_column_size(bp), pg_column_compression(bp), pg_column_toast_chunk_id(bp)
IS NOT NULL,
pg_column_size(bl), pg_column_compression(bl), pg_column_toast_chunk_id(bl)
IS NOT NULL)
FROM c ORDER BY id;
SELECT format('size u id=%s t=%s/%s/%s b=%s/%s/%s', id,
pg_column_size(t), pg_column_compression(t), pg_column_toast_chunk_id(t) IS
NOT NULL,
pg_column_size(b), pg_column_compression(b), pg_column_toast_chunk_id(b) IS
NOT NULL)
FROM u ORDER BY id;
SELECT format('size m id=%s tp=%s/%s/%s tl=%s/%s/%s', id,
pg_column_size(tp), pg_column_compression(tp), pg_column_toast_chunk_id(tp)
IS NOT NULL,
pg_column_size(tl), pg_column_compression(tl), pg_column_toast_chunk_id(tl)
IS NOT NULL)
FROM m ORDER BY id;
-- chunk ids: equal across builds when the corpus is filled the same way
SELECT format('chunkid c id=%s %s %s %s %s', id, pg_column_toast_chunk_id(tp),
pg_column_toast_chunk_id(tl), pg_column_toast_chunk_id(bp),
pg_column_toast_chunk_id(bl))
FROM c ORDER BY id;
-- substr / bytea substring at chunk-boundary offsets and lengths
SELECT format('substr c.%s id=%s o=%s l=%s %s', col, id, o, l, md5(x))
FROM (SELECT id, 'tp' col, o, l, substr(tp, o, l)::bytea x FROM c,
unnest('{$offs}'::int[]) o, unnest('{$lens}'::int[]) l UNION ALL
SELECT id, 'tl', o, l, substr(tl, o, l)::bytea FROM c,
unnest('{$offs}'::int[]) o, unnest('{$lens}'::int[]) l UNION ALL
SELECT id, 'bp', o, l, substring(bp FROM o FOR l) FROM c,
unnest('{$offs}'::int[]) o, unnest('{$lens}'::int[]) l UNION ALL
SELECT id, 'bl', o, l, substring(bl FROM o FOR l) FROM c,
unnest('{$offs}'::int[]) o, unnest('{$lens}'::int[]) l) s
ORDER BY col, id, o, l;
SELECT format('substr u.%s id=%s o=%s l=%s %s', col, id, o, l, md5(x))
FROM (SELECT id, 't' col, o, l, substr(t, o, l)::bytea x FROM u,
unnest('{$offs}'::int[]) o, unnest('{$lens}'::int[]) l UNION ALL
SELECT id, 'b', o, l, substring(b FROM o FOR l) FROM u,
unnest('{$offs}'::int[]) o, unnest('{$lens}'::int[]) l) s
ORDER BY col, id, o, l;
SELECT format('substr m.%s id=%s o=%s l=%s %s', col, id, o, l, md5(x))
FROM (SELECT id, 'tp' col, o, l, substr(tp, o, l)::bytea x FROM m,
unnest('{$offs}'::int[]) o, unnest('{$lens}'::int[]) l UNION ALL
SELECT id, 'tl', o, l, substr(tl, o, l)::bytea FROM m,
unnest('{$offs}'::int[]) o, unnest('{$lens}'::int[]) l) s
ORDER BY col, id, o, l;
-- prefix slices (pglz's partial-fetch path): left()
SELECT format('left c id=%s l=%s %s %s', id, l, md5(left(tp, l)), md5(left(tl,
l)))
FROM c, unnest('{1,10,1000,1996,2000,5000}'::int[]) l ORDER BY id, l;
SELECT format('amcheck %s %s %s', r, coalesce(blkno::text, '-'), coalesce(msg,
'none'))
FROM (SELECT 'c' r, * FROM verify_heapam('c', check_toast => true) UNION ALL
SELECT 'u', * FROM verify_heapam('u', check_toast => true) UNION ALL
SELECT 'm', * FROM verify_heapam('m', check_toast => true)) s;
SELECT format('amcheck-count %s', (SELECT count(*) FROM verify_heapam('c',
check_toast => true))
+ (SELECT count(*) FROM verify_heapam('u', check_toast => true))
+ (SELECT count(*) FROM verify_heapam('m', check_toast => true)));
SQL
# logical decoding of the fill (INSERT, UPDATE, DELETE with REPLICA IDENTITY
FULL):
# reassembles TOAST chunks in the reorder buffer. Values are hashed per
change.
if Q -c "SELECT 1 FROM pg_replication_slots WHERE slot_name = 'o'" | grep -q
1; then
Q -c "SELECT format('decode %s %s %s', row_number() OVER (), length(data),
md5(data))
FROM pg_logical_slot_peek_changes('o', NULL, NULL, 'include-xids',
'0')" >> $OUT
fi
}
case $MODE in
both) fill; read_all; stop ;;
fill) fill; stop ;;
read) start; read_all; stop ;;
esac
echo "lines=$([ -f $OUT ] && wc -l < $OUT || echo -) asan=$(cat $D/asan.*
2>/dev/null | grep -c AddressSanitizer || true) traps=$(grep -c -E 'TRAP|PANIC'
$D/log || true) datadir=$D"
[ $MODE = both ] && rm -rf $D || true
#!/bin/bash
# forge-pointers.sh PREFIX OUT.tsv
# Forged on-disk TOAST pointers of the forms the series decodes. PREFIX is
# an install prefix (holding bin/), built with zstd; use an assert-enabled
# build. Only the method bits of va_extinfo, the method byte of the long
# form and the saved size are rewritten, with the server stopped.
# Cases, one row and one external value each:
# zstd (long pointer): untouched; method byte 0..255 except 2; method
# bits 0, 1, 2; saved size -1 and +1; saved size = rawsize - 4 and
# = rawsize (the pointer then says "not compressed")
# pglz, lz4 (plain pointer): untouched; method bits 2 and 3
# For each: verify_heapam(check_toast => true), then pg_column_compression(),
# length(), octet_length(), md5(v), substr(v, 1, 5), substr(v, 3000, 10) and
# left(v, 100), each in its own psql; md5s taken before forging tell
# "intact" from "WRONG". A backend crash shows as CRASH.
set -eu
I=$1; OUT=$2
D=$(mktemp -d ${TMPDIR:-/tmp}/forge.XXXXXX); P=$((61000 + RANDOM % 900))
trap '"$I/bin/pg_ctl" -D $D/data -m immediate -w stop >/dev/null 2>&1 || true;
rm -rf $D' EXIT
export ASAN_OPTIONS=detect_leaks=0:abort_on_error=0:log_path=$D/asan
Q() { "$I/bin/psql" -X -qAt -h $D -p $P -U postgres -d postgres -v
ON_ERROR_STOP=1 "$@"; }
"$I/bin/initdb" -D $D/data -A trust --no-sync --no-data-checksums -U postgres
-E UTF8 --locale=C >/dev/null
printf "port = $P\nunix_socket_directories = '$D'\nlisten_addresses =
''\nautovacuum = off\nrestart_after_crash = on\n" >> $D/data/postgresql.conf
"$I/bin/pg_ctl" -D $D/data -l $D/log -w start >/dev/null
python3 -I - > $D/cases.txt <<'PY'
c = [('z', 'ctl', 0)] + [('z', 'byte', b) for b in range(256) if b != 2]
c += [('z', 'bits', b) for b in (0, 1, 2)]
c += [('z', k, 0) for k in ('ext_minus1', 'ext_plus1', 'ext_eq_raw',
'ext_gt_raw')]
for m in 'pl':
c += [(m, 'ctl', 0), (m, 'bits', 2), (m, 'bits', 3)]
for i, (m, k, a) in enumerate(c, 1):
print(i, m, k, a)
PY
Q <<'SQL'
CREATE EXTENSION amcheck;
CREATE EXTENSION pageinspect;
CREATE TABLE cases (id int, m text, kind text, arg int);
CREATE TABLE h (id int, hv text, h1 text, h2 text, h3 text);
CREATE FUNCTION val(i int) RETURNS text LANGUAGE sql IMMUTABLE AS $$
SELECT (SELECT string_agg(md5((i * 1000 + g)::text), '') FROM
generate_series(1, 60) g)
|| repeat('x', 4000) $$;
CREATE TABLE fz (id int, v text COMPRESSION zstd) WITH (toast_tuple_target =
128);
CREATE TABLE fp (id int, v text COMPRESSION pglz) WITH (toast_tuple_target =
128);
CREATE TABLE fl (id int, v text COMPRESSION lz4) WITH (toast_tuple_target =
128);
SQL
Q -c "\\copy cases FROM '$D/cases.txt' WITH (DELIMITER ' ')"
for m in z p l; do
Q -c "INSERT INTO f$m SELECT id, val(id) FROM cases WHERE m = '$m' ORDER BY
id"
Q -c "INSERT INTO h SELECT id, md5(v), md5(substr(v, 1, 5)), md5(substr(v,
3000, 10)), md5(left(v, 100)) FROM f$m"
# tuple data = int4 id, then the pointer: 0x01, vartag, va_rawsize,
# va_extinfo, va_valueid, va_toastrelid, [method byte]
Q -c "COPY (SELECT '$m', pg_relation_filepath('f$m'), b * 8192 + lp_off +
t_hoff,
encode(substr(t_data, 1, 4), 'hex'), get_byte(t_data, 4),
get_byte(t_data, 5),
encode(substr(t_data, 7, 8), 'hex')
FROM generate_series(0, pg_relation_size('f$m') / 8192 - 1) b,
heap_page_items(get_raw_page('f$m', b::int)) WHERE lp_len > 0) TO
STDOUT" >> $D/where.txt
done
Q -c CHECKPOINT
"$I/bin/pg_ctl" -D $D/data -m fast -w stop >/dev/null
python3 -I - $D/cases.txt $D/where.txt $D/data <<'PY'
import struct, sys
cases = {}
for line in open(sys.argv[1]):
i, m, k, a = line.split(); cases[int(i)] = (m, k, int(a))
for line in open(sys.argv[2]):
m, path, off, idhex, b4, tag, rawext = line.split()
i = struct.unpack('<i', bytes.fromhex(idhex))[0]
cm, k, a = cases[i]
assert cm == m and b4 == '1' and int(tag) == (19 if m == 'z' else 18), line
base = int(off) + 4 + 2
raw, ext = struct.unpack('<iI', bytes.fromhex(rawext))
size, bits = ext & 0x3FFFFFFF, ext >> 30
if k == 'ctl':
continue
if k == 'byte':
at, new = base + 16, bytes([a])
elif k == 'bits':
at, new = base + 4, struct.pack('<I', size | (a << 30))
else:
n = {'ext_minus1': size - 1, 'ext_plus1': size + 1,
'ext_eq_raw': raw - 4, 'ext_gt_raw': raw}[k]
at, new = base + 4, struct.pack('<I', n | (bits << 30))
with open(sys.argv[3] + '/' + path, 'r+b') as f:
f.seek(at); f.write(new)
PY
"$I/bin/pg_ctl" -D $D/data -l $D/log -w start >/dev/null
# wait until the server takes connections (a crash in a step restarts it)
up() { until "$I/bin/psql" -X -qAt -h $D -p $P -U postgres -d postgres -c
'SELECT 1' >/dev/null 2>&1; do sleep 0.2; done; }
for m in z p l; do
up
Q -c "COPY (SELECT x.id, regexp_replace(v.msg, 'toast value \\d+', 'toast
value N')
FROM verify_heapam('f$m', check_toast => true) v
JOIN (SELECT id, ctid FROM f$m) x
ON (x.ctid::text::point)[0] = v.blkno AND
(x.ctid::text::point)[1] = v.offnum
ORDER BY 1, 2) TO STDOUT" 2>&1 || echo "0 amcheck on f$m: ERROR"
done > $D/amcheck.txt
probe() { # one SQL in its own psql; last line, the ERROR, or CRASH
local o
up # a crash in an earlier step leaves the server in recovery for a moment
o=$("$I/bin/psql" -X -qAt -h $D -p $P -U postgres -d postgres -c "$1" 2>&1)
|| true
if echo "$o" | grep -q -E 'server closed the connection|connection to
server'; then
echo CRASH
elif echo "$o" | grep -q 'ERROR:'; then echo "$o" | grep -m1 'ERROR:' | sed
-E 's/.*ERROR: +/ERROR: /; s/[0-9]{5,}/N/g'
else echo "$o" | tail -1; fi
}
printf
'id\tmethod\tforged\targ\tamcheck\tpg_column_compression\tlength\toctet_length\tmd5\tsubstr_1_5\tsubstr_3000_10\tleft_100\n'
> $OUT
while read i m k a; do
am=$(awk -F'\t' -v i=$i '$1 == i {print $2}' $D/amcheck.txt | paste -sd';'
-); row="$i $m $k $a ${am:--}"
for e in "coalesce(pg_column_compression(v), 'NULL')" "length(v)::text"
"octet_length(v)::text" \
"CASE WHEN md5(v) = hv THEN 'intact' ELSE 'WRONG' END" \
"CASE WHEN md5(substr(v, 1, 5)) = h1 THEN 'intact' ELSE 'WRONG' END"
\
"CASE WHEN md5(substr(v, 3000, 10)) = h2 THEN 'intact' ELSE 'WRONG'
END" \
"CASE WHEN md5(left(v, 100)) = h3 THEN 'intact' ELSE 'WRONG' END"; do
row="$row $(probe "SELECT $e FROM f$m JOIN h USING (id) WHERE id = $i")"
done
echo "$row" >> $OUT
done < $D/cases.txt
echo "cases: $(($(wc -l < $OUT) - 1)); backend crashes: $(grep -c CRASH $OUT ||
true);" \
"ASan reports: $(ls $D/asan* 2>/dev/null | wc -l); TRAP lines: $(grep -c
TRAP $D/log || true);" \
"verify_heapam runs that failed: $(grep -c 'amcheck on f' $D/amcheck.txt
|| true)"
#!/bin/bash
# upgrade-check.sh MODE OLD NEW
# Do TOAST values written by an unpatched server read the same with the
# patched one? OLD and NEW are install prefixes (directories holding bin/).
# Reads go through toast-oracle.sh.txt, expected next to this script.
# same : fill a cluster with OLD, read it with OLD, then start the same
# data directory with NEW and read it again (only when the
# catalog version is the same, as in this series)
# upgrade : fill with OLD, read with OLD, pg_upgrade --copy to NEW, read
# with NEW; then write zstd values next to the old ones and run
# verify_heapam(check_toast => true) on every table
# Prints "identical" or the diff. Under "upgrade" the logical slot is not
# made (pg_upgrade refuses a slot with pending changes), so the decoding
# lines are absent from both sides.
set -eu
MODE=$1; OLD=$2; NEW=$3
H=$(cd "$(dirname "$0")" && pwd); O="bash $H/toast-oracle.sh.txt"
export ASAN_OPTIONS=detect_leaks=0
D=$(mktemp -d ${TMPDIR:-/tmp}/upg.XXXXXX)
trap 'rm -rf $D' EXIT
same() { if cmp -s "$1" "$2"; then echo " $3: identical ($(wc -l < $2) lines)"
else echo " $3: DIFFERS"; diff "$1" "$2" | head -20; fi; }
case $MODE in
same)
$O $OLD $D/old.txt fill $D/c > /dev/null
$O $OLD $D/old.txt read $D/c > /dev/null
$O $NEW $D/new.txt read $D/c > /dev/null
same $D/old.txt $D/new.txt "written by $($OLD/bin/postgres --version | cut
-d' ' -f3), read by the new build" ;;
upgrade)
NOSLOT=1 $O $OLD $D/old.txt fill $D/c > /dev/null
NOSLOT=1 $O $OLD $D/old.txt read $D/c > /dev/null
"$NEW/bin/initdb" -D $D/new -A trust --no-sync --no-data-checksums -U
postgres -E UTF8 --locale=C > /dev/null
printf 'autovacuum = off\nwal_level = logical\nmax_replication_slots = 4\n'
>> $D/new/postgresql.conf
PO=$((61000 + RANDOM % 900))
(cd $D && "$NEW/bin/pg_upgrade" -b $OLD/bin -B $NEW/bin -d $D/c/data -D
$D/new -p $PO -P $((PO + 1)) \
-s $D -U postgres --copy > $D/upgrade.out 2>&1) || { echo "pg_upgrade
failed"; tail -20 $D/upgrade.out; exit 1; }
rm -rf $D/c/data; mv $D/new $D/c/data
NOSLOT=1 $O $NEW $D/new.txt read $D/c > /dev/null
same $D/old.txt $D/new.txt "$($OLD/bin/postgres --version | cut -d' ' -f3)
after pg_upgrade"
P=$((61000 + RANDOM % 900))
"$NEW/bin/pg_ctl" -D $D/c/data -l $D/log -w -o "-p $P -k $D -c
listen_addresses=''" start > /dev/null
"$NEW/bin/psql" -X -qAt -h $D -p $P -U postgres -d postgres -v
ON_ERROR_STOP=1 <<'SQL'
ALTER TABLE c ALTER tp SET COMPRESSION zstd;
INSERT INTO c (kind, n, tp, tl) SELECT 'new-zstd', n, soft(n + 9, n), soft(n +
10, n) FROM sizes;
SELECT ' methods now in c.tp: ' || string_agg(DISTINCT
coalesce(pg_column_compression(tp), 'none'), ', ')
|| '; verify_heapam rows: ' || ((SELECT count(*) FROM verify_heapam('c',
check_toast => true))
+ (SELECT count(*) FROM verify_heapam('u', check_toast => true))
+ (SELECT count(*) FROM verify_heapam('m', check_toast => true)))
|| '; new values read back: ' || (SELECT bool_and(tp = soft(n + 9, n)) FROM c
WHERE kind = 'new-zstd')
FROM c;
SQL
"$NEW/bin/pg_ctl" -D $D/c/data -m fast -w stop > /dev/null ;;
esac