On Thu, Sep 10, 2026 at 02:14:00PM -0700, Bharath Rupireddy wrote: > 1/ Typo: "these can have different sizes." > 2/ Typo: "By default these values are chosen so that four chunk rows > will fit on a page..."? > 6/ Missing typedefs.list entry for varatt_external_oid8.
Fixed. > 3/ Can these be Oid instead of uint32? > > + uint32 va_valueid_lo; /* Low 32 bits of value ID */ > + uint32 va_valueid_hi; /* High 32 bits of value ID */ An Oid8 is not a subset of two Oids, so nope. > 4/ Unlike ONDISK_OID which needs to be 18 to detect existing on-disk > pointers (like the comment on top of this structure definition > explains), having ONDISK_OID8 = 4 seems fine because there is no > backward compatibility requirement for OID8 chunk_ids yet. Is my > understanding correct here? 18 is just used as an historical artifact. Using 4 is fine, yes. > 5/ This looks good, but do we also need to have one for > varatt_external_oid? I don't think so, because it has been this way > for many years for varatt_external_oid (previously varatt_external). Hmm. Why not. Added one with 1497ea77edd2. > 7/ I think we can deduplicate most of the code to reduce the if (OID8) > else (OID) branching. I tried to do so and attached a diff on top of > v16-0007. Please have a look. > 8/ Also, the code in toast_save_datum() now looks a bit complicated > and duplicated, and the comment about the race condition during > rewrite sits only in the OID8 block, which applies to both. I tried to > deduplicate it by moving the rewrite block to a separate function in > the attached diff. Please have a look. OK, that's a big chunk that reduces by close to 30% my previous bigger chunk of code. After a close read, this changes three things: - Introduction of toast_valueid_scankey_init(). It would be weird to add that in the last patch while this fixes some of the bloat from "Add support for oid8 TOAST values" for the scan keys. Find that pretty nice. - toast_external_data and toast_external_info_get() is what reduces a lot of the chunk of data. Hannu had an equivalent on his other thread for the tids where he wanted to extract all the fields individually into an intermediate structure before passing it around, instead of having a bunch of if/else block that depend on the vartag_external. That's actually a super nice addition. - toast_preserve_valueid() is also something that I've found rather nice. The inner loop handling the OID[8] assignment becomes super minimal, with the rewrite reasoning in the new routine. The last two points make most sense in the last patch, because they depend on the new vartag from what I can see. Finally, some numbers for the last patch that introduces the new vartag (with the biggest attribute fix included in last patch): - v17: 16 files changed, 430 insertions(+), 212 deletions(-) - v16: 15 files changed, 661 insertions(+), 219 deletions(-) In short you are cutting 240 lines of code for the last changes with your suggestions, and make the code much more readable. I'd say that this is nice. > 2/ Also, upon ALTER TABLE setting the reloption, since the relcache > entry gets invalidated, we would get the updated reltoastrelid, right? > Say I change the reloption from OID to OID8, before and after the > chunk_id type would just be OID. Is my understanding correct? Yes. That should be what happens. > 3/ When RelationGetToastChunkIdType() returns InvalidOid, the caller > assumes the OID pointer size. I think that's fine, since a relation > without a TOAST table never externalizes anything, so the pointer size > is only the floor for inline-compression candidates and the OID vs > OID8 difference doesn't matter. It also matches today's behavior. Is > that the intent? Yes, using the OID size is for HEAD, mostly. I did not want to break that, and an oid8-based size would be an outlier, so an oid-based size as fallback is the natural thing to do, IMO. Attached is a rebased v17, with the last 5 patches and your refactorings integrated in a cleaner manner, based on my points from above. Thanks again for all the reviews. -- Michael
From aa4b7411475f4ff7c745f74836c9853c70623087 Mon Sep 17 00:00:00 2001 From: Michael Paquier <[email protected]> Date: Mon, 7 Sep 2026 13:40:41 +0900 Subject: [PATCH v17 1/5] Refactor some TOAST value ID code to use Oid8 instead of Oid This change is a mechanical switch to change most of the code paths that assume TOAST value IDs to be Oids to become Oid8, easing an upcoming change to allow larger TOAST values, at 8 bytes. The areas touched are related to table AM, amcheck and logical decoding's reorder buffer. A good chunk of the changes involve switching printf() markers from %u to OID8_FORMAT. --- src/include/access/heaptoast.h | 2 +- src/include/access/tableam.h | 4 +- src/backend/access/common/toast_internals.c | 8 +-- src/backend/access/heap/heaptoast.c | 12 ++-- .../replication/logical/reorderbuffer.c | 23 +++++--- contrib/amcheck/verify_heapam.c | 55 +++++++++++-------- 6 files changed, 60 insertions(+), 44 deletions(-) diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h index b2346fd75bd4..aaae6e0fef69 100644 --- a/src/include/access/heaptoast.h +++ b/src/include/access/heaptoast.h @@ -142,7 +142,7 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc, * Fetch a slice from a toast value stored in a heap table. * ---------- */ -extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid, +extern void heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize, int32 sliceoffset, int32 slicelength, varlena *result); diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index ff03a2b816fc..ebec31a22e40 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -783,7 +783,7 @@ typedef struct TableAmRoutine * table implemented by this AM. See table_relation_fetch_toast_slice() * for more details. */ - void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid, + void (*relation_fetch_toast_slice) (Relation toastrel, Oid8 valueid, int32 attrsize, int32 sliceoffset, int32 slicelength, @@ -1987,7 +1987,7 @@ table_relation_toast_am(Relation rel) * stored. */ static inline void -table_relation_fetch_toast_slice(Relation toastrel, Oid valueid, +table_relation_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize, int32 sliceoffset, int32 slicelength, varlena *result) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 0f234661bfa9..ca5ed8dfabca 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -26,8 +26,8 @@ #include "utils/rel.h" #include "utils/snapmgr.h" -static bool toastrel_valueid_exists(Relation toastrel, Oid valueid); -static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); +static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid); +static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid); /* ---------- * toast_compress_datum - @@ -447,7 +447,7 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative) * ---------- */ static bool -toastrel_valueid_exists(Relation toastrel, Oid valueid) +toastrel_valueid_exists(Relation toastrel, Oid8 valueid) { bool result = false; ScanKeyData toastkey; @@ -495,7 +495,7 @@ toastrel_valueid_exists(Relation toastrel, Oid valueid) * ---------- */ static bool -toastid_valueid_exists(Oid toastrelid, Oid valueid) +toastid_valueid_exists(Oid toastrelid, Oid8 valueid) { bool result; Relation toastrel; diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c index 3b7c658a90e6..81154c17376c 100644 --- a/src/backend/access/heap/heaptoast.c +++ b/src/backend/access/heap/heaptoast.c @@ -623,7 +623,7 @@ toast_build_flattened_tuple(TupleDesc tupleDesc, * result is the varlena into which the results should be written. */ void -heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize, +heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize, int32 sliceoffset, int32 slicelength, varlena *result) { @@ -729,7 +729,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize, else { /* should never happen */ - elog(ERROR, "found toasted toast chunk for toast value %u in %s", + elog(ERROR, "found toasted toast chunk for toast value " OID8_FORMAT " in %s", valueid, RelationGetRelationName(toastrel)); chunksize = 0; /* keep compiler quiet */ chunkdata = NULL; @@ -741,13 +741,13 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize, if (curchunk != expectedchunk) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), - errmsg_internal("unexpected chunk number %d (expected %d) for toast value %u in %s", + errmsg_internal("unexpected chunk number %d (expected %d) for toast value " OID8_FORMAT " in %s", curchunk, expectedchunk, valueid, RelationGetRelationName(toastrel)))); if (curchunk > endchunk) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), - errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value %u in %s", + errmsg_internal("unexpected chunk number %d (out of range %d..%d) for toast value " OID8_FORMAT " in %s", curchunk, startchunk, endchunk, valueid, RelationGetRelationName(toastrel)))); @@ -756,7 +756,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize, if (chunksize != expected_size) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), - errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value %u in %s", + errmsg_internal("unexpected chunk size %d (expected %d) in chunk %d of %d for toast value " OID8_FORMAT " in %s", chunksize, expected_size, curchunk, totalchunks, valueid, RelationGetRelationName(toastrel)))); @@ -785,7 +785,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize, if (expectedchunk != (endchunk + 1)) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), - errmsg_internal("missing chunk number %d for toast value %u in %s", + errmsg_internal("missing chunk number %d for toast value " OID8_FORMAT " in %s", expectedchunk, valueid, RelationGetRelationName(toastrel)))); diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 457d544eec93..dd4b6e8f7f45 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -177,7 +177,7 @@ typedef struct ReorderBufferIterTXNState /* toast datastructures */ typedef struct ReorderBufferToastEnt { - Oid chunk_id; /* toast_table.chunk_id */ + Oid8 chunk_id; /* toast_table.chunk_id */ int32 last_chunk_seq; /* toast_table.chunk_seq of the last chunk we * have seen */ Size num_chunks; /* number of chunks we've already seen */ @@ -5010,7 +5010,7 @@ ReorderBufferToastInitHash(ReorderBuffer *rb, ReorderBufferTXN *txn) Assert(txn->toast_hash == NULL); - hash_ctl.keysize = sizeof(Oid); + hash_ctl.keysize = sizeof(Oid8); hash_ctl.entrysize = sizeof(ReorderBufferToastEnt); hash_ctl.hcxt = rb->context; txn->toast_hash = hash_create("ReorderBufferToastHash", 5, &hash_ctl, @@ -5034,8 +5034,10 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn, bool isnull; Pointer chunk; TupleDesc desc = RelationGetDescr(relation); - Oid chunk_id; + Oid8 chunk_id; int32 chunk_seq; + Oid valueid_type; + Datum valueid_datum; if (txn->toast_hash == NULL) ReorderBufferToastInitHash(rb, txn); @@ -5043,7 +5045,12 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn, Assert(IsToastRelation(relation)); newtup = change->data.tp.newtuple; - chunk_id = DatumGetObjectId(fastgetattr(newtup, 1, desc, &isnull)); + valueid_type = TupleDescAttr(desc, 0)->atttypid; + valueid_datum = fastgetattr(newtup, 1, desc, &isnull); + if (valueid_type == OID8OID) + chunk_id = DatumGetObjectId8(valueid_datum); + else + chunk_id = DatumGetObjectId(valueid_datum); Assert(!isnull); chunk_seq = DatumGetInt32(fastgetattr(newtup, 2, desc, &isnull)); Assert(!isnull); @@ -5061,11 +5068,11 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn, dlist_init(&ent->chunks); if (chunk_seq != 0) - elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq 0", + elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq 0", chunk_seq, chunk_id); } else if (found && chunk_seq != ent->last_chunk_seq + 1) - elog(ERROR, "got sequence entry %d for toast chunk %u instead of seq %d", + elog(ERROR, "got sequence entry %d for toast chunk " OID8_FORMAT " instead of seq %d", chunk_seq, chunk_id, ent->last_chunk_seq + 1); chunk = DatumGetPointer(fastgetattr(newtup, 3, desc, &isnull)); @@ -5174,6 +5181,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn, varlena *reconstructed; dlist_iter it; Size data_done = 0; + Oid8 toast_valueid; if (attr->attisdropped) continue; @@ -5194,13 +5202,14 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn, continue; VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer); + toast_valueid = toast_pointer.va_valueid; /* * Check whether the toast tuple changed, replace if so. */ ent = (ReorderBufferToastEnt *) hash_search(txn->toast_hash, - &toast_pointer.va_valueid, + &toast_valueid, HASH_FIND, NULL); if (ent == NULL) diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c index bbaaec89f8ae..33b00fa4dc5c 100644 --- a/contrib/amcheck/verify_heapam.c +++ b/contrib/amcheck/verify_heapam.c @@ -1564,9 +1564,11 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx, int32 chunksize; int32 expected_size; int32 max_chunk_size; + Oid8 toast_valueid; max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE; last_chunk_seq = (extsize - 1) / max_chunk_size; + toast_valueid = ta->toast_pointer.va_valueid; /* Sanity-check the sequence number. */ chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2, @@ -1574,16 +1576,16 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx, if (isnull) { report_toast_corruption(ctx, ta, - psprintf("toast value %u has toast chunk with null sequence number", - ta->toast_pointer.va_valueid)); + psprintf("toast value " OID8_FORMAT " has toast chunk with null sequence number", + toast_valueid)); return; } if (chunk_seq != *expected_chunk_seq) { /* Either the TOAST index is corrupt, or we don't have all chunks. */ report_toast_corruption(ctx, ta, - psprintf("toast value %u index scan returned chunk %d when expecting chunk %d", - ta->toast_pointer.va_valueid, + psprintf("toast value " OID8_FORMAT " index scan returned chunk %d when expecting chunk %d", + toast_valueid, chunk_seq, *expected_chunk_seq)); } *expected_chunk_seq = chunk_seq + 1; @@ -1594,8 +1596,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx, if (isnull) { report_toast_corruption(ctx, ta, - psprintf("toast value %u chunk %d has null data", - ta->toast_pointer.va_valueid, + psprintf("toast value " OID8_FORMAT " chunk %d has null data", + toast_valueid, chunk_seq)); return; } @@ -1614,8 +1616,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx, uint32 header = ((varattrib_4b *) chunk)->va_4byte.va_header; report_toast_corruption(ctx, ta, - psprintf("toast value %u chunk %d has invalid varlena header %0x", - ta->toast_pointer.va_valueid, + psprintf("toast value " OID8_FORMAT " chunk %d has invalid varlena header %0x", + toast_valueid, chunk_seq, header)); return; } @@ -1626,8 +1628,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx, if (chunk_seq > last_chunk_seq) { report_toast_corruption(ctx, ta, - psprintf("toast value %u chunk %d follows last expected chunk %d", - ta->toast_pointer.va_valueid, + psprintf("toast value " OID8_FORMAT " chunk %d follows last expected chunk %d", + toast_valueid, chunk_seq, last_chunk_seq)); return; } @@ -1637,8 +1639,8 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx, if (chunksize != expected_size) report_toast_corruption(ctx, ta, - psprintf("toast value %u chunk %d has size %u, but expected size %u", - ta->toast_pointer.va_valueid, + psprintf("toast value " OID8_FORMAT " chunk %d has size %u, but expected size %u", + toast_valueid, chunk_seq, chunksize, expected_size)); } @@ -1669,6 +1671,7 @@ check_tuple_attribute(HeapCheckContext *ctx) varlena *attr; char *tp; /* pointer to the tuple data */ uint16 infomask; + Oid8 toast_pointer_valueid; CompactAttribute *thisatt; varatt_external_oid toast_pointer; @@ -1777,12 +1780,13 @@ check_tuple_attribute(HeapCheckContext *ctx) * Must copy attr into toast_pointer for alignment considerations */ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + toast_pointer_valueid = toast_pointer.va_valueid; /* Toasted attributes too large to be untoasted should never be stored */ if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT) report_corruption(ctx, - psprintf("toast value %u rawsize %d exceeds limit %d", - toast_pointer.va_valueid, + psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d", + toast_pointer_valueid, toast_pointer.va_rawsize, VARLENA_SIZE_LIMIT)); @@ -1809,16 +1813,16 @@ check_tuple_attribute(HeapCheckContext *ctx) } if (!valid) report_corruption(ctx, - psprintf("toast value %u has invalid compression method id %d", - toast_pointer.va_valueid, cmid)); + psprintf("toast value " OID8_FORMAT " has invalid compression method id %d", + toast_pointer_valueid, cmid)); } /* The tuple header better claim to contain toasted values */ if (!(infomask & HEAP_HASEXTERNAL)) { report_corruption(ctx, - psprintf("toast value %u is external but tuple header flag HEAP_HASEXTERNAL not set", - toast_pointer.va_valueid)); + psprintf("toast value " OID8_FORMAT " is external but tuple header flag HEAP_HASEXTERNAL not set", + toast_pointer_valueid)); return true; } @@ -1826,8 +1830,8 @@ check_tuple_attribute(HeapCheckContext *ctx) if (!ctx->rel->rd_rel->reltoastrelid) { report_corruption(ctx, - psprintf("toast value %u is external but relation has no toast relation", - toast_pointer.va_valueid)); + psprintf("toast value " OID8_FORMAT " is external but relation has no toast relation", + toast_pointer_valueid)); return true; } @@ -1873,6 +1877,7 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta) int32 expected_chunk_seq = 0; int32 last_chunk_seq; int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE; + Oid8 toast_valueid; extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer); last_chunk_seq = (extsize - 1) / max_chunk_size; @@ -1903,14 +1908,16 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta) } systable_endscan_ordered(toastscan); + toast_valueid = ta->toast_pointer.va_valueid; + if (!found_toasttup) report_toast_corruption(ctx, ta, - psprintf("toast value %u not found in toast table", - ta->toast_pointer.va_valueid)); + psprintf("toast value " OID8_FORMAT " not found in toast table", + toast_valueid)); else if (expected_chunk_seq <= last_chunk_seq) report_toast_corruption(ctx, ta, - psprintf("toast value %u was expected to end at chunk %d, but ended while expecting chunk %d", - ta->toast_pointer.va_valueid, + psprintf("toast value " OID8_FORMAT " was expected to end at chunk %d, but ended while expecting chunk %d", + toast_valueid, last_chunk_seq, expected_chunk_seq)); } -- 2.55.0
From 5466a0440149f35b48fe4c39b8bd49a9283e07de Mon Sep 17 00:00:00 2001 From: Michael Paquier <[email protected]> Date: Wed, 13 Aug 2025 19:55:39 +0900 Subject: [PATCH v17 2/5] Switch pg_column_toast_chunk_id() return value from oid to oid8 This is required for a follow-up patch that will add support for 8-byte TOAST values, with this function being changed so as it is able to support the largest TOAST value type available. XXX: Bump catalog version. --- src/include/catalog/pg_proc.dat | 2 +- src/backend/utils/adt/varlena.c | 2 +- src/test/isolation/expected/cluster-toast-value-reuse.out | 2 +- src/test/isolation/specs/cluster-toast-value-reuse.spec | 2 +- src/test/modules/injection_points/specs/repack_toast.spec | 2 +- src/test/regress/expected/misc_functions.out | 2 +- src/test/regress/sql/misc_functions.sql | 2 +- doc/src/sgml/func/func-admin.sgml | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 74d386b868bf..f46427258e3d 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7907,7 +7907,7 @@ proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '6316', descr => 'chunk ID of on-disk TOASTed value', proname => 'pg_column_toast_chunk_id', provolatile => 's', - prorettype => 'oid', proargtypes => 'any', + prorettype => 'oid8', proargtypes => 'any', prosrc => 'pg_column_toast_chunk_id' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index f6a41e709ae0..4533dac11a6d 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -4290,7 +4290,7 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS) VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); - PG_RETURN_OID(toast_pointer.va_valueid); + PG_RETURN_OID8(toast_pointer.va_valueid); } /* diff --git a/src/test/isolation/expected/cluster-toast-value-reuse.out b/src/test/isolation/expected/cluster-toast-value-reuse.out index cb14ddcee34b..21859fc0dce2 100644 --- a/src/test/isolation/expected/cluster-toast-value-reuse.out +++ b/src/test/isolation/expected/cluster-toast-value-reuse.out @@ -21,7 +21,7 @@ step s2_verify_chunk_ids: SELECT o.id AS chunk_ids_preserved FROM cluster_chunk_id o JOIN cluster_toast_value c ON o.id = c.id - WHERE o.chunk_id != pg_column_toast_chunk_id(c.value); + WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value); chunk_ids_preserved ------------------- diff --git a/src/test/isolation/specs/cluster-toast-value-reuse.spec b/src/test/isolation/specs/cluster-toast-value-reuse.spec index 9a2d10600b39..96cf899db296 100644 --- a/src/test/isolation/specs/cluster-toast-value-reuse.spec +++ b/src/test/isolation/specs/cluster-toast-value-reuse.spec @@ -60,7 +60,7 @@ step s2_verify_chunk_ids { SELECT o.id AS chunk_ids_preserved FROM cluster_chunk_id o JOIN cluster_toast_value c ON o.id = c.id - WHERE o.chunk_id != pg_column_toast_chunk_id(c.value); + WHERE o.chunk_id::oid8 != pg_column_toast_chunk_id(c.value); } # Run UPDATE with its transaction still open, then store the chunk IDs. diff --git a/src/test/modules/injection_points/specs/repack_toast.spec b/src/test/modules/injection_points/specs/repack_toast.spec index fa76bcf28ef9..cc8f034d0167 100644 --- a/src/test/modules/injection_points/specs/repack_toast.spec +++ b/src/test/modules/injection_points/specs/repack_toast.spec @@ -59,7 +59,7 @@ setup CREATE TABLE relfilenodes(node oid); - CREATE TABLE data_s1 (i int, j text, j_toast oid, k text, k_toast oid); + CREATE TABLE data_s1 (i int, j text, j_toast oid8, k text, k_toast oid8); CREATE TABLE data_s2 (LIKE data_s1); } diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index 2990e0c4f286..35d6ee2ce9eb 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -785,7 +785,7 @@ SELECT t.relname AS toastrel FROM pg_class c WHERE c.relname = 'test_chunk_id' \gset SELECT pg_column_toast_chunk_id(a) IS NULL, - pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel) + pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel) FROM test_chunk_id; ?column? | ?column? ----------+---------- diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql index 950d9ab1a4a1..94b91964c046 100644 --- a/src/test/regress/sql/misc_functions.sql +++ b/src/test/regress/sql/misc_functions.sql @@ -316,7 +316,7 @@ SELECT t.relname AS toastrel FROM pg_class c WHERE c.relname = 'test_chunk_id' \gset SELECT pg_column_toast_chunk_id(a) IS NULL, - pg_column_toast_chunk_id(b) IN (SELECT chunk_id FROM pg_toast.:toastrel) + pg_column_toast_chunk_id(b) IN (SELECT chunk_id::oid8 FROM pg_toast.:toastrel) FROM test_chunk_id; DROP TABLE test_chunk_id; diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml index 54eeb42e5bce..70d762e73328 100644 --- a/doc/src/sgml/func/func-admin.sgml +++ b/doc/src/sgml/func/func-admin.sgml @@ -1590,7 +1590,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <primary>pg_column_toast_chunk_id</primary> </indexterm> <function>pg_column_toast_chunk_id</function> ( <type>"any"</type> ) - <returnvalue>oid</returnvalue> + <returnvalue>oid8</returnvalue> </para> <para> Shows the <structfield>chunk_id</structfield> of an on-disk -- 2.55.0
From 21d7e746779ae13008becc83e204b56a0d4b2860 Mon Sep 17 00:00:00 2001 From: Michael Paquier <[email protected]> Date: Fri, 11 Sep 2026 16:17:53 +0900 Subject: [PATCH v17 3/5] Add support for oid8 TOAST values This commit adds the possibility to define TOAST tables with oid8 as value ID, based on the reloption toast_value_type. All the external TOAST pointers still rely on varatt_external and a single vartag. The values inserted in the oid8 TOAST tables are fed from the OID8 value generator, unfortunately casted to OID for now, as we do not have a vartag_external able to store Oid8 values yet. This will be changed in an upcoming patch that adds more vartag_external types and its associated structures, with the code being able to use a different external TOAST pointer depending on the attribute type of chunk_id in TOAST relations. All the changes done here are mechanical, with all the TOAST code able to do chunk ID lookups based on the two types now supported. XXX: Catalog version bump required. --- src/include/access/toast_internals.h | 4 + src/include/catalog/pg_opclass.dat | 3 +- src/include/utils/rel.h | 1 + src/backend/access/common/reloptions.c | 3 +- src/backend/access/common/toast_internals.c | 91 +++++++++++++++------ src/backend/access/heap/heaptoast.c | 7 +- src/backend/catalog/toasting.c | 25 +++++- src/test/regress/expected/reloptions.out | 15 +++- src/test/regress/sql/reloptions.sql | 8 ++ doc/src/sgml/ref/create_table.sgml | 5 +- doc/src/sgml/storage.sgml | 7 +- contrib/amcheck/verify_heapam.c | 13 +-- 12 files changed, 137 insertions(+), 45 deletions(-) diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index bf45889a6428..c0015f9cc6cf 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/skey.h" #include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" @@ -52,6 +53,9 @@ extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); extern Datum toast_save_datum(Relation rel, Datum value, varlena *oldexternal, uint32 options); +extern void toast_valueid_scankey_init(ScanKey entry, AttrNumber attno, + Oid toast_typid, Oid8 valueid); + extern int toast_open_indexes(Relation toastrel, LOCKMODE lock, Relation **toastidxs, diff --git a/src/include/catalog/pg_opclass.dat b/src/include/catalog/pg_opclass.dat index df170b80840b..b84c2bb7a8c3 100644 --- a/src/include/catalog/pg_opclass.dat +++ b/src/include/catalog/pg_opclass.dat @@ -179,7 +179,8 @@ opcintype => 'xid8' }, { opcmethod => 'hash', opcname => 'oid8_ops', opcfamily => 'hash/oid8_ops', opcintype => 'oid8' }, -{ opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops', +{ oid => '8285', oid_symbol => 'OID8_BTREE_OPS_OID', + opcmethod => 'btree', opcname => 'oid8_ops', opcfamily => 'btree/oid8_ops', opcintype => 'oid8' }, { opcmethod => 'hash', opcname => 'cid_ops', opcfamily => 'hash/cid_ops', opcintype => 'cid' }, diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index a88549dcc555..d61432de705e 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -346,6 +346,7 @@ typedef enum StdRdOptToastValueType { STDRD_OPTION_TOAST_VALUE_TYPE_INVALID = 0, STDRD_OPTION_TOAST_VALUE_TYPE_OID, + STDRD_OPTION_TOAST_VALUE_TYPE_OID8, } StdRdOptToastValueType; typedef struct StdRdOptions diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c index 82491a31a9a4..ea9a04179090 100644 --- a/src/backend/access/common/reloptions.c +++ b/src/backend/access/common/reloptions.c @@ -553,6 +553,7 @@ static relopt_enum_elt_def StdRdOptToastValueTypes[] = { /* no value for INVALID */ {"oid", STDRD_OPTION_TOAST_VALUE_TYPE_OID}, + {"oid8", STDRD_OPTION_TOAST_VALUE_TYPE_OID8}, {(const char *) NULL} /* list terminator */ }; @@ -578,7 +579,7 @@ static relopt_enum enumRelOpts[] = }, StdRdOptToastValueTypes, STDRD_OPTION_TOAST_VALUE_TYPE_OID, - gettext_noop("Valid values are \"oid\".") + gettext_noop("Valid values are \"oid\" and \"oid8\".") }, { { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index ca5ed8dfabca..f17e7b8813cb 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -25,6 +25,7 @@ #include "utils/fmgroids.h" #include "utils/rel.h" #include "utils/snapmgr.h" +#include "utils/lsyscache.h" static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid); static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid); @@ -131,6 +132,7 @@ toast_save_datum(Relation rel, Datum value, Pointer dval = DatumGetPointer(value); int num_indexes; int validIndex; + Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1); Assert(!VARATT_IS_EXTERNAL(dval)); @@ -200,24 +202,32 @@ toast_save_datum(Relation rel, Datum value, toast_pointer.va_toastrelid = RelationGetRelid(toastrel); /* - * Choose an OID to use as the value ID for this toast value. + * Choose a new value to use as the value ID for this toast value, be it + * for OID or OID8 TOAST relations. * - * Normally we just choose an unused OID within the toast table. But + * Normally we just choose an unused value within the toast table. But * during table-rewriting operations where we are preserving an existing - * toast table OID, we want to preserve toast value OIDs too. So, if + * toast table OID, we want to preserve toast value IDs too. So, if * rd_toastoid is set and we had a prior external value from that same * toast table, re-use its value ID. If we didn't have a prior external * value (which is a corner case, but possible if the table's attstorage * options have been changed), we have to pick a value ID that doesn't - * conflict with either new or existing toast value OIDs. + * conflict with either new or existing toast value IDs. If the TOAST + * table uses 8-byte value IDs, we should not really care much about + * that. */ if (!OidIsValid(rel->rd_toastoid)) { /* normal case: just choose an unused OID */ - toast_pointer.va_valueid = - GetNewOidWithIndex(toastrel, - RelationGetRelid(toastidxs[validIndex]), - (AttrNumber) 1); + if (toast_typid == OID8OID) + toast_pointer.va_valueid = GetNewObjectId8(); + else + { + toast_pointer.va_valueid = + GetNewOidWithIndex(toastrel, + RelationGetRelid(toastidxs[validIndex]), + (AttrNumber) 1); + } } else { @@ -263,17 +273,22 @@ toast_save_datum(Relation rel, Datum value, if (toast_pointer.va_valueid == InvalidOid) { /* - * new value; must choose an OID that doesn't conflict in either - * old or new toast table + * new value; must choose a value that doesn't conflict in either + * old or new toast table. */ - do + if (toast_typid == OID8OID) + toast_pointer.va_valueid = GetNewObjectId8(); + else { - toast_pointer.va_valueid = - GetNewOidWithIndex(toastrel, - RelationGetRelid(toastidxs[validIndex]), - (AttrNumber) 1); - } while (toastid_valueid_exists(rel->rd_toastoid, - toast_pointer.va_valueid)); + do + { + toast_pointer.va_valueid = + GetNewOidWithIndex(toastrel, + RelationGetRelid(toastidxs[validIndex]), + (AttrNumber) 1); + } while (toastid_valueid_exists(rel->rd_toastoid, + toast_pointer.va_valueid)); + } } } @@ -303,7 +318,10 @@ toast_save_datum(Relation rel, Datum value, /* * Build a tuple and store it */ - t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid); + if (toast_typid == OID8OID) + t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid); + else + t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid); t_values[1] = Int32GetDatum(chunk_seq++); SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ); memcpy(VARDATA(&chunk_data), data_p, chunk_size); @@ -366,6 +384,27 @@ toast_save_datum(Relation rel, Datum value, return PointerGetDatum(result); } +/* ---------- + * toast_valueid_scankey_init - + * + * Initialize a scan key that matches the value ID column of a TOAST table. + * ---------- + */ +void +toast_valueid_scankey_init(ScanKey entry, AttrNumber attno, + Oid toast_typid, Oid8 valueid) +{ + Assert(toast_typid == OIDOID || toast_typid == OID8OID); + if (toast_typid == OID8OID) + ScanKeyInit(entry, attno, + BTEqualStrategyNumber, F_OID8EQ, + ObjectId8GetDatum(valueid)); + else + ScanKeyInit(entry, attno, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum((Oid) valueid)); +} + /* ---------- * toast_delete_datum - * @@ -405,10 +444,9 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative) /* * Setup a scan key to find chunks with matching va_valueid */ - ScanKeyInit(&toastkey, - (AttrNumber) 1, - BTEqualStrategyNumber, F_OIDEQ, - ObjectIdGetDatum(toast_pointer.va_valueid)); + toast_valueid_scankey_init(&toastkey, (AttrNumber) 1, + TupleDescAttr(toastrel->rd_att, 0)->atttypid, + toast_pointer.va_valueid); /* * Find all the chunks. (We don't actually care whether we see them in @@ -455,6 +493,7 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid) int num_indexes; int validIndex; Relation *toastidxs; + Oid toast_typid; /* Fetch a valid index relation */ validIndex = toast_open_indexes(toastrel, @@ -462,13 +501,13 @@ toastrel_valueid_exists(Relation toastrel, Oid8 valueid) &toastidxs, &num_indexes); + toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid; + /* * Setup a scan key to find chunks with matching va_valueid */ - ScanKeyInit(&toastkey, - (AttrNumber) 1, - BTEqualStrategyNumber, F_OIDEQ, - ObjectIdGetDatum(valueid)); + toast_valueid_scankey_init(&toastkey, (AttrNumber) 1, toast_typid, + valueid); /* * Is there any such chunk? diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c index 81154c17376c..b9c89593ecf4 100644 --- a/src/backend/access/heap/heaptoast.c +++ b/src/backend/access/heap/heaptoast.c @@ -640,6 +640,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize, int num_indexes; int validIndex; int32 max_chunk_size; + Oid toast_typid; /* Look for the valid index of toast relation */ validIndex = toast_open_indexes(toastrel, @@ -647,6 +648,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize, &toastidxs, &num_indexes); + toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid; max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE; totalchunks = ((attrsize - 1) / max_chunk_size) + 1; @@ -655,10 +657,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize, Assert(endchunk <= totalchunks); /* Set up a scan key to fetch from the index. */ - ScanKeyInit(&toastkey[0], - (AttrNumber) 1, - BTEqualStrategyNumber, F_OIDEQ, - ObjectIdGetDatum(valueid)); + toast_valueid_scankey_init(&toastkey[0], (AttrNumber) 1, toast_typid, valueid); /* * No additional condition if fetching all chunks. Otherwise, use an diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index a4a7b7c91b31..a7079661666c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -32,6 +32,7 @@ #include "nodes/makefuncs.h" #include "utils/fmgroids.h" #include "utils/rel.h" +#include "utils/lsyscache.h" #include "utils/syscache.h" static void CheckAndCreateToastTable(Oid relOid, Datum reloptions, @@ -173,6 +174,9 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, case STDRD_OPTION_TOAST_VALUE_TYPE_OID: toast_chunkid_typid = OIDOID; break; + case STDRD_OPTION_TOAST_VALUE_TYPE_OID8: + toast_chunkid_typid = OID8OID; + break; case STDRD_OPTION_TOAST_VALUE_TYPE_INVALID: elog(ERROR, "unexpected toast_value_type value %d", value_type); @@ -210,7 +214,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("toast chunk_id type not set while in binary upgrade mode"))); - if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID) + if (binary_upgrade_next_toast_chunk_id_typoid != OIDOID && + binary_upgrade_next_toast_chunk_id_typoid != OID8OID) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("cannot support toast chunk_id type %u in binary upgrade mode", @@ -235,6 +240,19 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, snprintf(toast_idxname, sizeof(toast_idxname), "pg_toast_%u_index", relOid); + /* + * Special case here. If OIDOldToast is defined, we need to rely on the + * existing table for the job because we do not want to create an + * inconsistent relation that would conflict with the parent and break + * the world. + */ + if (OidIsValid(OIDOldToast)) + { + toast_chunkid_typid = get_atttype(OIDOldToast, 1); + if (!OidIsValid(toast_chunkid_typid)) + elog(ERROR, "cache lookup failed for relation %u", OIDOldToast); + } + /* this is pretty painful... need a tuple descriptor */ tupdesc = CreateTemplateTupleDesc(3); TupleDescInitEntry(tupdesc, (AttrNumber) 1, @@ -353,7 +371,10 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, collationIds[0] = InvalidOid; collationIds[1] = InvalidOid; - opclassIds[0] = OID_BTREE_OPS_OID; + if (toast_chunkid_typid == OID8OID) + opclassIds[0] = OID8_BTREE_OPS_OID; + else + opclassIds[0] = OID_BTREE_OPS_OID; opclassIds[1] = INT4_BTREE_OPS_OID; coloptions[0] = 0; diff --git a/src/test/regress/expected/reloptions.out b/src/test/regress/expected/reloptions.out index 9b4c8587aa49..95e4528f7793 100644 --- a/src/test/regress/expected/reloptions.out +++ b/src/test/regress/expected/reloptions.out @@ -199,7 +199,7 @@ CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid'); ERROR: unrecognized parameter "toast_value_type" CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8'); ERROR: invalid value for enum option "toast_value_type": int8 -DETAIL: Valid values are "oid". +DETAIL: Valid values are "oid" and "oid8". CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid'); SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass; reloptions @@ -207,6 +207,19 @@ SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass; {toast_value_type=oid} (1 row) +-- The option is only consulted when the TOAST relation is created, so +-- changing it afterwards leaves chunk_id alone. +ALTER TABLE reloptions_test2 SET (toast_value_type = 'oid8'); +SELECT a.atttypid::regtype AS chunk_id_type + FROM pg_class AS c, pg_attribute AS a + WHERE c.oid = 'reloptions_test2'::regclass AND + a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'; + chunk_id_type +--------------- + oid +(1 row) + +ALTER TABLE reloptions_test2 RESET (toast_value_type); DROP TABLE reloptions_test2; -- relkinds not supported. CREATE INDEX reloptions_test_idx0 ON reloptions_test (s) diff --git a/src/test/regress/sql/reloptions.sql b/src/test/regress/sql/reloptions.sql index 503d15d16e51..80939ee8b5bd 100644 --- a/src/test/regress/sql/reloptions.sql +++ b/src/test/regress/sql/reloptions.sql @@ -117,6 +117,14 @@ CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast.toast_value_type = 'oid'); CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'int8'); CREATE TABLE reloptions_test2 (s VARCHAR) WITH (toast_value_type = 'oid'); SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test2'::regclass; +-- The option is only consulted when the TOAST relation is created, so +-- changing it afterwards leaves chunk_id alone. +ALTER TABLE reloptions_test2 SET (toast_value_type = 'oid8'); +SELECT a.atttypid::regtype AS chunk_id_type + FROM pg_class AS c, pg_attribute AS a + WHERE c.oid = 'reloptions_test2'::regclass AND + a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id'; +ALTER TABLE reloptions_test2 RESET (toast_value_type); DROP TABLE reloptions_test2; -- relkinds not supported. CREATE INDEX reloptions_test_idx0 ON reloptions_test (s) diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 79622c1a23d6..6f9a87ef449f 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -1663,7 +1663,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM <para> Specifies the attribute type of <literal>chunk_id</literal> to use when creating a <acronym>TOAST</acronym> relation for this table. The - default is <literal>oid</literal>. + default is <literal>oid</literal>; <literal>oid8</literal> assigns + <type>oid8</type> instead, which allows a larger number of distinct + <acronym>TOAST</acronym>ed values at the price of four extra bytes per + out-of-line pointer. This parameter cannot be set for <acronym>TOAST</acronym> tables. </para> <para> diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index 90aae3defcbc..21bd67c11ea0 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -421,14 +421,15 @@ most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is c so that four chunk rows will fit on a page, making it about 2000 bytes). Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table belonging to the owning table. Every -<acronym>TOAST</acronym> table has the columns <structfield>chunk_id</structfield> (an OID -identifying the particular <acronym>TOAST</acronym>ed value), +<acronym>TOAST</acronym> table has the columns +<structfield>chunk_id</structfield> (an OID or an OID8 identifying +the particular <acronym>TOAST</acronym>ed value), <structfield>chunk_seq</structfield> (a sequence number for the chunk within its value), and <structfield>chunk_data</structfield> (the actual data of the chunk). A unique index on <structfield>chunk_id</structfield> and <structfield>chunk_seq</structfield> provides fast retrieval of the values. A pointer datum representing an out-of-line on-disk <acronym>TOAST</acronym>ed value therefore needs to store the OID of the -<acronym>TOAST</acronym> table in which to look and the OID of the specific value +<acronym>TOAST</acronym> table in which to look and the specific value (its <structfield>chunk_id</structfield>). For convenience, pointer datums also store the logical datum size (original uncompressed data length), physical stored size (different if compression was applied), and the compression method used, if diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c index 33b00fa4dc5c..c89fab8764ac 100644 --- a/contrib/amcheck/verify_heapam.c +++ b/contrib/amcheck/verify_heapam.c @@ -28,7 +28,6 @@ #include "storage/procarray.h" #include "storage/read_stream.h" #include "utils/builtins.h" -#include "utils/fmgroids.h" #include "utils/rel.h" #include "utils/tuplestore.h" @@ -1876,8 +1875,12 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta) uint32 extsize; int32 expected_chunk_seq = 0; int32 last_chunk_seq; - int32 max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE; + int32 max_chunk_size; Oid8 toast_valueid; + Oid toast_typid; + + toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid; + max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE; extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer); last_chunk_seq = (extsize - 1) / max_chunk_size; @@ -1885,10 +1888,8 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta) /* * Setup a scan key to find chunks in toast table with matching va_valueid */ - ScanKeyInit(&toastkey, - (AttrNumber) 1, - BTEqualStrategyNumber, F_OIDEQ, - ObjectIdGetDatum(ta->toast_pointer.va_valueid)); + toast_valueid_scankey_init(&toastkey, (AttrNumber) 1, toast_typid, + ta->toast_pointer.va_valueid); /* * Check if any chunks for this toasted object exist in the toast table, -- 2.55.0
From d32734c973ea57ead8afce118e30e92cbd910ffb Mon Sep 17 00:00:00 2001 From: Michael Paquier <[email protected]> Date: Thu, 10 Sep 2026 16:24:52 +0900 Subject: [PATCH v17 4/5] Add battery of tests related oid8 The tests added by this commit cover a large ground in terms of the introduction of oid8 as external TOAST pointer, with the following areas covered, as found during development of the feature: - Tables with oid8 TOAST tables in general, under strings.sql. - Relation rewrites do not affect the TOAST relation after initial creation. - oid8 TOAST with pg_column_compression(). - test_decoding, for external and extended TOAST data with oid8 pointers. - amcheck, with oid8 TOAST relations, external and extended formats with heap verified. - UPDATE with out-of-line datum that belongs to another TOAST table. This tracks a bug in toast_tuple_init(), where external vartags could be mixed once multiple types are introduced. That's something that absan would trigger. These tests are added before the commit that has adds the new external vartag, just because it is what it is. --- src/test/regress/expected/compression.out | 41 +++ src/test/regress/expected/strings.out | 300 +++++++++++++++++++--- src/test/regress/sql/compression.sql | 22 ++ src/test/regress/sql/strings.sql | 181 ++++++++++--- contrib/amcheck/expected/check_heap.out | 40 +++ contrib/amcheck/sql/check_heap.sql | 27 ++ contrib/test_decoding/expected/toast.out | 39 +++ contrib/test_decoding/sql/toast.sql | 20 ++ 8 files changed, 595 insertions(+), 75 deletions(-) diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out index 09f198149aa4..e9128193515a 100644 --- a/src/test/regress/expected/compression.out +++ b/src/test/regress/expected/compression.out @@ -66,6 +66,47 @@ SELECT SUBSTR(f1, 200, 5) FROM cmdata2; (1 row) DROP TABLE cmdata2; +-- pg_column_compression() with oid and oid8 +CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid'); +CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8'); +ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL; +ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL; +INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000)); +INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000)); +SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid; + uncompressed +-------------- + t +(1 row) + +SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8; + uncompressed +-------------- + t +(1 row) + +-- out-of-line and compressed. +TRUNCATE toastcomp_oid; +TRUNCATE toastcomp_oid8; +ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED; +ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED; +ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128); +ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128); +INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000)); +INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000)); +SELECT pg_column_compression(f1) FROM toastcomp_oid; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM toastcomp_oid8; + pg_column_compression +----------------------- + pglz +(1 row) + +DROP TABLE toastcomp_oid, toastcomp_oid8; --test column type update varlena/non-varlena CREATE TABLE cmdata2 (f1 int); \d+ cmdata2 diff --git a/src/test/regress/expected/strings.out b/src/test/regress/expected/strings.out index 313c50394650..7e3a641259c2 100644 --- a/src/test/regress/expected/strings.out +++ b/src/test/regress/expected/strings.out @@ -1980,21 +1980,37 @@ SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar"; (1 row) -- --- test substr with toasted text values +-- Test substr with toasted text values, for all types of TOAST relations +-- supported. -- -CREATE TABLE toasttest(f1 text); -insert into toasttest values(repeat('1234567890',10000)); -insert into toasttest values(repeat('1234567890',10000)); +CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid'); +CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8'); +insert into toasttest_oid values(repeat('1234567890',10000)); +insert into toasttest_oid values(repeat('1234567890',10000)); +insert into toasttest_oid8 values(repeat('1234567890',10000)); +insert into toasttest_oid8 values(repeat('1234567890',10000)); -- -- Ensure that some values are uncompressed, to test the faster substring -- operation used in that case -- -alter table toasttest alter column f1 set storage external; -insert into toasttest values(repeat('1234567890',10000)); -insert into toasttest values(repeat('1234567890',10000)); +alter table toasttest_oid alter column f1 set storage external; +insert into toasttest_oid values(repeat('1234567890',10000)); +insert into toasttest_oid values(repeat('1234567890',10000)); +alter table toasttest_oid8 alter column f1 set storage external; +insert into toasttest_oid8 values(repeat('1234567890',10000)); +insert into toasttest_oid8 values(repeat('1234567890',10000)); -- If the starting position is zero or less, then return from the start of the string -- adjusting the length to be consistent with the "negative start" per SQL. -SELECT substr(f1, -1, 5) from toasttest; +SELECT substr(f1, -1, 5) from toasttest_oid; + substr +-------- + 123 + 123 + 123 + 123 +(4 rows) + +SELECT substr(f1, -1, 5) from toasttest_oid8; substr -------- 123 @@ -2004,11 +2020,22 @@ SELECT substr(f1, -1, 5) from toasttest; (4 rows) -- If the length is less than zero, an ERROR is thrown. -SELECT substr(f1, 5, -1) from toasttest; +SELECT substr(f1, 5, -1) from toasttest_oid; +ERROR: negative substring length not allowed +SELECT substr(f1, 5, -1) from toasttest_oid8; ERROR: negative substring length not allowed -- If no third argument (length) is provided, the length to the end of the -- string is assumed. -SELECT substr(f1, 99995) from toasttest; +SELECT substr(f1, 99995) from toasttest_oid; + substr +-------- + 567890 + 567890 + 567890 + 567890 +(4 rows) + +SELECT substr(f1, 99995) from toasttest_oid8; substr -------- 567890 @@ -2019,7 +2046,7 @@ SELECT substr(f1, 99995) from toasttest; -- If start plus length is > string length, the result is truncated to -- string length -SELECT substr(f1, 99995, 10) from toasttest; +SELECT substr(f1, 99995, 10) from toasttest_oid; substr -------- 567890 @@ -2028,50 +2055,105 @@ SELECT substr(f1, 99995, 10) from toasttest; 567890 (4 rows) -TRUNCATE TABLE toasttest; -INSERT INTO toasttest values (repeat('1234567890',300)); -INSERT INTO toasttest values (repeat('1234567890',300)); -INSERT INTO toasttest values (repeat('1234567890',300)); -INSERT INTO toasttest values (repeat('1234567890',300)); +SELECT substr(f1, 99995, 10) from toasttest_oid8; + substr +-------- + 567890 + 567890 + 567890 + 567890 +(4 rows) + +-- TRUNCATE cases for TOAST relations with OID values. +TRUNCATE TABLE toasttest_oid; +INSERT INTO toasttest_oid values (repeat('1234567890',300)); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); -- expect >0 blocks SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty - FROM pg_class where relname = 'toasttest'; + FROM pg_class where relname = 'toasttest_oid'; is_empty ---------- f (1 row) -TRUNCATE TABLE toasttest; -ALTER TABLE toasttest set (toast_tuple_target = 4080); -INSERT INTO toasttest values (repeat('1234567890',300)); -INSERT INTO toasttest values (repeat('1234567890',300)); -INSERT INTO toasttest values (repeat('1234567890',300)); -INSERT INTO toasttest values (repeat('1234567890',300)); +TRUNCATE TABLE toasttest_oid; +ALTER TABLE toasttest_oid set (toast_tuple_target = 4080); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); -- expect 0 blocks SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty - FROM pg_class where relname = 'toasttest'; + FROM pg_class where relname = 'toasttest_oid'; is_empty ---------- t (1 row) -DROP TABLE toasttest; +DROP TABLE toasttest_oid; +-- TRUNCATE cases for TOAST relation with int8 values. +TRUNCATE TABLE toasttest_oid8; +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +-- expect >0 blocks +SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty + FROM pg_class where relname = 'toasttest_oid8'; + is_empty +---------- + f +(1 row) + +TRUNCATE TABLE toasttest_oid8; +ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +-- expect 0 blocks +SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty + FROM pg_class where relname = 'toasttest_oid8'; + is_empty +---------- + t +(1 row) + +DROP TABLE toasttest_oid8; -- --- test substr with toasted bytea values +-- test substr with toasted bytea values, for all types of TOAST relations +-- supported. Do not drop these two relations, for pg_upgrade. -- -CREATE TABLE toasttest(f1 bytea); -insert into toasttest values(decode(repeat('1234567890',10000),'escape')); -insert into toasttest values(decode(repeat('1234567890',10000),'escape')); +CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid'); +CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8'); +insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape')); +insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape')); +insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape')); +insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape')); -- -- Ensure that some values are uncompressed, to test the faster substring -- operation used in that case -- -alter table toasttest alter column f1 set storage external; -insert into toasttest values(decode(repeat('1234567890',10000),'escape')); -insert into toasttest values(decode(repeat('1234567890',10000),'escape')); +alter table toasttest_oid alter column f1 set storage external; +insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape')); +insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape')); +alter table toasttest_oid8 alter column f1 set storage external; +insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape')); +insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape')); -- If the starting position is zero or less, then return from the start of the string -- adjusting the length to be consistent with the "negative start" per SQL. -SELECT substr(f1, -1, 5) from toasttest; +SELECT substr(f1, -1, 5) from toasttest_oid; + substr +-------- + 123 + 123 + 123 + 123 +(4 rows) + +SELECT substr(f1, -1, 5) from toasttest_oid8; substr -------- 123 @@ -2081,11 +2163,22 @@ SELECT substr(f1, -1, 5) from toasttest; (4 rows) -- If the length is less than zero, an ERROR is thrown. -SELECT substr(f1, 5, -1) from toasttest; +SELECT substr(f1, 5, -1) from toasttest_oid; +ERROR: negative substring length not allowed +SELECT substr(f1, 5, -1) from toasttest_oid8; ERROR: negative substring length not allowed -- If no third argument (length) is provided, the length to the end of the -- string is assumed. -SELECT substr(f1, 99995) from toasttest; +SELECT substr(f1, 99995) from toasttest_oid; + substr +-------- + 567890 + 567890 + 567890 + 567890 +(4 rows) + +SELECT substr(f1, 99995) from toasttest_oid8; substr -------- 567890 @@ -2096,7 +2189,7 @@ SELECT substr(f1, 99995) from toasttest; -- If start plus length is > string length, the result is truncated to -- string length -SELECT substr(f1, 99995, 10) from toasttest; +SELECT substr(f1, 99995, 10) from toasttest_oid; substr -------- 567890 @@ -2105,7 +2198,140 @@ SELECT substr(f1, 99995, 10) from toasttest; 567890 (4 rows) -DROP TABLE toasttest; +SELECT substr(f1, 99995, 10) from toasttest_oid8; + substr +-------- + 567890 + 567890 + 567890 + 567890 +(4 rows) + +-- A relation rewrite leaves the TOAST value attributes unchanged. +VACUUM FULL toasttest_oid; +VACUUM FULL toasttest_oid8; +SELECT c1.relname, a.atttypid::regtype + FROM pg_attribute AS a, + pg_class AS c1, + pg_class AS c2 + WHERE + c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND + c1.reltoastrelid = c2.oid AND + a.attrelid = c2.oid AND + a.attname = 'chunk_id' + ORDER BY c1.relname COLLATE "C"; + relname | atttypid +----------------+---------- + toasttest_oid | oid + toasttest_oid8 | oid8 +(2 rows) + +-- Check that data slices are still accessible. +SELECT substr(f1, 99995) from toasttest_oid; + substr +-------- + 567890 + 567890 + 567890 + 567890 +(4 rows) + +SELECT substr(f1, 99995) from toasttest_oid8; + substr +-------- + 567890 + 567890 + 567890 + 567890 +(4 rows) + +SELECT substr(f1, 99995, 10) from toasttest_oid; + substr +-------- + 567890 + 567890 + 567890 + 567890 +(4 rows) + +SELECT substr(f1, 99995, 10) from toasttest_oid8; + substr +-------- + 567890 + 567890 + 567890 + 567890 +(4 rows) + +-- ALTER TABLE after TOAST table creation does not affect relation rewrite. +ALTER TABLE toasttest_oid SET (toast_value_type = oid8); +ALTER TABLE toasttest_oid8 SET (toast_value_type = oid); +VACUUM FULL toasttest_oid; +VACUUM FULL toasttest_oid8; +SELECT c1.relname, a.atttypid::regtype + FROM pg_attribute AS a, + pg_class AS c1, + pg_class AS c2 + WHERE + c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND + c1.reltoastrelid = c2.oid AND + a.attrelid = c2.oid AND + a.attname = 'chunk_id' + ORDER BY c1.relname COLLATE "C"; + relname | atttypid +----------------+---------- + toasttest_oid | oid + toasttest_oid8 | oid8 +(2 rows) + +ALTER TABLE toasttest_oid RESET (toast_value_type); +ALTER TABLE toasttest_oid8 RESET (toast_value_type); +-- Reset column storage to its default +ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED; +ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED; +-- UPDATE with out-of-line datum that belongs to another TOAST table. +CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid'); +CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8'); +ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL; +ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL; +SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class + WHERE oid = 'toastupd_oid'::regclass \gset +SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class + WHERE oid = 'toastupd_oid8'::regclass \gset +INSERT INTO toastupd_oid VALUES (repeat('a', 100000)); +INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000)); +-- old value is an oid8 pointer, new value an oid pointer. +UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid; +SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8; + length | substr +--------+-------- + 100000 | aaa +(1 row) + +-- replaced value must be gone, leaving a single value behind. +SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0; + count +------- + 1 +(1 row) + +-- reverse: old value is an oid pointer, new value an oid8 pointer. +TRUNCATE toastupd_oid; +INSERT INTO toastupd_oid VALUES (repeat('c', 100000)); +UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8; +SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid; + length | substr +--------+-------- + 100000 | aaa +(1 row) + +SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0; + count +------- + 1 +(1 row) + +DROP TABLE toastupd_oid, toastupd_oid8; -- test internally compressing datums -- this tests compressing a datum to a very small size which exercises a -- corner case in packed-varlena handling: even though small, the compressed diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql index ce5ea37a660c..229143f06ddd 100644 --- a/src/test/regress/sql/compression.sql +++ b/src/test/regress/sql/compression.sql @@ -37,6 +37,28 @@ SELECT pg_column_compression(f1) FROM cmdata2; SELECT SUBSTR(f1, 200, 5) FROM cmdata2; DROP TABLE cmdata2; +-- pg_column_compression() with oid and oid8 +CREATE TABLE toastcomp_oid(f1 text) WITH (toast_value_type = 'oid'); +CREATE TABLE toastcomp_oid8(f1 text) WITH (toast_value_type = 'oid8'); +ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTERNAL; +ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL; +INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000)); +INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000)); +SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid; +SELECT pg_column_compression(f1) IS NULL AS uncompressed FROM toastcomp_oid8; +-- out-of-line and compressed. +TRUNCATE toastcomp_oid; +TRUNCATE toastcomp_oid8; +ALTER TABLE toastcomp_oid ALTER COLUMN f1 SET STORAGE EXTENDED; +ALTER TABLE toastcomp_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED; +ALTER TABLE toastcomp_oid SET (toast_tuple_target = 128); +ALTER TABLE toastcomp_oid8 SET (toast_tuple_target = 128); +INSERT INTO toastcomp_oid VALUES (repeat('1234567890', 10000)); +INSERT INTO toastcomp_oid8 VALUES (repeat('1234567890', 10000)); +SELECT pg_column_compression(f1) FROM toastcomp_oid; +SELECT pg_column_compression(f1) FROM toastcomp_oid8; +DROP TABLE toastcomp_oid, toastcomp_oid8; + --test column type update varlena/non-varlena CREATE TABLE cmdata2 (f1 int); \d+ cmdata2 diff --git a/src/test/regress/sql/strings.sql b/src/test/regress/sql/strings.sql index 38946e8954df..4d6a1e02359c 100644 --- a/src/test/regress/sql/strings.sql +++ b/src/test/regress/sql/strings.sql @@ -563,89 +563,194 @@ SELECT text 'text' || char(20) ' and characters' AS "Concat text to char"; SELECT text 'text' || varchar ' and varchar' AS "Concat text to varchar"; -- --- test substr with toasted text values +-- Test substr with toasted text values, for all types of TOAST relations +-- supported. -- -CREATE TABLE toasttest(f1 text); +CREATE TABLE toasttest_oid(f1 text) with (toast_value_type = 'oid'); +CREATE TABLE toasttest_oid8(f1 text) with (toast_value_type = 'oid8'); -insert into toasttest values(repeat('1234567890',10000)); -insert into toasttest values(repeat('1234567890',10000)); +insert into toasttest_oid values(repeat('1234567890',10000)); +insert into toasttest_oid values(repeat('1234567890',10000)); +insert into toasttest_oid8 values(repeat('1234567890',10000)); +insert into toasttest_oid8 values(repeat('1234567890',10000)); -- -- Ensure that some values are uncompressed, to test the faster substring -- operation used in that case -- -alter table toasttest alter column f1 set storage external; -insert into toasttest values(repeat('1234567890',10000)); -insert into toasttest values(repeat('1234567890',10000)); +alter table toasttest_oid alter column f1 set storage external; +insert into toasttest_oid values(repeat('1234567890',10000)); +insert into toasttest_oid values(repeat('1234567890',10000)); +alter table toasttest_oid8 alter column f1 set storage external; +insert into toasttest_oid8 values(repeat('1234567890',10000)); +insert into toasttest_oid8 values(repeat('1234567890',10000)); -- If the starting position is zero or less, then return from the start of the string -- adjusting the length to be consistent with the "negative start" per SQL. -SELECT substr(f1, -1, 5) from toasttest; +SELECT substr(f1, -1, 5) from toasttest_oid; +SELECT substr(f1, -1, 5) from toasttest_oid8; -- If the length is less than zero, an ERROR is thrown. -SELECT substr(f1, 5, -1) from toasttest; +SELECT substr(f1, 5, -1) from toasttest_oid; +SELECT substr(f1, 5, -1) from toasttest_oid8; -- If no third argument (length) is provided, the length to the end of the -- string is assumed. -SELECT substr(f1, 99995) from toasttest; +SELECT substr(f1, 99995) from toasttest_oid; +SELECT substr(f1, 99995) from toasttest_oid8; -- If start plus length is > string length, the result is truncated to -- string length -SELECT substr(f1, 99995, 10) from toasttest; +SELECT substr(f1, 99995, 10) from toasttest_oid; +SELECT substr(f1, 99995, 10) from toasttest_oid8; -TRUNCATE TABLE toasttest; -INSERT INTO toasttest values (repeat('1234567890',300)); -INSERT INTO toasttest values (repeat('1234567890',300)); -INSERT INTO toasttest values (repeat('1234567890',300)); -INSERT INTO toasttest values (repeat('1234567890',300)); +-- TRUNCATE cases for TOAST relations with OID values. +TRUNCATE TABLE toasttest_oid; +INSERT INTO toasttest_oid values (repeat('1234567890',300)); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); -- expect >0 blocks SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty - FROM pg_class where relname = 'toasttest'; - -TRUNCATE TABLE toasttest; -ALTER TABLE toasttest set (toast_tuple_target = 4080); -INSERT INTO toasttest values (repeat('1234567890',300)); -INSERT INTO toasttest values (repeat('1234567890',300)); -INSERT INTO toasttest values (repeat('1234567890',300)); -INSERT INTO toasttest values (repeat('1234567890',300)); + FROM pg_class where relname = 'toasttest_oid'; +TRUNCATE TABLE toasttest_oid; +ALTER TABLE toasttest_oid set (toast_tuple_target = 4080); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); +INSERT INTO toasttest_oid values (repeat('1234567890',300)); -- expect 0 blocks SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty - FROM pg_class where relname = 'toasttest'; + FROM pg_class where relname = 'toasttest_oid'; +DROP TABLE toasttest_oid; -DROP TABLE toasttest; +-- TRUNCATE cases for TOAST relation with int8 values. +TRUNCATE TABLE toasttest_oid8; +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +-- expect >0 blocks +SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty + FROM pg_class where relname = 'toasttest_oid8'; +TRUNCATE TABLE toasttest_oid8; +ALTER TABLE toasttest_oid8 set (toast_tuple_target = 4080); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +INSERT INTO toasttest_oid8 values (repeat('1234567890',300)); +-- expect 0 blocks +SELECT pg_relation_size(reltoastrelid) = 0 AS is_empty + FROM pg_class where relname = 'toasttest_oid8'; +DROP TABLE toasttest_oid8; -- --- test substr with toasted bytea values +-- test substr with toasted bytea values, for all types of TOAST relations +-- supported. Do not drop these two relations, for pg_upgrade. -- -CREATE TABLE toasttest(f1 bytea); +CREATE TABLE toasttest_oid(f1 bytea) WITH (toast_value_type = 'oid'); +CREATE TABLE toasttest_oid8(f1 bytea) WITH (toast_value_type = 'oid8'); -insert into toasttest values(decode(repeat('1234567890',10000),'escape')); -insert into toasttest values(decode(repeat('1234567890',10000),'escape')); +insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape')); +insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape')); +insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape')); +insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape')); -- -- Ensure that some values are uncompressed, to test the faster substring -- operation used in that case -- -alter table toasttest alter column f1 set storage external; -insert into toasttest values(decode(repeat('1234567890',10000),'escape')); -insert into toasttest values(decode(repeat('1234567890',10000),'escape')); +alter table toasttest_oid alter column f1 set storage external; +insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape')); +insert into toasttest_oid values(decode(repeat('1234567890',10000),'escape')); +alter table toasttest_oid8 alter column f1 set storage external; +insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape')); +insert into toasttest_oid8 values(decode(repeat('1234567890',10000),'escape')); -- If the starting position is zero or less, then return from the start of the string -- adjusting the length to be consistent with the "negative start" per SQL. -SELECT substr(f1, -1, 5) from toasttest; +SELECT substr(f1, -1, 5) from toasttest_oid; +SELECT substr(f1, -1, 5) from toasttest_oid8; -- If the length is less than zero, an ERROR is thrown. -SELECT substr(f1, 5, -1) from toasttest; +SELECT substr(f1, 5, -1) from toasttest_oid; +SELECT substr(f1, 5, -1) from toasttest_oid8; -- If no third argument (length) is provided, the length to the end of the -- string is assumed. -SELECT substr(f1, 99995) from toasttest; +SELECT substr(f1, 99995) from toasttest_oid; +SELECT substr(f1, 99995) from toasttest_oid8; -- If start plus length is > string length, the result is truncated to -- string length -SELECT substr(f1, 99995, 10) from toasttest; +SELECT substr(f1, 99995, 10) from toasttest_oid; +SELECT substr(f1, 99995, 10) from toasttest_oid8; -DROP TABLE toasttest; +-- A relation rewrite leaves the TOAST value attributes unchanged. +VACUUM FULL toasttest_oid; +VACUUM FULL toasttest_oid8; +SELECT c1.relname, a.atttypid::regtype + FROM pg_attribute AS a, + pg_class AS c1, + pg_class AS c2 + WHERE + c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND + c1.reltoastrelid = c2.oid AND + a.attrelid = c2.oid AND + a.attname = 'chunk_id' + ORDER BY c1.relname COLLATE "C"; +-- Check that data slices are still accessible. +SELECT substr(f1, 99995) from toasttest_oid; +SELECT substr(f1, 99995) from toasttest_oid8; +SELECT substr(f1, 99995, 10) from toasttest_oid; +SELECT substr(f1, 99995, 10) from toasttest_oid8; + +-- ALTER TABLE after TOAST table creation does not affect relation rewrite. +ALTER TABLE toasttest_oid SET (toast_value_type = oid8); +ALTER TABLE toasttest_oid8 SET (toast_value_type = oid); +VACUUM FULL toasttest_oid; +VACUUM FULL toasttest_oid8; +SELECT c1.relname, a.atttypid::regtype + FROM pg_attribute AS a, + pg_class AS c1, + pg_class AS c2 + WHERE + c1.relname IN ('toasttest_oid', 'toasttest_oid8') AND + c1.reltoastrelid = c2.oid AND + a.attrelid = c2.oid AND + a.attname = 'chunk_id' + ORDER BY c1.relname COLLATE "C"; +ALTER TABLE toasttest_oid RESET (toast_value_type); +ALTER TABLE toasttest_oid8 RESET (toast_value_type); + +-- Reset column storage to its default +ALTER TABLE toasttest_oid ALTER COLUMN f1 SET STORAGE EXTENDED; +ALTER TABLE toasttest_oid8 ALTER COLUMN f1 SET STORAGE EXTENDED; + +-- UPDATE with out-of-line datum that belongs to another TOAST table. +CREATE TABLE toastupd_oid(f1 text) WITH (toast_value_type = 'oid'); +CREATE TABLE toastupd_oid8(f1 text) WITH (toast_value_type = 'oid8'); +ALTER TABLE toastupd_oid ALTER COLUMN f1 SET STORAGE EXTERNAL; +ALTER TABLE toastupd_oid8 ALTER COLUMN f1 SET STORAGE EXTERNAL; +SELECT reltoastrelid::regclass AS upd_oid_toast FROM pg_class + WHERE oid = 'toastupd_oid'::regclass \gset +SELECT reltoastrelid::regclass AS upd_oid8_toast FROM pg_class + WHERE oid = 'toastupd_oid8'::regclass \gset +INSERT INTO toastupd_oid VALUES (repeat('a', 100000)); +INSERT INTO toastupd_oid8 VALUES (repeat('b', 100000)); +-- old value is an oid8 pointer, new value an oid pointer. +UPDATE toastupd_oid8 SET f1 = toastupd_oid.f1 FROM toastupd_oid; +SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid8; +-- replaced value must be gone, leaving a single value behind. +SELECT count(*) FROM :upd_oid8_toast WHERE chunk_seq = 0; +-- reverse: old value is an oid pointer, new value an oid8 pointer. +TRUNCATE toastupd_oid; +INSERT INTO toastupd_oid VALUES (repeat('c', 100000)); +UPDATE toastupd_oid SET f1 = toastupd_oid8.f1 FROM toastupd_oid8; +SELECT length(f1), substr(f1, 1, 3) FROM toastupd_oid; +SELECT count(*) FROM :upd_oid_toast WHERE chunk_seq = 0; + +DROP TABLE toastupd_oid, toastupd_oid8; -- test internally compressing datums diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.out index 979e5e84e723..8686fba712d0 100644 --- a/contrib/amcheck/expected/check_heap.out +++ b/contrib/amcheck/expected/check_heap.out @@ -199,6 +199,44 @@ SELECT * FROM verify_heapam('test_partition', -------+--------+--------+----- (0 rows) +-- Check TOAST relations of both chunk_id: oid and oid8 +CREATE TABLE test_toast_oid (a int, b text) WITH (toast_value_type = 'oid'); +CREATE TABLE test_toast_oid8 (a int, b text) WITH (toast_value_type = 'oid8'); +-- Uncompressed out-of-line values +ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTERNAL; +ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL; +INSERT INTO test_toast_oid (a, b) + (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs); +INSERT INTO test_toast_oid8 (a, b) + (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs); +-- Compressed out-of-line values. +ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED; +ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED; +INSERT INTO test_toast_oid (a, b) + (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs); +INSERT INTO test_toast_oid8 (a, b) + (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs); +SELECT c.relname, a.atttypid::regtype AS chunk_id_type + FROM pg_class AS c, pg_attribute AS a + WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND + a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id' + ORDER BY c.relname COLLATE "C"; + relname | chunk_id_type +-----------------+--------------- + test_toast_oid | oid + test_toast_oid8 | oid8 +(2 rows) + +SELECT * FROM verify_heapam('test_toast_oid', check_toast := true); + blkno | offnum | attnum | msg +-------+--------+--------+----- +(0 rows) + +SELECT * FROM verify_heapam('test_toast_oid8', check_toast := true); + blkno | offnum | attnum | msg +-------+--------+--------+----- +(0 rows) + -- Check that indexes are rejected CREATE INDEX test_index ON test_partition (a); SELECT * FROM verify_heapam('test_index', @@ -232,6 +270,8 @@ SELECT * FROM verify_heapam('test_foreign_table', ERROR: cannot check relation "test_foreign_table" DETAIL: This operation is not supported for foreign tables. -- cleanup +DROP TABLE test_toast_oid; +DROP TABLE test_toast_oid8; DROP TABLE heaptest; DROP TABLESPACE regress_test_stats_tblspc; DROP TABLE test_partition; diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sql index 1745bae634e5..eb46fe710351 100644 --- a/contrib/amcheck/sql/check_heap.sql +++ b/contrib/amcheck/sql/check_heap.sql @@ -112,6 +112,31 @@ SELECT * FROM verify_heapam('test_partition', startblock := NULL, endblock := NULL); +-- Check TOAST relations of both chunk_id: oid and oid8 +CREATE TABLE test_toast_oid (a int, b text) WITH (toast_value_type = 'oid'); +CREATE TABLE test_toast_oid8 (a int, b text) WITH (toast_value_type = 'oid8'); +-- Uncompressed out-of-line values +ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTERNAL; +ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTERNAL; +INSERT INTO test_toast_oid (a, b) + (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs); +INSERT INTO test_toast_oid8 (a, b) + (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(1,5) gs); +-- Compressed out-of-line values. +ALTER TABLE test_toast_oid ALTER COLUMN b SET STORAGE EXTENDED; +ALTER TABLE test_toast_oid8 ALTER COLUMN b SET STORAGE EXTENDED; +INSERT INTO test_toast_oid (a, b) + (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs); +INSERT INTO test_toast_oid8 (a, b) + (SELECT gs, repeat('xyzzy', 20000) FROM generate_series(6,10) gs); +SELECT c.relname, a.atttypid::regtype AS chunk_id_type + FROM pg_class AS c, pg_attribute AS a + WHERE c.relname IN ('test_toast_oid', 'test_toast_oid8') AND + a.attrelid = c.reltoastrelid AND a.attname = 'chunk_id' + ORDER BY c.relname COLLATE "C"; +SELECT * FROM verify_heapam('test_toast_oid', check_toast := true); +SELECT * FROM verify_heapam('test_toast_oid8', check_toast := true); + -- Check that indexes are rejected CREATE INDEX test_index ON test_partition (a); SELECT * FROM verify_heapam('test_index', @@ -139,6 +164,8 @@ SELECT * FROM verify_heapam('test_foreign_table', endblock := NULL); -- cleanup +DROP TABLE test_toast_oid; +DROP TABLE test_toast_oid8; DROP TABLE heaptest; DROP TABLESPACE regress_test_stats_tblspc; DROP TABLE test_partition; diff --git a/contrib/test_decoding/expected/toast.out b/contrib/test_decoding/expected/toast.out index a757e7dc8d54..8bef6cc32af4 100644 --- a/contrib/test_decoding/expected/toast.out +++ b/contrib/test_decoding/expected/toast.out @@ -382,6 +382,45 @@ SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', COMMIT (4 rows) +-- Test decoding of TOAST values with oid8 +CREATE TABLE toasted_oid8 (id serial primary key, data text) + WITH (toast_value_type = 'oid8'); +-- uncompressed external toast data +ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL; +INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000)); +-- compressed external toast data +ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTENDED; +INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000)); +-- update without changing the toasted column, reported as unchanged +UPDATE toasted_oid8 SET id = id + 10 WHERE id = 1; +-- update changing the toasted column +UPDATE toasted_oid8 SET data = repeat('abcdefghij', 20000) WHERE id = 2; +DELETE FROM toasted_oid8; +-- Check that the values are reassembled in full. +SELECT regexp_replace(data, '^(.{60}).*(.{20})$', '\1..\2') AS shortened, + length(data) AS len + FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); + shortened | len +------------------------------------------------------------------------------------+-------- + BEGIN | 5 + table public.toasted_oid8: INSERT: id[integer]:1 data[text]:..2345678901234567890' | 200062 + COMMIT | 6 + BEGIN | 5 + table public.toasted_oid8: INSERT: id[integer]:2 data[text]:..2345678901234567890' | 200062 + COMMIT | 6 + BEGIN | 5 + table public.toasted_oid8: UPDATE: old-key: id[integer]:1 ne..nchanged-toast-datum | 116 + COMMIT | 6 + BEGIN | 5 + table public.toasted_oid8: UPDATE: id[integer]:2 data[text]:..bcdefghijabcdefghij' | 200062 + COMMIT | 6 + BEGIN | 5 + table public.toasted_oid8: DELETE: id[integer]:11 | 49 + table public.toasted_oid8: DELETE: id[integer]:2 | 48 + COMMIT | 6 +(16 rows) + +DROP TABLE toasted_oid8; SELECT pg_drop_replication_slot('regression_slot'); pg_drop_replication_slot -------------------------- diff --git a/contrib/test_decoding/sql/toast.sql b/contrib/test_decoding/sql/toast.sql index d1c560a174d6..8a2c49a5c9a7 100644 --- a/contrib/test_decoding/sql/toast.sql +++ b/contrib/test_decoding/sql/toast.sql @@ -324,4 +324,24 @@ INSERT INTO tbl2 VALUES(1); commit; SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); +-- Test decoding of TOAST values with oid8 +CREATE TABLE toasted_oid8 (id serial primary key, data text) + WITH (toast_value_type = 'oid8'); +-- uncompressed external toast data +ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTERNAL; +INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000)); +-- compressed external toast data +ALTER TABLE toasted_oid8 ALTER COLUMN data SET STORAGE EXTENDED; +INSERT INTO toasted_oid8(data) VALUES (repeat('1234567890', 20000)); +-- update without changing the toasted column, reported as unchanged +UPDATE toasted_oid8 SET id = id + 10 WHERE id = 1; +-- update changing the toasted column +UPDATE toasted_oid8 SET data = repeat('abcdefghij', 20000) WHERE id = 2; +DELETE FROM toasted_oid8; +-- Check that the values are reassembled in full. +SELECT regexp_replace(data, '^(.{60}).*(.{20})$', '\1..\2') AS shortened, + length(data) AS len + FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); +DROP TABLE toasted_oid8; + SELECT pg_drop_replication_slot('regression_slot'); -- 2.55.0
From 5199347bfc9a3daa45a2058c721167b5d0b1ca1a Mon Sep 17 00:00:00 2001 From: Michael Paquier <[email protected]> Date: Fri, 11 Sep 2026 15:57:31 +0900 Subject: [PATCH v17 5/5] Add support for TOAST pointers as oid8 This introduces a new varlena external pointer structure, varatt_external_oid8, that carries a 64-bit value ID (Oid8) for TOAST tables using oid8 as their chunk_id type. XXX: Catalog version bump required. --- src/include/access/detoast.h | 49 ++++ src/include/access/heaptoast.h | 22 +- src/include/utils/rel.h | 9 + src/include/utils/relcache.h | 1 + src/include/varatt.h | 78 ++++-- src/backend/access/common/detoast.c | 75 +++--- src/backend/access/common/toast_compression.c | 8 +- src/backend/access/common/toast_internals.c | 245 ++++++++++-------- src/backend/access/heap/heaptoast.c | 2 +- src/backend/access/table/toast_helper.c | 22 +- .../replication/logical/reorderbuffer.c | 15 +- src/backend/utils/adt/varlena.c | 8 +- src/backend/utils/cache/relcache.c | 37 +++ doc/src/sgml/storage.sgml | 12 +- contrib/amcheck/verify_heapam.c | 57 ++-- src/tools/pgindent/typedefs.list | 2 + 16 files changed, 430 insertions(+), 212 deletions(-) diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 2b1428b1e81c..e9d8076f77cd 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -12,6 +12,8 @@ #ifndef DETOAST_H #define DETOAST_H +#include "varatt.h" + /* * Macro to fetch the possibly-unaligned contents of an EXTERNAL datum * into a local "varatt_external_oid" toast pointer. This should be @@ -30,9 +32,56 @@ do { \ /* Size of an EXTERNAL datum that contains a standard TOAST pointer */ #define TOAST_OID_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid)) +/* Size of an EXTERNAL datum that contains an Oid8 TOAST pointer */ +#define TOAST_OID8_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_external_oid8)) + /* Size of an EXTERNAL datum that contains an indirection pointer */ #define INDIRECT_POINTER_SIZE (VARHDRSZ_EXTERNAL + sizeof(varatt_indirect)) +/* + * Decoded contents of an on-disk TOAST pointer, independent of whether the + * value ID type. + */ +typedef struct toast_external_data +{ + vartag_external tag; /* VARTAG_ONDISK_* */ + int32 rawsize; /* original data size (includes header) */ + uint32 extinfo; /* saved size + compression method */ + Oid8 valueid; /* value ID (can be widened from Oid) */ + Oid toastrelid; /* OID of the TOAST table containing it */ +} toast_external_data; + +/* + * Decode an on-disk TOAST pointer into a toast_external_data. + */ +static inline void +toast_external_info_get(const struct varlena *attr, toast_external_data *toast_ext_data) +{ + Assert(VARATT_IS_EXTERNAL_ONDISK(attr)); + + toast_ext_data->tag = VARTAG_EXTERNAL(attr); + if (toast_ext_data->tag == VARTAG_ONDISK_OID8) + { + varatt_external_oid8 toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + toast_ext_data->rawsize = toast_pointer.va_rawsize; + toast_ext_data->extinfo = toast_pointer.va_extinfo; + toast_ext_data->valueid = VARATT_EXTERNAL_OID8_GET_VALUEID(toast_pointer); + toast_ext_data->toastrelid = toast_pointer.va_toastrelid; + } + else + { + varatt_external_oid toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + toast_ext_data->rawsize = toast_pointer.va_rawsize; + toast_ext_data->extinfo = toast_pointer.va_extinfo; + toast_ext_data->valueid = toast_pointer.va_valueid; + toast_ext_data->toastrelid = toast_pointer.va_toastrelid; + } +} + /* ---------- * detoast_external_attr() - * diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h index aaae6e0fef69..4c3fda8a5f9a 100644 --- a/src/include/access/heaptoast.h +++ b/src/include/access/heaptoast.h @@ -69,13 +69,14 @@ /* * When we store an oversize datum externally, we divide it into chunks - * containing at most TOAST_OID_MAX_CHUNK_SIZE data bytes. This number *must* - * be small enough that the completed toast-table tuple (including the - * ID and sequence fields and all overhead) will fit on a page. + * containing at most TOAST_OID_MAX_CHUNK_SIZE or TOAST_OID8_MAX_CHUNK_SIZE + * data bytes, depending on the chunk_id type of the TOAST table. These + * numbers *must* be small enough that the completed toast-table tuple + * (including the ID and sequence fields and all overhead) will fit on a page. * The coding here sets the size on the theory that we want to fit * EXTERN_TUPLES_PER_PAGE tuples of maximum size onto a page. * - * NB: Changing TOAST_OID_MAX_CHUNK_SIZE requires an initdb. + * NB: Changing these values requires an initdb. */ #define EXTERN_TUPLES_PER_PAGE 4 /* tweak only this */ @@ -88,6 +89,19 @@ sizeof(int32) - \ VARHDRSZ) +#define TOAST_OID8_MAX_CHUNK_SIZE \ + (EXTERN_TUPLE_MAX_SIZE - \ + MAXALIGN(SizeofHeapTupleHeader) - \ + sizeof(Oid8) - \ + sizeof(int32) - \ + VARHDRSZ) + +/* + * Select the chunk size for a TOAST table from its value type. + */ +#define TOAST_MAX_CHUNK_SIZE(toast_typid) \ + ((toast_typid) == OID8OID ? TOAST_OID8_MAX_CHUNK_SIZE : TOAST_OID_MAX_CHUNK_SIZE) + /* ---------- * heap_toast_insert_or_update - * diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index d61432de705e..916b67f5504f 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -250,6 +250,15 @@ typedef struct RelationData */ Oid rd_toastoid; /* Real TOAST table's OID, or InvalidOid */ + /* + * Type OID of the "chunk_id" column of this relation's TOAST table, i.e. + * OIDOID or OID8OID. + * + * This data is filled on demand by RelationGetToastChunkIdType(), not + * at relcache build time, so as to save in syscache lookups. + */ + Oid rd_toastchunkidtype; + bool pgstat_enabled; /* should relation stats be counted */ /* use "struct" here to avoid needing to include pgstat.h: */ struct PgStat_RelationStatus *pgstat_info; /* statistics collection area */ diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h index 89c27aa1529f..e17c84908332 100644 --- a/src/include/utils/relcache.h +++ b/src/include/utils/relcache.h @@ -57,6 +57,7 @@ extern List *RelationGetIndexList(Relation relation); extern List *RelationGetStatExtList(Relation relation); extern Oid RelationGetPrimaryKeyIndex(Relation relation, bool deferrable_ok); extern Oid RelationGetReplicaIndex(Relation relation); +extern Oid RelationGetToastChunkIdType(Relation relation); extern List *RelationGetIndexExpressions(Relation relation); extern List *RelationGetDummyIndexExpressions(Relation relation); extern List *RelationGetIndexPredicate(Relation relation); diff --git a/src/include/varatt.h b/src/include/varatt.h index 89b47db08332..77041f7673a9 100644 --- a/src/include/varatt.h +++ b/src/include/varatt.h @@ -42,6 +42,44 @@ StaticAssertDecl((sizeof(int32) + sizeof(uint32) + 2 * sizeof(Oid)) == sizeof(varatt_external_oid), "varatt_external_oid should have no padding"); +/* + * varatt_external_oid8 is a "TOAST pointer" for TOAST tables that use + * Oid8 (64-bit) as their chunk_id type. Same layout as varatt_external_oid + * except for the value ID, which is 8 bytes wide. The value ID is split + * into two uint32 to force alignment. + * + * This struct must not contain any padding, because we sometimes compare + * these pointers using memcmp. + */ +typedef struct varatt_external_oid8 +{ + int32 va_rawsize; /* Original data size (includes header) */ + uint32 va_extinfo; /* External saved size (without header) and + * compression method */ + uint32 va_valueid_lo; /* Low 32 bits of value ID */ + uint32 va_valueid_hi; /* High 32 bits of value ID */ + Oid va_toastrelid; /* RelID of TOAST table containing it */ +} varatt_external_oid8; + +StaticAssertDecl((sizeof(int32) + 3 * sizeof(uint32) + sizeof(Oid)) == + sizeof(varatt_external_oid8), + "varatt_external_oid8 should have no padding"); + +/* Get/set the 64-bit value ID from the lo/hi halves */ +static inline Oid8 +VARATT_EXTERNAL_OID8_GET_VALUEID(varatt_external_oid8 toast_pointer) +{ + return ((Oid8) toast_pointer.va_valueid_lo) | + (((Oid8) toast_pointer.va_valueid_hi) << 32); +} + +static inline void +VARATT_EXTERNAL_OID8_SET_VALUEID(varatt_external_oid8 *toast_pointer, Oid8 id) +{ + toast_pointer->va_valueid_lo = (uint32) id; + toast_pointer->va_valueid_hi = (uint32) (id >> 32); +} + /* * These macros define the "saved size" portion of va_extinfo. Its remaining * two high-order bits identify the compression method. @@ -91,6 +129,7 @@ typedef enum vartag_external VARTAG_INDIRECT = 1, VARTAG_EXPANDED_RO = 2, VARTAG_EXPANDED_RW = 3, + VARTAG_ONDISK_OID8 = 4, VARTAG_ONDISK_OID = 18 } vartag_external; @@ -112,6 +151,8 @@ VARTAG_SIZE(vartag_external tag) return sizeof(varatt_expanded); else if (tag == VARTAG_ONDISK_OID) return sizeof(varatt_external_oid); + else if (tag == VARTAG_ONDISK_OID8) + return sizeof(varatt_external_oid8); else { Assert(false); @@ -365,7 +406,12 @@ VARATT_IS_EXTERNAL(const void *PTR) static inline bool VARATT_IS_EXTERNAL_ONDISK(const void *PTR) { - return VARATT_IS_EXTERNAL(PTR) && VARTAG_EXTERNAL(PTR) == VARTAG_ONDISK_OID; + vartag_external tag; + + if (!VARATT_IS_EXTERNAL(PTR)) + return false; + tag = VARTAG_EXTERNAL(PTR); + return (tag == VARTAG_ONDISK_OID || tag == VARTAG_ONDISK_OID8); } /* Is varlena datum an indirect pointer? */ @@ -508,43 +554,31 @@ VARDATA_COMPRESSED_GET_COMPRESS_METHOD(const void *PTR) } /* - * Same for external Datums; but note argument is a struct - * varatt_external_oid. + * Same for external Datums, saved into an va_extinfo. */ static inline Size -VARATT_EXTERNAL_OID_GET_EXTSIZE(varatt_external_oid toast_pointer) +VARATT_EXTINFO_GET_EXTSIZE(uint32 extinfo) { - return toast_pointer.va_extinfo & VARLENA_EXTSIZE_MASK; + return extinfo & VARLENA_EXTSIZE_MASK; } static inline uint32 -VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(varatt_external_oid toast_pointer) +VARATT_EXTINFO_GET_COMPRESS_METHOD(uint32 extinfo) { - return toast_pointer.va_extinfo >> VARLENA_EXTSIZE_BITS; + return extinfo >> VARLENA_EXTSIZE_BITS; } -/* Set size and compress method of an externally-stored varlena datum */ -/* This has to remain a macro; beware multiple evaluations! */ -#define VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, len, cm) \ - do { \ - Assert((cm) == TOAST_PGLZ_COMPRESSION_ID || \ - (cm) == TOAST_LZ4_COMPRESSION_ID); \ - ((toast_pointer).va_extinfo = \ - (len) | ((uint32) (cm) << VARLENA_EXTSIZE_BITS)); \ - } while (0) - /* - * Testing whether an externally-stored value is compressed now requires - * comparing size stored in va_extinfo (the actual length of the external data) + * Testing whether an externally-stored value is compressed requires comparing + * the saved size stored in va_extinfo (the actual length of the external data) * to rawsize (the original uncompressed datum's size). The latter includes * VARHDRSZ overhead, the former doesn't. We never use compression unless it * actually saves space, so we expect either equality or less-than. */ static inline bool -VARATT_EXTERNAL_OID_IS_COMPRESSED(varatt_external_oid toast_pointer) +VARATT_EXTINFO_IS_COMPRESSED(uint32 extinfo, int32 rawsize) { - return VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer) < - (Size) (toast_pointer.va_rawsize - VARHDRSZ); + return VARATT_EXTINFO_GET_EXTSIZE(extinfo) < (Size) (rawsize - VARHDRSZ); } #endif diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index a50aea7b6b12..7e5863262b3d 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -225,12 +225,18 @@ detoast_attr_slice(varlena *attr, if (VARATT_IS_EXTERNAL_ONDISK(attr)) { - varatt_external_oid toast_pointer; + toast_external_data toast_ext_data; + int32 extsize; + uint32 compress_method; + bool is_compressed; - VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + toast_external_info_get(attr, &toast_ext_data); + extsize = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo); + compress_method = VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_ext_data.extinfo); + is_compressed = VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize); /* fast path for non-compressed external datums */ - if (!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer)) + if (!is_compressed) return toast_fetch_datum_slice(attr, sliceoffset, slicelength); /* @@ -240,7 +246,7 @@ detoast_attr_slice(varlena *attr, */ if (slicelimit >= 0) { - int32 max_size = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer); + int32 max_size = extsize; /* * Determine maximum amount of compressed data needed for a prefix @@ -251,14 +257,9 @@ detoast_attr_slice(varlena *attr, * determine how much compressed data we need to be sure of being * able to decompress the required slice. */ - if (VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer) == - TOAST_PGLZ_COMPRESSION_ID) + if (compress_method == TOAST_PGLZ_COMPRESSION_ID) max_size = pglz_maximum_compressed_size(slicelimit, max_size); - /* - * Fetch enough compressed slices (compressed marker will get set - * automatically). - */ preslice = toast_fetch_datum_slice(attr, 0, max_size); } else @@ -344,20 +345,22 @@ toast_fetch_datum(varlena *attr) { Relation toastrel; varlena *result; - varatt_external_oid toast_pointer; + toast_external_data toast_ext_data; int32 attrsize; + Oid toastrelid; + Oid8 valueid; if (!VARATT_IS_EXTERNAL_ONDISK(attr)) elog(ERROR, "toast_fetch_datum shouldn't be called for non-ondisk datums"); - /* Must copy to access aligned fields */ - VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); - - attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer); + toast_external_info_get(attr, &toast_ext_data); + attrsize = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo); + toastrelid = toast_ext_data.toastrelid; + valueid = toast_ext_data.valueid; result = (varlena *) palloc(attrsize + VARHDRSZ); - if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer)) + if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize)) SET_VARSIZE_COMPRESSED(result, attrsize + VARHDRSZ); else SET_VARSIZE(result, attrsize + VARHDRSZ); @@ -369,10 +372,10 @@ toast_fetch_datum(varlena *attr) /* * Open the toast relation and its indexes */ - toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock); + toastrel = table_open(toastrelid, AccessShareLock); /* Fetch all chunks */ - table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid, + table_relation_fetch_toast_slice(toastrel, valueid, attrsize, 0, attrsize, result); /* Close toast table */ @@ -398,23 +401,27 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset, { Relation toastrel; varlena *result; - varatt_external_oid toast_pointer; + toast_external_data toast_ext_data; int32 attrsize; + Oid toastrelid; + Oid8 valueid; + bool is_compressed; if (!VARATT_IS_EXTERNAL_ONDISK(attr)) elog(ERROR, "toast_fetch_datum_slice shouldn't be called for non-ondisk datums"); - /* Must copy to access aligned fields */ - VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + toast_external_info_get(attr, &toast_ext_data); + attrsize = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo); + toastrelid = toast_ext_data.toastrelid; + valueid = toast_ext_data.valueid; + is_compressed = VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize); /* * It's nonsense to fetch slices of a compressed datum unless when it's a * prefix -- this isn't lo_* we can't return a compressed datum which is * meaningful to toast later. */ - Assert(!VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) || 0 == sliceoffset); - - attrsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer); + Assert(!is_compressed || 0 == sliceoffset); if (sliceoffset >= attrsize) { @@ -427,7 +434,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset, * space required by va_tcinfo, which is stored at the beginning as an * int32 value. */ - if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer) && slicelength > 0) + if (is_compressed && slicelength > 0) slicelength = slicelength + sizeof(int32); /* @@ -440,7 +447,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset, result = (varlena *) palloc(slicelength + VARHDRSZ); - if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer)) + if (is_compressed) SET_VARSIZE_COMPRESSED(result, slicelength + VARHDRSZ); else SET_VARSIZE(result, slicelength + VARHDRSZ); @@ -449,10 +456,10 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset, return result; /* Can save a lot of work at this point! */ /* Open the toast relation */ - toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock); + toastrel = table_open(toastrelid, AccessShareLock); /* Fetch all chunks */ - table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid, + table_relation_fetch_toast_slice(toastrel, valueid, attrsize, sliceoffset, slicelength, result); @@ -550,10 +557,10 @@ toast_raw_datum_size(Datum value) if (VARATT_IS_EXTERNAL_ONDISK(attr)) { /* va_rawsize is the size of the original datum -- including header */ - varatt_external_oid toast_pointer; + toast_external_data toast_ext_data; - VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); - result = toast_pointer.va_rawsize; + toast_external_info_get(attr, &toast_ext_data); + result = toast_ext_data.rawsize; } else if (VARATT_IS_EXTERNAL_INDIRECT(attr)) { @@ -610,10 +617,10 @@ toast_datum_size(Datum value) * compressed or not. We do not count the size of the toast pointer * ... should we? */ - varatt_external_oid toast_pointer; + toast_external_data toast_ext_data; - VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); - result = VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer); + toast_external_info_get(attr, &toast_ext_data); + result = VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo); } else if (VARATT_IS_EXTERNAL_INDIRECT(attr)) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c index 3b1bd6519261..09cdf880e4ff 100644 --- a/src/backend/access/common/toast_compression.c +++ b/src/backend/access/common/toast_compression.c @@ -262,12 +262,12 @@ toast_get_compression_id(varlena *attr) */ if (VARATT_IS_EXTERNAL_ONDISK(attr)) { - varatt_external_oid toast_pointer; + toast_external_data toast_ext_data; - VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + toast_external_info_get(attr, &toast_ext_data); - if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer)) - cmid = VARATT_EXTERNAL_OID_GET_COMPRESS_METHOD(toast_pointer); + if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize)) + cmid = VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_ext_data.extinfo); } else if (VARATT_IS_COMPRESSED(attr)) cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(attr); diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index f17e7b8813cb..328f98e338b9 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -25,7 +25,6 @@ #include "utils/fmgroids.h" #include "utils/rel.h" #include "utils/snapmgr.h" -#include "utils/lsyscache.h" static bool toastrel_valueid_exists(Relation toastrel, Oid8 valueid); static bool toastid_valueid_exists(Oid toastrelid, Oid8 valueid); @@ -104,6 +103,57 @@ toast_compress_datum(Datum value, char cmethod) } } +/* ---------- + * toast_preserve_valueid - + * + * During a table rewrite that preserves rd_toastoid, we want to preserve + * toast value IDs too. If the datum previously had an external value from + * that same toast table, return its value ID so the caller can re-use it, + * otherwise return InvalidOid8. + * + * This works for both Oid and Oid8 value IDs, as the value ID is decoded + * independently of the pointer's vartag. + * + * There is a corner case here: the table rewrite might have to copy both + * live and recently-dead versions of a row, and those versions could easily + * reference the same toast value. When we copy the second or later version + * of such a row, preserving the value ID means we select one that's already + * in the new toast table. We detect that and set *data_todo to 0 so the + * caller falls through without writing the data again. + * + * While annoying and ugly-looking, this is a good thing because it ensures + * that we wind up with only one copy of the toast value when there is only + * one copy in the old toast table. Before we detected this case, we'd have + * made multiple copies, wasting space; and what's worse, the copies + * belonging to already-deleted heap tuples would not be reclaimed by VACUUM. + * ---------- + */ +static Oid8 +toast_preserve_valueid(Relation toastrel, Oid toastoid, + varlena *oldexternal, int32 *data_todo) +{ + toast_external_data old; + + if (!OidIsValid(toastoid) || oldexternal == NULL) + return InvalidOid8; + + Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal)); + toast_external_info_get(oldexternal, &old); + + /* + * Only re-use the value ID if the old pointer came from this same toast + * table. Since that is the very same relation, its chunk_id type + * necessarily matches, so no separate vartag check is needed. + */ + if (old.toastrelid != toastoid) + return InvalidOid8; + + if (toastrel_valueid_exists(toastrel, old.valueid)) + *data_todo = 0; + + return old.valueid; +} + /* ---------- * toast_save_datum - * @@ -125,14 +175,20 @@ toast_save_datum(Relation rel, Datum value, TupleDesc toasttupDesc; CommandId mycid = GetCurrentCommandId(true); varlena *result; - varatt_external_oid toast_pointer; int32 chunk_seq = 0; char *data_p; int32 data_todo; Pointer dval = DatumGetPointer(value); int num_indexes; int validIndex; - Oid toast_typid = get_atttype(rel->rd_rel->reltoastrelid, 1); + Oid toast_typid = RelationGetToastChunkIdType(rel); + int32 max_chunk_size; + + /* Fields that will be assembled into the TOAST pointer at the end */ + int32 va_rawsize; + uint32 va_extinfo; + Oid8 va_valueid; + Oid va_toastrelid; Assert(!VARATT_IS_EXTERNAL(dval)); @@ -164,28 +220,32 @@ toast_save_datum(Relation rel, Datum value, { data_p = VARDATA_SHORT(dval); data_todo = VARSIZE_SHORT(dval) - VARHDRSZ_SHORT; - toast_pointer.va_rawsize = data_todo + VARHDRSZ; /* as if not short */ - toast_pointer.va_extinfo = data_todo; + va_rawsize = data_todo + VARHDRSZ; /* as if not short */ + va_extinfo = data_todo; } else if (VARATT_IS_COMPRESSED(dval)) { + uint32 cmid = VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval); + data_p = VARDATA(dval); data_todo = VARSIZE(dval) - VARHDRSZ; /* rawsize in a compressed datum is just the size of the payload */ - toast_pointer.va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ; + va_rawsize = VARDATA_COMPRESSED_GET_EXTSIZE(dval) + VARHDRSZ; /* set external size and compression method */ - VARATT_EXTERNAL_SET_SIZE_AND_COMPRESS_METHOD(toast_pointer, data_todo, - VARDATA_COMPRESSED_GET_COMPRESS_METHOD(dval)); + Assert(cmid == TOAST_PGLZ_COMPRESSION_ID || + cmid == TOAST_LZ4_COMPRESSION_ID); + va_extinfo = data_todo | (cmid << VARLENA_EXTSIZE_BITS); + /* Assert that the numbers look like it's compressed */ - Assert(VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer)); + Assert(VARATT_EXTINFO_IS_COMPRESSED(va_extinfo, va_rawsize)); } else { data_p = VARDATA(dval); data_todo = VARSIZE(dval) - VARHDRSZ; - toast_pointer.va_rawsize = VARSIZE(dval); - toast_pointer.va_extinfo = data_todo; + va_rawsize = VARSIZE(dval); + va_extinfo = data_todo; } /* @@ -197,101 +257,49 @@ toast_save_datum(Relation rel, Datum value, * if we have to substitute such an OID. */ if (OidIsValid(rel->rd_toastoid)) - toast_pointer.va_toastrelid = rel->rd_toastoid; + va_toastrelid = rel->rd_toastoid; else - toast_pointer.va_toastrelid = RelationGetRelid(toastrel); + va_toastrelid = RelationGetRelid(toastrel); /* - * Choose a new value to use as the value ID for this toast value, be it - * for OID or OID8 TOAST relations. - * - * Normally we just choose an unused value within the toast table. But - * during table-rewriting operations where we are preserving an existing - * toast table OID, we want to preserve toast value IDs too. So, if - * rd_toastoid is set and we had a prior external value from that same - * toast table, re-use its value ID. If we didn't have a prior external - * value (which is a corner case, but possible if the table's attstorage - * options have been changed), we have to pick a value ID that doesn't - * conflict with either new or existing toast value IDs. If the TOAST - * table uses 8-byte value IDs, we should not really care much about - * that. + * Choose the value ID for this toast value. During a table rewrite that + * preserves rd_toastoid we re-use the prior value ID if we can (see + * toast_preserve_valueid); otherwise we pick a fresh one. For Oid value + * IDs the fresh one must not conflict with old or new toast values; for + * Oid8 the ID space is large enough that conflicts are not a concern. */ - if (!OidIsValid(rel->rd_toastoid)) + if (toast_typid == OID8OID) { - /* normal case: just choose an unused OID */ - if (toast_typid == OID8OID) - toast_pointer.va_valueid = GetNewObjectId8(); - else + va_valueid = toast_preserve_valueid(toastrel, rel->rd_toastoid, + oldexternal, &data_todo); + if (va_valueid == InvalidOid8) { - toast_pointer.va_valueid = - GetNewOidWithIndex(toastrel, - RelationGetRelid(toastidxs[validIndex]), - (AttrNumber) 1); + /* The Oid8 space is large enough that we need not check conflicts */ + va_valueid = GetNewObjectId8(); } } else { - /* rewrite case: check to see if value was in old toast table */ - toast_pointer.va_valueid = InvalidOid; - if (oldexternal != NULL) - { - varatt_external_oid old_toast_pointer; - - Assert(VARATT_IS_EXTERNAL_ONDISK(oldexternal)); - /* Must copy to access aligned fields */ - VARATT_EXTERNAL_GET_POINTER(old_toast_pointer, oldexternal); - if (old_toast_pointer.va_toastrelid == rel->rd_toastoid) - { - /* This value came from the old toast table; reuse its OID */ - toast_pointer.va_valueid = old_toast_pointer.va_valueid; - - /* - * There is a corner case here: the table rewrite might have - * to copy both live and recently-dead versions of a row, and - * those versions could easily reference the same toast value. - * When we copy the second or later version of such a row, - * reusing the OID will mean we select an OID that's already - * in the new toast table. Check for that, and if so, just - * fall through without writing the data again. - * - * While annoying and ugly-looking, this is a good thing - * because it ensures that we wind up with only one copy of - * the toast value when there is only one copy in the old - * toast table. Before we detected this case, we'd have made - * multiple copies, wasting space; and what's worse, the - * copies belonging to already-deleted heap tuples would not - * be reclaimed by VACUUM. - */ - if (toastrel_valueid_exists(toastrel, - toast_pointer.va_valueid)) - { - /* Match, so short-circuit the data storage loop below */ - data_todo = 0; - } - } - } - if (toast_pointer.va_valueid == InvalidOid) + va_valueid = toast_preserve_valueid(toastrel, rel->rd_toastoid, + oldexternal, &data_todo); + if (va_valueid == InvalidOid) { /* - * new value; must choose a value that doesn't conflict in either - * old or new toast table. + * Choose an unused OID. During a rewrite we must also avoid IDs + * that are still present in the old toast table. */ - if (toast_typid == OID8OID) - toast_pointer.va_valueid = GetNewObjectId8(); - else - { - do - { - toast_pointer.va_valueid = - GetNewOidWithIndex(toastrel, - RelationGetRelid(toastidxs[validIndex]), - (AttrNumber) 1); - } while (toastid_valueid_exists(rel->rd_toastoid, - toast_pointer.va_valueid)); - } + do + va_valueid = + GetNewOidWithIndex(toastrel, + RelationGetRelid(toastidxs[validIndex]), + (AttrNumber) 1); + while (OidIsValid(rel->rd_toastoid) && + toastid_valueid_exists(rel->rd_toastoid, va_valueid)); } } + max_chunk_size = TOAST_MAX_CHUNK_SIZE(toast_typid); + /* * Split up the item into chunks */ @@ -313,15 +321,15 @@ toast_save_datum(Relation rel, Datum value, /* * Calculate the size of this chunk */ - chunk_size = Min(TOAST_OID_MAX_CHUNK_SIZE, data_todo); + chunk_size = Min(max_chunk_size, data_todo); /* * Build a tuple and store it */ if (toast_typid == OID8OID) - t_values[0] = ObjectId8GetDatum(toast_pointer.va_valueid); + t_values[0] = ObjectId8GetDatum(va_valueid); else - t_values[0] = ObjectIdGetDatum(toast_pointer.va_valueid); + t_values[0] = ObjectIdGetDatum((Oid) va_valueid); t_values[1] = Int32GetDatum(chunk_seq++); SET_VARSIZE(&chunk_data, chunk_size + VARHDRSZ); memcpy(VARDATA(&chunk_data), data_p, chunk_size); @@ -375,11 +383,34 @@ toast_save_datum(Relation rel, Datum value, table_close(toastrel, NoLock); /* - * Create the TOAST pointer value that we'll return + * Create the TOAST pointer value that we'll return. */ - result = (varlena *) palloc(TOAST_OID_POINTER_SIZE); - SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID); - memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer)); + if (toast_typid == OID8OID) + { + varatt_external_oid8 toast_pointer; + + toast_pointer.va_rawsize = va_rawsize; + toast_pointer.va_extinfo = va_extinfo; + VARATT_EXTERNAL_OID8_SET_VALUEID(&toast_pointer, va_valueid); + toast_pointer.va_toastrelid = va_toastrelid; + + result = (varlena *) palloc(TOAST_OID8_POINTER_SIZE); + SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID8); + memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer)); + } + else + { + varatt_external_oid toast_pointer; + + toast_pointer.va_rawsize = va_rawsize; + toast_pointer.va_extinfo = va_extinfo; + toast_pointer.va_valueid = (Oid) va_valueid; + toast_pointer.va_toastrelid = va_toastrelid; + + result = (varlena *) palloc(TOAST_OID_POINTER_SIZE); + SET_VARTAG_EXTERNAL(result, VARTAG_ONDISK_OID); + memcpy(VARDATA_EXTERNAL(result), &toast_pointer, sizeof(toast_pointer)); + } return PointerGetDatum(result); } @@ -415,7 +446,7 @@ void toast_delete_datum(Relation rel, Datum value, bool is_speculative) { varlena *attr = (varlena *) DatumGetPointer(value); - varatt_external_oid toast_pointer; + toast_external_data toast_ext_data; Relation toastrel; Relation *toastidxs; ScanKeyData toastkey; @@ -427,26 +458,26 @@ toast_delete_datum(Relation rel, Datum value, bool is_speculative) if (!VARATT_IS_EXTERNAL_ONDISK(attr)) return; - /* Must copy to access aligned fields */ - VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + /* + * Decode the pointer to get the toast relation OID and value ID. The + * vartag tells us everything we need - no TOAST table schema lookup + * required. + */ + toast_external_info_get(attr, &toast_ext_data); /* * Open the toast relation and its indexes */ - toastrel = table_open(toast_pointer.va_toastrelid, RowExclusiveLock); - - /* Fetch valid relation used for process */ + toastrel = table_open(toast_ext_data.toastrelid, RowExclusiveLock); validIndex = toast_open_indexes(toastrel, RowExclusiveLock, &toastidxs, &num_indexes); - /* - * Setup a scan key to find chunks with matching va_valueid - */ + /* Set up a scan key to find chunks with matching value ID */ toast_valueid_scankey_init(&toastkey, (AttrNumber) 1, TupleDescAttr(toastrel->rd_att, 0)->atttypid, - toast_pointer.va_valueid); + toast_ext_data.valueid); /* * Find all the chunks. (We don't actually care whether we see them in diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c index b9c89593ecf4..a3d5cdecb3c1 100644 --- a/src/backend/access/heap/heaptoast.c +++ b/src/backend/access/heap/heaptoast.c @@ -649,7 +649,7 @@ heap_fetch_toast_slice(Relation toastrel, Oid8 valueid, int32 attrsize, &num_indexes); toast_typid = TupleDescAttr(toastrel->rd_att, 0)->atttypid; - max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE; + max_chunk_size = TOAST_MAX_CHUNK_SIZE(toast_typid); totalchunks = ((attrsize - 1) / max_chunk_size) + 1; startchunk = sliceoffset / max_chunk_size; diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index 6f2d691cc681..2613d9dc095b 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -69,12 +69,16 @@ toast_tuple_init(ToastTupleContext *ttc) /* * If the old value is stored on disk, check if it has changed so * we have to delete it later. + * + * Note that TOAST pointers could have different vartags, for oid + * or oid8, and these can have different sizes. */ if (att->attlen == -1 && !ttc->ttc_oldisnull[i] && VARATT_IS_EXTERNAL_ONDISK(old_value)) { if (ttc->ttc_isnull[i] || !VARATT_IS_EXTERNAL_ONDISK(new_value) || + VARTAG_EXTERNAL(old_value) != VARTAG_EXTERNAL(new_value) || memcmp(old_value, new_value, VARSIZE_EXTERNAL(old_value)) != 0) { @@ -171,8 +175,9 @@ toast_tuple_init(ToastTupleContext *ttc) * The column must have attstorage EXTERNAL or EXTENDED if check_main is * false, and must have attstorage MAIN if check_main is true. * - * The column must have a minimum size of MAXALIGN(TOAST_OID_POINTER_SIZE); - * if not, no benefit is to be expected by compressing it. + * The column must be larger than the TOAST pointer that would replace it; + * if not, no benefit is to be expected by compressing it. Note that this + * choice depends on the TOAST value type, oid or oid8. * * The return value is the index of the biggest suitable column, or * -1 if there is none. @@ -184,10 +189,21 @@ toast_tuple_find_biggest_attribute(ToastTupleContext *ttc, TupleDesc tupleDesc = ttc->ttc_rel->rd_att; int numAttrs = tupleDesc->natts; int biggest_attno = -1; - int32 biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE); + int32 biggest_size; int32 skip_colflags = TOASTCOL_IGNORE; int i; + /* + * Size of the TOAST pointer this relation would use. A relation without + * a TOAST table cannot have any of its attributes moved out-of-line, but + * it can still have some of them compressed. Fall back to the oid size + * in that case. + */ + if (RelationGetToastChunkIdType(ttc->ttc_rel) == OID8OID) + biggest_size = MAXALIGN(TOAST_OID8_POINTER_SIZE); + else + biggest_size = MAXALIGN(TOAST_OID_POINTER_SIZE); + if (for_compression) skip_colflags |= TOASTCOL_INCOMPRESSIBLE; diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index dd4b6e8f7f45..69a479cd31da 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -5175,7 +5175,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn, varlena *varlena_pointer; /* va_rawsize is the size of the original datum -- including header */ - varatt_external_oid toast_pointer; + toast_external_data toast_ext_data; varatt_indirect redirect_pointer; varlena *new_datum = NULL; varlena *reconstructed; @@ -5198,11 +5198,11 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn, varlena_pointer = (varlena *) DatumGetPointer(attrs[natt]); /* no need to do anything if the tuple isn't external */ - if (!VARATT_IS_EXTERNAL(varlena_pointer)) + if (!VARATT_IS_EXTERNAL_ONDISK(varlena_pointer)) continue; - VARATT_EXTERNAL_GET_POINTER(toast_pointer, varlena_pointer); - toast_valueid = toast_pointer.va_valueid; + toast_external_info_get(varlena_pointer, &toast_ext_data); + toast_valueid = toast_ext_data.valueid; /* * Check whether the toast tuple changed, replace if so. @@ -5220,7 +5220,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn, free[natt] = true; - reconstructed = palloc0(toast_pointer.va_rawsize); + reconstructed = palloc0(toast_ext_data.rawsize); ent->reconstructed = reconstructed; @@ -5245,10 +5245,11 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn, VARSIZE(chunk) - VARHDRSZ); data_done += VARSIZE(chunk) - VARHDRSZ; } - Assert(data_done == VARATT_EXTERNAL_OID_GET_EXTSIZE(toast_pointer)); + + Assert(data_done == VARATT_EXTINFO_GET_EXTSIZE(toast_ext_data.extinfo)); /* make sure its marked as compressed or not */ - if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer)) + if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize)) SET_VARSIZE_COMPRESSED(reconstructed, data_done + VARHDRSZ); else SET_VARSIZE(reconstructed, data_done + VARHDRSZ); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 4533dac11a6d..f8c16bd4c871 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -4261,7 +4261,8 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS) { int typlen; varlena *attr; - varatt_external_oid toast_pointer; + toast_external_data toast_ext_data; + Oid8 result; /* On first call, get the input type's typlen, and save at *fn_extra */ if (fcinfo->flinfo->fn_extra == NULL) @@ -4288,9 +4289,10 @@ pg_column_toast_chunk_id(PG_FUNCTION_ARGS) if (!VARATT_IS_EXTERNAL_ONDISK(attr)) PG_RETURN_NULL(); - VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + toast_external_info_get(attr, &toast_ext_data); + result = toast_ext_data.valueid; - PG_RETURN_OID8(toast_pointer.va_valueid); + PG_RETURN_OID8(result); } /* diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index f475d7039772..65dfdbda100a 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -1205,6 +1205,9 @@ retry: relation->rd_fkeylist = NIL; relation->rd_fkeyvalid = false; + /* TOAST type data is not loaded till asked for */ + relation->rd_toastchunkidtype = InvalidOid; + /* partitioning data is not loaded till asked for */ relation->rd_partkey = NULL; relation->rd_partkeycxt = NULL; @@ -5093,6 +5096,39 @@ RelationGetReplicaIndex(Relation relation) return relation->rd_replidindex; } +/* + * RelationGetToastChunkIdType -- get the type of the relation's TOAST + * table "chunk_id" column + * + * Returns OIDOID or OID8OID, or InvalidOid if the relation has no TOAST + * table. + */ +Oid +RelationGetToastChunkIdType(Relation relation) +{ + Oid toastrelid = relation->rd_rel->reltoastrelid; + Oid typid; + + /* Quick exit if we already computed the value */ + if (OidIsValid(relation->rd_toastchunkidtype)) + return relation->rd_toastchunkidtype; + + /* Nothing to report without a TOAST table */ + if (!OidIsValid(toastrelid)) + return InvalidOid; + + typid = get_atttype(toastrelid, 1); + if (!OidIsValid(typid)) + elog(ERROR, "cache lookup failed for TOAST relation %u", + toastrelid); + if (typid != OIDOID && typid != OID8OID) + elog(ERROR, "unexpected type %u for chunk_id in TOAST relation %u", + typid, toastrelid); + + relation->rd_toastchunkidtype = typid; + return typid; +} + /* * RelationGetIndexExpressions -- get the index expressions for an index * @@ -6530,6 +6566,7 @@ load_relcache_init_file(bool shared) rel->rd_firstRelfilelocatorSubid = InvalidSubTransactionId; rel->rd_droppedSubid = InvalidSubTransactionId; rel->rd_amcache = NULL; + rel->rd_toastchunkidtype = InvalidOid; rel->pgstat_info = NULL; /* diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index 21bd67c11ea0..83de016eaa58 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -417,8 +417,10 @@ described in more detail below. <para> Out-of-line values are divided (after compression if used) into chunks of at -most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> bytes (by default this value is chosen -so that four chunk rows will fit on a page, making it about 2000 bytes). +most <symbol>TOAST_OID_MAX_CHUNK_SIZE</symbol> or +<symbol>TOAST_OID8_MAX_CHUNK_SIZE</symbol> bytes depending on the +<structfield>chunk_id</structfield> type (by default these values are chosen +so that four chunk rows will fit on a page, making them about 2000 bytes). Each chunk is stored as a separate row in the <acronym>TOAST</acronym> table belonging to the owning table. Every <acronym>TOAST</acronym> table has the columns @@ -434,8 +436,10 @@ retrieval of the values. A pointer datum representing an out-of-line on-disk logical datum size (original uncompressed data length), physical stored size (different if compression was applied), and the compression method used, if any. Allowing for the varlena header bytes, -the total size of an on-disk <acronym>TOAST</acronym> pointer datum is therefore 18 -bytes regardless of the actual size of the represented value. +the total size of an on-disk <acronym>TOAST</acronym> pointer datum is 18 +bytes when using an OID as <structfield>chunk_id</structfield>, or 22 bytes +when using an OID8 as <structfield>chunk_id</structfield>, regardless of the +actual size of the represented value. </para> <para> diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c index c89fab8764ac..edecbd792b0c 100644 --- a/contrib/amcheck/verify_heapam.c +++ b/contrib/amcheck/verify_heapam.c @@ -74,7 +74,9 @@ typedef enum SkipPages */ typedef struct ToastedAttribute { - varatt_external_oid toast_pointer; + vartag_external tag; /* VARTAG_ONDISK_OID or VARTAG_ONDISK_OID8 */ + Oid8 va_valueid; /* value ID (works for both Oid and Oid8) */ + uint32 va_extinfo; /* external size and compression method */ BlockNumber blkno; /* block in main table */ OffsetNumber offnum; /* offset in main table */ AttrNumber attnum; /* attribute in main table */ @@ -1565,9 +1567,12 @@ check_toast_tuple(HeapTuple toasttup, HeapCheckContext *ctx, int32 max_chunk_size; Oid8 toast_valueid; - max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE; + toast_valueid = ta->va_valueid; + + max_chunk_size = ta->tag == VARTAG_ONDISK_OID8 + ? TOAST_OID8_MAX_CHUNK_SIZE + : TOAST_OID_MAX_CHUNK_SIZE; last_chunk_seq = (extsize - 1) / max_chunk_size; - toast_valueid = ta->toast_pointer.va_valueid; /* Sanity-check the sequence number. */ chunk_seq = DatumGetInt32(fastgetattr(toasttup, 2, @@ -1671,8 +1676,11 @@ check_tuple_attribute(HeapCheckContext *ctx) char *tp; /* pointer to the tuple data */ uint16 infomask; Oid8 toast_pointer_valueid; + int32 va_rawsize; + uint32 va_extinfo; CompactAttribute *thisatt; - varatt_external_oid toast_pointer; + vartag_external va_tag_value; + toast_external_data toast_ext_data; infomask = ctx->tuphdr->t_infomask; thisatt = TupleDescCompactAttr(RelationGetDescr(ctx->rel), ctx->attnum); @@ -1731,7 +1739,7 @@ check_tuple_attribute(HeapCheckContext *ctx) { uint8 va_tag = VARTAG_EXTERNAL(tp + ctx->offset); - if (va_tag != VARTAG_ONDISK_OID) + if (va_tag != VARTAG_ONDISK_OID && va_tag != VARTAG_ONDISK_OID8) { report_corruption(ctx, psprintf("toasted attribute has unexpected TOAST tag %u", @@ -1775,27 +1783,28 @@ check_tuple_attribute(HeapCheckContext *ctx) /* It is external, and we're looking at a page on disk */ - /* - * Must copy attr into toast_pointer for alignment considerations - */ - VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); - toast_pointer_valueid = toast_pointer.va_valueid; + /* Must copy attr into a decoded pointer for alignment considerations */ + toast_external_info_get(attr, &toast_ext_data); + va_tag_value = toast_ext_data.tag; + toast_pointer_valueid = toast_ext_data.valueid; + va_rawsize = toast_ext_data.rawsize; + va_extinfo = toast_ext_data.extinfo; /* Toasted attributes too large to be untoasted should never be stored */ - if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT) + if (va_rawsize > VARLENA_SIZE_LIMIT) report_corruption(ctx, psprintf("toast value " OID8_FORMAT " rawsize %d exceeds limit %d", toast_pointer_valueid, - toast_pointer.va_rawsize, + va_rawsize, VARLENA_SIZE_LIMIT)); - if (VARATT_EXTERNAL_OID_IS_COMPRESSED(toast_pointer)) + if (VARATT_EXTINFO_IS_COMPRESSED(toast_ext_data.extinfo, toast_ext_data.rawsize)) { ToastCompressionId cmid; bool valid = false; /* Compressed attributes should have a valid compression method */ - cmid = TOAST_COMPRESS_METHOD(&toast_pointer); + cmid = VARATT_EXTINFO_GET_COMPRESS_METHOD(toast_ext_data.extinfo); switch (cmid) { /* List of all valid compression method IDs */ @@ -1849,7 +1858,10 @@ check_tuple_attribute(HeapCheckContext *ctx) ta = palloc0_object(ToastedAttribute); - VARATT_EXTERNAL_GET_POINTER(ta->toast_pointer, attr); + /* The pointer has already been decoded above, just reuse it */ + ta->tag = va_tag_value; + ta->va_valueid = toast_pointer_valueid; + ta->va_extinfo = va_extinfo; ta->blkno = ctx->blkno; ta->offnum = ctx->offnum; ta->attnum = ctx->attnum; @@ -1879,17 +1891,18 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta) Oid8 toast_valueid; Oid toast_typid; - toast_typid = TupleDescAttr(ctx->toast_rel->rd_att, 0)->atttypid; - max_chunk_size = TOAST_OID_MAX_CHUNK_SIZE; - - extsize = VARATT_EXTERNAL_OID_GET_EXTSIZE(ta->toast_pointer); - last_chunk_seq = (extsize - 1) / max_chunk_size; + toast_valueid = ta->va_valueid; + extsize = VARATT_EXTINFO_GET_EXTSIZE(ta->va_extinfo); + toast_typid = (ta->tag == VARTAG_ONDISK_OID8) ? OID8OID : OIDOID; /* * Setup a scan key to find chunks in toast table with matching va_valueid */ + max_chunk_size = TOAST_MAX_CHUNK_SIZE(toast_typid); toast_valueid_scankey_init(&toastkey, (AttrNumber) 1, toast_typid, - ta->toast_pointer.va_valueid); + toast_valueid); + + last_chunk_seq = (extsize - 1) / max_chunk_size; /* * Check if any chunks for this toasted object exist in the toast table, @@ -1909,8 +1922,6 @@ check_toasted_attribute(HeapCheckContext *ctx, ToastedAttribute *ta) } systable_endscan_ordered(toastscan); - toast_valueid = ta->toast_pointer.va_valueid; - if (!found_toasttup) report_toast_corruption(ctx, ta, psprintf("toast value " OID8_FORMAT " not found in toast table", diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index de21cea65f96..9985c8b21e31 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -4358,6 +4358,7 @@ timeout_params timerCA tlist_vinfo toast_compress_header +toast_external_data tokenize_error_callback_arg transferMode transfer_thread_arg @@ -4414,6 +4415,7 @@ vacuumingOptions validate_string_relopt varatt_expanded varatt_external_oid +varatt_external_oid8 varatt_indirect varattrib_1b varattrib_1b_e -- 2.55.0
signature.asc
Description: PGP signature
