On 28/03/2026 02:14, Tomas Vondra wrote:
* 0002 - +1 to getting rid of HASH_SEGMENT, but I don't see the point of renaming DEF_SEGSIZE to HASH_SEGSIZE. Isn't that a bit unnecessary?
DEF_SEGSIZE stands for "default segsize", but after this commit it's not merely the default, it's the same hard-coded constant for every hash table. That's why it seems prudent to rename it.
* 0003 - I'd probably rename CurrentDynaHashCxt to something that doesn't seem like a "global" variable, e.g. "dynahashCxt"
Renamed it to "hcxt", as that's what the corresponding field in HTAB is called.
* 0004 - seems fine, +1 to get rid of unused pieces
To be clear, the init_size/max_size are not completely unused at the moment: the lock manager sets max_size to 2 * init_size, and wait_event.c used constants 16 and 128.
The point is that it doesn't give you a very wide range of scalability, and I think it's better to not be flexible in that fashion. I would call it sloppiness rather than flexibility.
* 0005 - seems fine * 0006 - Doesn't this completely change the alignment? ShmemHashAlloc used to call ShmemAllocRaw, which is very careful to use CACHELINEALIGN. But now ShmemHashAlloc just does MAXALIGN, which ShmemAllocRaw claims is not enough on modern systems.
dynahash.c allocates multiple elements in each alloc() call, so even though ShmemAllocRaw() returns a cacheline-aligned block, the individual elements were not cacheline-aligned before this patch either. See element_alloc() and choose_nelem_alloc().
* 0007 - this left one comment referencing HASH_DIRSIZE in dynahash.c
Fixed - Heikki
From 2c8a2f6f2a45c6a736f2f5ad494e74acf05180aa Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas <[email protected]> Date: Mon, 30 Mar 2026 18:20:52 +0300 Subject: [PATCH v2 1/8] Use ShmemInitStruct to allocate shmem for semaphores This makes them visible visible in pg_shmem_allocations Reviewed-by: Tomas Vondra <[email protected]> Discussion: https://www.postgresql.org/message-id/[email protected] --- src/backend/port/posix_sema.c | 4 +++- src/backend/port/sysv_sema.c | 5 ++++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/src/backend/port/posix_sema.c b/src/backend/port/posix_sema.c index e368e5ee7ed..40205b7d400 100644 --- a/src/backend/port/posix_sema.c +++ b/src/backend/port/posix_sema.c @@ -196,6 +196,7 @@ void PGReserveSemaphores(int maxSemas) { struct stat statbuf; + bool found; /* * We use the data directory's inode number to seed the search for free @@ -216,7 +217,8 @@ PGReserveSemaphores(int maxSemas) #else sharedSemas = (PGSemaphore) - ShmemAlloc(PGSemaphoreShmemSize(maxSemas)); + ShmemInitStruct("Semaphores", PGSemaphoreShmemSize(maxSemas), &found); + Assert(!found); #endif numSems = 0; diff --git a/src/backend/port/sysv_sema.c b/src/backend/port/sysv_sema.c index 86c4d359ef7..4b2bf84072f 100644 --- a/src/backend/port/sysv_sema.c +++ b/src/backend/port/sysv_sema.c @@ -330,6 +330,7 @@ void PGReserveSemaphores(int maxSemas) { struct stat statbuf; + bool found; /* * We use the data directory's inode number to seed the search for free @@ -344,7 +345,9 @@ PGReserveSemaphores(int maxSemas) DataDir))); sharedSemas = (PGSemaphore) - ShmemAlloc(PGSemaphoreShmemSize(maxSemas)); + ShmemInitStruct("Semaphores", PGSemaphoreShmemSize(maxSemas), &found); + Assert(!found); + numSharedSemas = 0; maxSharedSemas = maxSemas; -- 2.47.3
From 6b2a6d9e7f718cfc0db43d88815ce2852b7f4957 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas <[email protected]> Date: Mon, 30 Mar 2026 18:20:56 +0300 Subject: [PATCH v2 2/8] Remove HASH_SEGMENT option It's been unused forever. There's no urgency in removing it now, but it was just something that caught my eye. Aleksander Alekseev proposed this a long time ago [0], but Tom Lane was worried about third-party extensions using it. I believe that to not be an issue; I tried grepping through all extensions found on github and didn't find any references to it. [0] https://www.postgresql.org/message-id/20160418180711.55ac82c0@fujitsu Reviewed-by: Tomas Vondra <[email protected]> Discussion: https://www.postgresql.org/message-id/[email protected] --- src/backend/utils/hash/dynahash.c | 65 +++++++++++-------------------- src/include/utils/hsearch.h | 4 +- 2 files changed, 23 insertions(+), 46 deletions(-) diff --git a/src/backend/utils/hash/dynahash.c b/src/backend/utils/hash/dynahash.c index 11f5778eba1..af3f7142afb 100644 --- a/src/backend/utils/hash/dynahash.c +++ b/src/backend/utils/hash/dynahash.c @@ -108,11 +108,11 @@ /* * Constants * - * A hash table has a top-level "directory", each of whose entries points - * to a "segment" of ssize bucket headers. The maximum number of hash - * buckets is thus dsize * ssize (but dsize may be expansible). Of course, - * the number of records in the table can be larger, but we don't want a - * whole lot of records per bucket or performance goes down. + * A hash table has a top-level "directory", each of whose entries points to a + * "segment" of HASH_SEGSIZE bucket headers. The maximum number of hash + * buckets is thus dsize * HASH_SEGSIZE (but dsize may be expansible). Of + * course, the number of records in the table can be larger, but we don't want + * a whole lot of records per bucket or performance goes down. * * In a hash table allocated in shared memory, the directory cannot be * expanded because it must stay at a fixed address. The directory size @@ -120,8 +120,8 @@ * a good idea of the maximum number of entries!). For non-shared hash * tables, the initial directory size can be left at the default. */ -#define DEF_SEGSIZE 256 -#define DEF_SEGSIZE_SHIFT 8 /* must be log2(DEF_SEGSIZE) */ +#define HASH_SEGSIZE 256 +#define HASH_SEGSIZE_SHIFT 8 /* must be log2(HASH_SEGSIZE) */ #define DEF_DIRSIZE 256 /* Number of freelists to be used for a partitioned hash table. */ @@ -192,8 +192,6 @@ struct HASHHDR Size entrysize; /* total user element size in bytes */ int64 num_partitions; /* # partitions (must be power of 2), or 0 */ int64 max_dsize; /* 'dsize' limit if directory is fixed size */ - int64 ssize; /* segment size --- must be power of 2 */ - int sshift; /* segment shift = log2(ssize) */ int nelem_alloc; /* number of entries to allocate at once */ bool isfixed; /* if true, don't enlarge */ @@ -235,8 +233,6 @@ struct HTAB /* We keep local copies of these fixed values to reduce contention */ Size keysize; /* hash key length in bytes */ - int64 ssize; /* segment size --- must be power of 2 */ - int sshift; /* segment shift = log2(ssize) */ /* * In a USE_VALGRIND build, non-shared hashtables keep an slist chain of @@ -499,8 +495,6 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) /* make local copies of some heavily-used values */ hctl = hashp->hctl; hashp->keysize = hctl->keysize; - hashp->ssize = hctl->ssize; - hashp->sshift = hctl->sshift; return hashp; } @@ -544,14 +538,6 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) hctl->num_partitions = info->num_partitions; } - if (flags & HASH_SEGMENT) - { - hctl->ssize = info->ssize; - hctl->sshift = my_log2(info->ssize); - /* ssize had better be a power of 2 */ - Assert(hctl->ssize == (1L << hctl->sshift)); - } - /* * SHM hash tables have fixed directory size passed by the caller. */ @@ -567,8 +553,6 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) /* make local copies of heavily-used constant fields */ hashp->keysize = hctl->keysize; - hashp->ssize = hctl->ssize; - hashp->sshift = hctl->sshift; /* Build the hash directory structure */ if (!init_htab(hashp, nelem)) @@ -649,9 +633,6 @@ hdefault(HTAB *hashp) /* table has no fixed maximum size */ hctl->max_dsize = NO_MAX_DSIZE; - hctl->ssize = DEF_SEGSIZE; - hctl->sshift = DEF_SEGSIZE_SHIFT; - hctl->isfixed = false; /* can be enlarged */ #ifdef HASH_STATISTICS @@ -733,7 +714,7 @@ init_htab(HTAB *hashp, int64 nelem) /* * Figure number of directory segments needed, round up to a power of 2 */ - nsegs = (nbuckets - 1) / hctl->ssize + 1; + nsegs = (nbuckets - 1) / HASH_SEGSIZE + 1; nsegs = next_pow2_int(nsegs); /* @@ -793,7 +774,7 @@ hash_estimate_size(int64 num_entries, Size entrysize) /* estimate number of buckets wanted */ nBuckets = next_pow2_int64(num_entries); /* # of segments needed for nBuckets */ - nSegments = next_pow2_int64((nBuckets - 1) / DEF_SEGSIZE + 1); + nSegments = next_pow2_int64((nBuckets - 1) / HASH_SEGSIZE + 1); /* directory entries */ nDirEntries = DEF_DIRSIZE; while (nDirEntries < nSegments) @@ -805,7 +786,7 @@ hash_estimate_size(int64 num_entries, Size entrysize) size = add_size(size, mul_size(nDirEntries, sizeof(HASHSEGMENT))); /* segments */ size = add_size(size, mul_size(nSegments, - MAXALIGN(DEF_SEGSIZE * sizeof(HASHBUCKET)))); + MAXALIGN(HASH_SEGSIZE * sizeof(HASHBUCKET)))); /* elements --- allocated in groups of choose_nelem_alloc() entries */ elementAllocCnt = choose_nelem_alloc(entrysize); nElementAllocs = (num_entries - 1) / elementAllocCnt + 1; @@ -836,7 +817,7 @@ hash_select_dirsize(int64 num_entries) /* estimate number of buckets wanted */ nBuckets = next_pow2_int64(num_entries); /* # of segments needed for nBuckets */ - nSegments = next_pow2_int64((nBuckets - 1) / DEF_SEGSIZE + 1); + nSegments = next_pow2_int64((nBuckets - 1) / HASH_SEGSIZE + 1); /* directory entries */ nDirEntries = DEF_DIRSIZE; while (nDirEntries < nSegments) @@ -1417,7 +1398,6 @@ hash_seq_search(HASH_SEQ_STATUS *status) HTAB *hashp; HASHHDR *hctl; uint32 max_bucket; - int64 ssize; int64 segment_num; int64 segment_ndx; HASHSEGMENT segp; @@ -1457,7 +1437,6 @@ hash_seq_search(HASH_SEQ_STATUS *status) curBucket = status->curBucket; hashp = status->hashp; hctl = hashp->hctl; - ssize = hashp->ssize; max_bucket = hctl->max_bucket; if (curBucket > max_bucket) @@ -1469,8 +1448,8 @@ hash_seq_search(HASH_SEQ_STATUS *status) /* * first find the right segment in the table directory. */ - segment_num = curBucket >> hashp->sshift; - segment_ndx = MOD(curBucket, ssize); + segment_num = curBucket >> HASH_SEGSIZE_SHIFT; + segment_ndx = MOD(curBucket, HASH_SEGSIZE); segp = hashp->dir[segment_num]; @@ -1489,7 +1468,7 @@ hash_seq_search(HASH_SEQ_STATUS *status) hash_seq_term(status); return NULL; /* search is done */ } - if (++segment_ndx >= ssize) + if (++segment_ndx >= HASH_SEGSIZE) { segment_num++; segment_ndx = 0; @@ -1566,8 +1545,8 @@ expand_table(HTAB *hashp) #endif new_bucket = hctl->max_bucket + 1; - new_segnum = new_bucket >> hashp->sshift; - new_segndx = MOD(new_bucket, hashp->ssize); + new_segnum = new_bucket >> HASH_SEGSIZE_SHIFT; + new_segndx = MOD(new_bucket, HASH_SEGSIZE); if (new_segnum >= hctl->nsegs) { @@ -1606,8 +1585,8 @@ expand_table(HTAB *hashp) * split at this point. With a different way of reducing the hash value, * that might not be true! */ - old_segnum = old_bucket >> hashp->sshift; - old_segndx = MOD(old_bucket, hashp->ssize); + old_segnum = old_bucket >> HASH_SEGSIZE_SHIFT; + old_segndx = MOD(old_bucket, HASH_SEGSIZE); old_seg = hashp->dir[old_segnum]; new_seg = hashp->dir[new_segnum]; @@ -1684,12 +1663,12 @@ seg_alloc(HTAB *hashp) HASHSEGMENT segp; CurrentDynaHashCxt = hashp->hcxt; - segp = (HASHSEGMENT) hashp->alloc(sizeof(HASHBUCKET) * hashp->ssize); + segp = (HASHSEGMENT) hashp->alloc(sizeof(HASHBUCKET) * HASH_SEGSIZE); if (!segp) return NULL; - MemSet(segp, 0, sizeof(HASHBUCKET) * hashp->ssize); + MemSet(segp, 0, sizeof(HASHBUCKET) * HASH_SEGSIZE); return segp; } @@ -1786,8 +1765,8 @@ hash_initial_lookup(HTAB *hashp, uint32 hashvalue, HASHBUCKET **bucketptr) bucket = calc_bucket(hctl, hashvalue); - segment_num = bucket >> hashp->sshift; - segment_ndx = MOD(bucket, hashp->ssize); + segment_num = bucket >> HASH_SEGSIZE_SHIFT; + segment_ndx = MOD(bucket, HASH_SEGSIZE); segp = hashp->dir[segment_num]; diff --git a/src/include/utils/hsearch.h b/src/include/utils/hsearch.h index cdc530fb732..0721849c036 100644 --- a/src/include/utils/hsearch.h +++ b/src/include/utils/hsearch.h @@ -66,8 +66,6 @@ typedef struct HASHCTL { /* Used if HASH_PARTITION flag is set: */ int64 num_partitions; /* # partitions (must be power of 2) */ - /* Used if HASH_SEGMENT flag is set: */ - int64 ssize; /* segment size */ /* Used if HASH_DIRSIZE flag is set: */ int64 dsize; /* (initial) directory size */ int64 max_dsize; /* limit to dsize if dir size is limited */ @@ -90,7 +88,7 @@ typedef struct HASHCTL /* Flag bits for hash_create; most indicate which parameters are supplied */ #define HASH_PARTITION 0x0001 /* Hashtable is used w/partitioned locking */ -#define HASH_SEGMENT 0x0002 /* Set segment size */ +/* 0x0002 is unused */ #define HASH_DIRSIZE 0x0004 /* Set directory size (initial and max) */ #define HASH_ELEM 0x0008 /* Set keysize and entrysize (now required!) */ #define HASH_STRINGS 0x0010 /* Select support functions for string keys */ -- 2.47.3
From 684d8f9f9c22636c628af1dc5b5694b214e1d194 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas <[email protected]> Date: Mon, 30 Mar 2026 18:20:58 +0300 Subject: [PATCH v2 3/8] Change the signature of dynahash's alloc function Instead of passing the current memory context to the alloc function via a shared variable, pass it directly as an argument. Reviewed-by: Tomas Vondra <[email protected]> Discussion: https://www.postgresql.org/message-id/[email protected] --- src/backend/storage/ipc/shmem.c | 15 +++++++-- src/backend/utils/hash/dynahash.c | 52 ++++++++++++++++--------------- src/include/utils/hsearch.h | 3 +- 3 files changed, 42 insertions(+), 28 deletions(-) diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c index 3cc38450949..a6fd4c30059 100644 --- a/src/backend/storage/ipc/shmem.c +++ b/src/backend/storage/ipc/shmem.c @@ -100,6 +100,7 @@ typedef struct ShmemAllocatorData #define ShmemIndexLock (&ShmemAllocator->index_lock) +static void *ShmemHashAlloc(Size size, void *alloc_arg); static void *ShmemAllocRaw(Size size, Size *allocated_size); /* shared memory global variables */ @@ -184,7 +185,8 @@ InitShmemAllocator(PGShmemHeader *seghdr) info.keysize = SHMEM_INDEX_KEYSIZE; info.entrysize = sizeof(ShmemIndexEnt); info.dsize = info.max_dsize = hash_select_dirsize(SHMEM_INDEX_SIZE); - info.alloc = ShmemAllocNoError; + info.alloc = ShmemHashAlloc; + info.alloc_arg = NULL; hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE; if (!IsUnderPostmaster) { @@ -233,6 +235,14 @@ ShmemAllocNoError(Size size) return ShmemAllocRaw(size, &allocated_size); } +static void * +ShmemHashAlloc(Size size, void *alloc_arg) +{ + Size allocated_size; + + return ShmemAllocRaw(size, &allocated_size); +} + /* * ShmemAllocRaw -- allocate align chunk and return allocated size * @@ -339,7 +349,8 @@ ShmemInitHash(const char *name, /* table string name for shmem index */ * The shared memory allocator must be specified too. */ infoP->dsize = infoP->max_dsize = hash_select_dirsize(max_size); - infoP->alloc = ShmemAllocNoError; + infoP->alloc = ShmemHashAlloc; + infoP->alloc_arg = NULL; hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE; /* look it up in the shmem index */ diff --git a/src/backend/utils/hash/dynahash.c b/src/backend/utils/hash/dynahash.c index af3f7142afb..16cf103b8e4 100644 --- a/src/backend/utils/hash/dynahash.c +++ b/src/backend/utils/hash/dynahash.c @@ -224,6 +224,7 @@ struct HTAB HashCompareFunc match; /* key comparison function */ HashCopyFunc keycopy; /* key copying function */ HashAllocFunc alloc; /* memory allocator */ + void *alloc_arg; /* opaque argument to pass to allocator function */ MemoryContext hcxt; /* memory context if default allocator used */ char *tabname; /* table name (for error messages) */ bool isshared; /* true if table is in shared memory */ @@ -264,7 +265,7 @@ struct HTAB /* * Private function prototypes */ -static void *DynaHashAlloc(Size size); +static void *DynaHashAlloc(Size size, void *alloc_arg); static HASHSEGMENT seg_alloc(HTAB *hashp); static bool element_alloc(HTAB *hashp, int nelem, int freelist_idx); static bool dir_realloc(HTAB *hashp); @@ -287,14 +288,13 @@ static bool has_seq_scans(HTAB *hashp); /* * memory allocation support */ -static MemoryContext CurrentDynaHashCxt = NULL; - static void * -DynaHashAlloc(Size size) +DynaHashAlloc(Size size, void *alloc_arg) { - Assert(MemoryContextIsValid(CurrentDynaHashCxt)); - return MemoryContextAllocExtended(CurrentDynaHashCxt, size, - MCXT_ALLOC_NO_OOM); + MemoryContext cxt = (MemoryContext) alloc_arg; + + Assert(MemoryContextIsValid(cxt)); + return MemoryContextAllocExtended(cxt, size, MCXT_ALLOC_NO_OOM); } @@ -355,6 +355,7 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) { HTAB *hashp; HASHHDR *hctl; + MemoryContext hcxt; /* * Hash tables now allocate space for key and data, but you have to say @@ -377,22 +378,21 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) if (flags & HASH_SHARED_MEM) { /* Set up to allocate the hash header */ - CurrentDynaHashCxt = TopMemoryContext; + hcxt = TopMemoryContext; } else { /* Create the hash table's private memory context */ if (flags & HASH_CONTEXT) - CurrentDynaHashCxt = info->hcxt; + hcxt = info->hcxt; else - CurrentDynaHashCxt = TopMemoryContext; - CurrentDynaHashCxt = AllocSetContextCreate(CurrentDynaHashCxt, - "dynahash", - ALLOCSET_DEFAULT_SIZES); + hcxt = TopMemoryContext; + hcxt = AllocSetContextCreate(hcxt, "dynahash", + ALLOCSET_DEFAULT_SIZES); } /* Initialize the hash header, plus a copy of the table name */ - hashp = (HTAB *) MemoryContextAlloc(CurrentDynaHashCxt, + hashp = (HTAB *) MemoryContextAlloc(hcxt, sizeof(HTAB) + strlen(tabname) + 1); MemSet(hashp, 0, sizeof(HTAB)); @@ -401,7 +401,7 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) /* If we have a private context, label it with hashtable's name */ if (!(flags & HASH_SHARED_MEM)) - MemoryContextSetIdentifier(CurrentDynaHashCxt, hashp->tabname); + MemoryContextSetIdentifier(hcxt, hashp->tabname); /* * Select the appropriate hash function (see comments at head of file). @@ -473,9 +473,15 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) /* And select the entry allocation function, too. */ if (flags & HASH_ALLOC) + { hashp->alloc = info->alloc; + hashp->alloc_arg = info->alloc_arg; + } else + { hashp->alloc = DynaHashAlloc; + hashp->alloc_arg = hcxt; + } if (flags & HASH_SHARED_MEM) { @@ -504,13 +510,13 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) /* setup hash table defaults */ hashp->hctl = NULL; hashp->dir = NULL; - hashp->hcxt = CurrentDynaHashCxt; + hashp->hcxt = hcxt; hashp->isshared = false; } if (!hashp->hctl) { - hashp->hctl = (HASHHDR *) hashp->alloc(sizeof(HASHHDR)); + hashp->hctl = (HASHHDR *) hashp->alloc(sizeof(HASHHDR), hashp->alloc_arg); if (!hashp->hctl) ereport(ERROR, (errcode(ERRCODE_OUT_OF_MEMORY), @@ -732,9 +738,8 @@ init_htab(HTAB *hashp, int64 nelem) /* Allocate a directory */ if (!(hashp->dir)) { - CurrentDynaHashCxt = hashp->hcxt; hashp->dir = (HASHSEGMENT *) - hashp->alloc(hctl->dsize * sizeof(HASHSEGMENT)); + hashp->alloc(hctl->dsize * sizeof(HASHSEGMENT), hashp->alloc_arg); if (!hashp->dir) return false; } @@ -1636,8 +1641,7 @@ dir_realloc(HTAB *hashp) new_dirsize = new_dsize * sizeof(HASHSEGMENT); old_p = hashp->dir; - CurrentDynaHashCxt = hashp->hcxt; - p = (HASHSEGMENT *) hashp->alloc((Size) new_dirsize); + p = (HASHSEGMENT *) hashp->alloc((Size) new_dirsize, hashp->alloc_arg); if (p != NULL) { @@ -1662,8 +1666,7 @@ seg_alloc(HTAB *hashp) { HASHSEGMENT segp; - CurrentDynaHashCxt = hashp->hcxt; - segp = (HASHSEGMENT) hashp->alloc(sizeof(HASHBUCKET) * HASH_SEGSIZE); + segp = (HASHSEGMENT) hashp->alloc(sizeof(HASHBUCKET) * HASH_SEGSIZE, hashp->alloc_arg); if (!segp) return NULL; @@ -1703,8 +1706,7 @@ element_alloc(HTAB *hashp, int nelem, int freelist_idx) #endif /* Allocate the memory. */ - CurrentDynaHashCxt = hashp->hcxt; - allocedBlock = hashp->alloc(requestSize); + allocedBlock = hashp->alloc(requestSize, hashp->alloc_arg); if (!allocedBlock) return false; diff --git a/src/include/utils/hsearch.h b/src/include/utils/hsearch.h index 0721849c036..03bc1c171cd 100644 --- a/src/include/utils/hsearch.h +++ b/src/include/utils/hsearch.h @@ -41,7 +41,7 @@ typedef void *(*HashCopyFunc) (void *dest, const void *src, Size keysize); * Note: there is no free function API; can't destroy a hashtable unless you * use the default allocator. */ -typedef void *(*HashAllocFunc) (Size request); +typedef void *(*HashAllocFunc) (Size request, void *alloc_arg); /* * HASHELEMENT is the private part of a hashtable entry. The caller's data @@ -80,6 +80,7 @@ typedef struct HASHCTL HashCopyFunc keycopy; /* key copying function */ /* Used if HASH_ALLOC flag is set: */ HashAllocFunc alloc; /* memory allocator */ + void *alloc_arg; /* opaque argument to pass to allocator function */ /* Used if HASH_CONTEXT flag is set: */ MemoryContext hcxt; /* memory context to use for allocations */ /* Used if HASH_SHARED_MEM flag is set: */ -- 2.47.3
From 939598de12ebbe94532e662b95fec5654483eebb Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas <[email protected]> Date: Mon, 30 Mar 2026 18:21:01 +0300 Subject: [PATCH v2 4/8] Merge init and max size options on shmem hash tables Replace the separate init and max size options with a single size option. We didn't make much use of the feature, most callers already used the same size for both. Allowing the table to grow much beyond the initial size is bad for performance anyway as we don't support resizing the hash table's dictionary part. Reviewed-by: Tomas Vondra <[email protected]> Discussion: https://www.postgresql.org/message-id/[email protected] --- .../pg_stat_statements/pg_stat_statements.c | 2 +- src/backend/storage/buffer/buf_table.c | 2 +- src/backend/storage/ipc/shmem.c | 16 ++++---------- src/backend/storage/lmgr/lock.c | 7 +----- src/backend/storage/lmgr/predicate.c | 3 --- src/backend/utils/activity/wait_event.c | 22 ++++++++----------- src/include/storage/shmem.h | 2 +- 7 files changed, 17 insertions(+), 37 deletions(-) diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c index 7975476b890..5494d41dca1 100644 --- a/contrib/pg_stat_statements/pg_stat_statements.c +++ b/contrib/pg_stat_statements/pg_stat_statements.c @@ -561,7 +561,7 @@ pgss_shmem_startup(void) info.keysize = sizeof(pgssHashKey); info.entrysize = sizeof(pgssEntry); pgss_hash = ShmemInitHash("pg_stat_statements hash", - pgss_max, pgss_max, + pgss_max, &info, HASH_ELEM | HASH_BLOBS); diff --git a/src/backend/storage/buffer/buf_table.c b/src/backend/storage/buffer/buf_table.c index 23d85fd32e2..d04ef74b850 100644 --- a/src/backend/storage/buffer/buf_table.c +++ b/src/backend/storage/buffer/buf_table.c @@ -60,7 +60,7 @@ InitBufTable(int size) info.num_partitions = NUM_BUFFER_PARTITIONS; SharedBufHash = ShmemInitHash("Shared Buffer Lookup Table", - size, size, + size, &info, HASH_ELEM | HASH_BLOBS | HASH_PARTITION | HASH_FIXED_SIZE); } diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c index a6fd4c30059..fd2c7c4590a 100644 --- a/src/backend/storage/ipc/shmem.c +++ b/src/backend/storage/ipc/shmem.c @@ -313,14 +313,7 @@ ShmemAddrIsValid(const void *addr) * table at once. (In practice, all creations are done in the postmaster * process; child processes should always be attaching to existing tables.) * - * max_size is the estimated maximum number of hashtable entries. This is - * not a hard limit, but the access efficiency will degrade if it is - * exceeded substantially (since it's used to compute directory size and - * the hash table buckets will get overfull). - * - * init_size is the number of hashtable entries to preallocate. For a table - * whose maximum size is certain, this should be equal to max_size; that - * ensures that no run-time out-of-shared-memory failures can occur. + * nelems is the maximum number of hashtable entries. * * *infoP and hash_flags must specify at least the entry sizes and key * comparison semantics (see hash_create()). Flag bits and values specific @@ -333,8 +326,7 @@ ShmemAddrIsValid(const void *addr) */ HTAB * ShmemInitHash(const char *name, /* table string name for shmem index */ - int64 init_size, /* initial table size */ - int64 max_size, /* max size of the table */ + int64 nelems, /* size of the table */ HASHCTL *infoP, /* info about key and bucket size */ int hash_flags) /* info about infoP */ { @@ -348,7 +340,7 @@ ShmemInitHash(const char *name, /* table string name for shmem index */ * * The shared memory allocator must be specified too. */ - infoP->dsize = infoP->max_dsize = hash_select_dirsize(max_size); + infoP->dsize = infoP->max_dsize = hash_select_dirsize(nelems); infoP->alloc = ShmemHashAlloc; infoP->alloc_arg = NULL; hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE; @@ -368,7 +360,7 @@ ShmemInitHash(const char *name, /* table string name for shmem index */ /* Pass location of hashtable header to hash_create */ infoP->hctl = (HASHHDR *) location; - return hash_create(name, init_size, infoP, hash_flags); + return hash_create(name, nelems, infoP, hash_flags); } /* diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c index 234643e4dd7..ae524bb02d6 100644 --- a/src/backend/storage/lmgr/lock.c +++ b/src/backend/storage/lmgr/lock.c @@ -445,8 +445,7 @@ void LockManagerShmemInit(void) { HASHCTL info; - int64 init_table_size, - max_table_size; + int64 max_table_size; bool found; /* @@ -454,7 +453,6 @@ LockManagerShmemInit(void) * calculations must agree with LockManagerShmemSize! */ max_table_size = NLOCKENTS(); - init_table_size = max_table_size / 2; /* * Allocate hash table for LOCK structs. This stores per-locked-object @@ -465,14 +463,12 @@ LockManagerShmemInit(void) info.num_partitions = NUM_LOCK_PARTITIONS; LockMethodLockHash = ShmemInitHash("LOCK hash", - init_table_size, max_table_size, &info, HASH_ELEM | HASH_BLOBS | HASH_PARTITION); /* Assume an average of 2 holders per lock */ max_table_size *= 2; - init_table_size *= 2; /* * Allocate hash table for PROCLOCK structs. This stores @@ -484,7 +480,6 @@ LockManagerShmemInit(void) info.num_partitions = NUM_LOCK_PARTITIONS; LockMethodProcLockHash = ShmemInitHash("PROCLOCK hash", - init_table_size, max_table_size, &info, HASH_ELEM | HASH_FUNCTION | HASH_PARTITION); diff --git a/src/backend/storage/lmgr/predicate.c b/src/backend/storage/lmgr/predicate.c index ae0e96aee5f..4c559b2d3ee 100644 --- a/src/backend/storage/lmgr/predicate.c +++ b/src/backend/storage/lmgr/predicate.c @@ -1182,7 +1182,6 @@ PredicateLockShmemInit(void) info.num_partitions = NUM_PREDICATELOCK_PARTITIONS; PredicateLockTargetHash = ShmemInitHash("PREDICATELOCKTARGET hash", - max_predicate_lock_targets, max_predicate_lock_targets, &info, HASH_ELEM | HASH_BLOBS | @@ -1218,7 +1217,6 @@ PredicateLockShmemInit(void) max_predicate_locks = max_predicate_lock_targets * 2; PredicateLockHash = ShmemInitHash("PREDICATELOCK hash", - max_predicate_locks, max_predicate_locks, &info, HASH_ELEM | HASH_FUNCTION | @@ -1297,7 +1295,6 @@ PredicateLockShmemInit(void) info.entrysize = sizeof(SERIALIZABLEXID); SerializableXidHash = ShmemInitHash("SERIALIZABLEXID hash", - max_serializable_xacts, max_serializable_xacts, &info, HASH_ELEM | HASH_BLOBS | diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index e5a2289f0b0..2b76967776c 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -56,16 +56,14 @@ uint32 *my_wait_event_info = &local_my_wait_event_info; * For simplicity, we use the same ID counter across types of custom events. * We could end that anytime the need arises. * - * The size of the hash table is based on the assumption that - * WAIT_EVENT_CUSTOM_HASH_INIT_SIZE is enough for most cases, and it seems - * unlikely that the number of entries will reach - * WAIT_EVENT_CUSTOM_HASH_MAX_SIZE. + * The size of the hash table is based on the assumption that usually only a + * handful of entries are needed, but since it's small in absolute terms + * anyway, we leave a generous amount of headroom. */ static HTAB *WaitEventCustomHashByInfo; /* find names from infos */ static HTAB *WaitEventCustomHashByName; /* find infos from names */ -#define WAIT_EVENT_CUSTOM_HASH_INIT_SIZE 16 -#define WAIT_EVENT_CUSTOM_HASH_MAX_SIZE 128 +#define WAIT_EVENT_CUSTOM_HASH_SIZE 128 /* hash table entries */ typedef struct WaitEventCustomEntryByInfo @@ -106,9 +104,9 @@ WaitEventCustomShmemSize(void) Size sz; sz = MAXALIGN(sizeof(WaitEventCustomCounterData)); - sz = add_size(sz, hash_estimate_size(WAIT_EVENT_CUSTOM_HASH_MAX_SIZE, + sz = add_size(sz, hash_estimate_size(WAIT_EVENT_CUSTOM_HASH_SIZE, sizeof(WaitEventCustomEntryByInfo))); - sz = add_size(sz, hash_estimate_size(WAIT_EVENT_CUSTOM_HASH_MAX_SIZE, + sz = add_size(sz, hash_estimate_size(WAIT_EVENT_CUSTOM_HASH_SIZE, sizeof(WaitEventCustomEntryByName))); return sz; } @@ -138,8 +136,7 @@ WaitEventCustomShmemInit(void) info.entrysize = sizeof(WaitEventCustomEntryByInfo); WaitEventCustomHashByInfo = ShmemInitHash("WaitEventCustom hash by wait event information", - WAIT_EVENT_CUSTOM_HASH_INIT_SIZE, - WAIT_EVENT_CUSTOM_HASH_MAX_SIZE, + WAIT_EVENT_CUSTOM_HASH_SIZE, &info, HASH_ELEM | HASH_BLOBS); @@ -148,8 +145,7 @@ WaitEventCustomShmemInit(void) info.entrysize = sizeof(WaitEventCustomEntryByName); WaitEventCustomHashByName = ShmemInitHash("WaitEventCustom hash by name", - WAIT_EVENT_CUSTOM_HASH_INIT_SIZE, - WAIT_EVENT_CUSTOM_HASH_MAX_SIZE, + WAIT_EVENT_CUSTOM_HASH_SIZE, &info, HASH_ELEM | HASH_STRINGS); } @@ -238,7 +234,7 @@ WaitEventCustomNew(uint32 classId, const char *wait_event_name) /* Allocate a new event Id */ SpinLockAcquire(&WaitEventCustomCounter->mutex); - if (WaitEventCustomCounter->nextId >= WAIT_EVENT_CUSTOM_HASH_MAX_SIZE) + if (WaitEventCustomCounter->nextId >= WAIT_EVENT_CUSTOM_HASH_SIZE) { SpinLockRelease(&WaitEventCustomCounter->mutex); ereport(ERROR, diff --git a/src/include/storage/shmem.h b/src/include/storage/shmem.h index 2a9e9becd26..0ae609aca8b 100644 --- a/src/include/storage/shmem.h +++ b/src/include/storage/shmem.h @@ -32,7 +32,7 @@ extern void InitShmemAllocator(PGShmemHeader *seghdr); extern void *ShmemAlloc(Size size); extern void *ShmemAllocNoError(Size size); extern bool ShmemAddrIsValid(const void *addr); -extern HTAB *ShmemInitHash(const char *name, int64 init_size, int64 max_size, +extern HTAB *ShmemInitHash(const char *name, int64 nelems, HASHCTL *infoP, int hash_flags); extern void *ShmemInitStruct(const char *name, Size size, bool *foundPtr); extern Size add_size(Size s1, Size s2); -- 2.47.3
From fba2dfb4f4068229e923d8b2d9b1ba62b3e1be11 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas <[email protected]> Date: Mon, 30 Mar 2026 18:21:04 +0300 Subject: [PATCH v2 5/8] Prevent shared memory hash tables from growing beyond initial size Set HASH_FIXED_SIZE on all shared memory hash tables, to prevent them from growing after the initial allocation. It was always weirdly indeterministic that if you e.g. took a lot of locks, consuming all the unused shared memory for the lock table, you couldn't use that space for other things anymore, but if you used it for other things first, then it was no longer available for locks. Better to be strict and predictable. Increase SHMEM_INDEX_SIZE because we were already above the old value, and it's now a hard limit. XXX Many callers of ShmemInitHash() still pass HASH_FIXED_SIZE, but that's now unnecessary because ShmemInitHash() will set it anyway. I didn't go clean them up yet. Reviewed-by: Tomas Vondra <[email protected]> Discussion: https://www.postgresql.org/message-id/[email protected] --- src/backend/storage/ipc/shmem.c | 20 ++++++++------------ src/include/storage/shmem.h | 4 ++-- 2 files changed, 10 insertions(+), 14 deletions(-) diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c index fd2c7c4590a..47065bb3603 100644 --- a/src/backend/storage/ipc/shmem.c +++ b/src/backend/storage/ipc/shmem.c @@ -24,9 +24,8 @@ * available to POSTGRES: fixed-size structures, queues and hash * tables. Fixed-size structures contain things like global variables * for a module and should never be allocated after the shared memory - * initialization phase. Hash tables have a fixed maximum size, but - * their actual size can vary dynamically. When entries are added - * to the table, more space is allocated. Queues link data structures + * initialization phase. Hash tables have a fixed maximum size and + * cannot grow beyond that. Queues link data structures * that have been allocated either within fixed-size structures or as hash * buckets. Each shared data structure has a string name to identify * it (assigned in the module that declares it). @@ -56,11 +55,7 @@ * * (d) memory allocation model: shared memory can never be * freed, once allocated. Each hash table has its own free list, - * so hash buckets can be reused when an item is deleted. However, - * if one hash table grows very large and then shrinks, its space - * cannot be redistributed to other tables. We could build a simple - * hash bucket garbage collector if need be. Right now, it seems - * unnecessary. + * so hash buckets can be reused when an item is deleted. */ #include "postgres.h" @@ -187,7 +182,7 @@ InitShmemAllocator(PGShmemHeader *seghdr) info.dsize = info.max_dsize = hash_select_dirsize(SHMEM_INDEX_SIZE); info.alloc = ShmemHashAlloc; info.alloc_arg = NULL; - hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE; + hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE; if (!IsUnderPostmaster) { size = hash_get_shared_size(&info, hash_flags); @@ -318,7 +313,7 @@ ShmemAddrIsValid(const void *addr) * *infoP and hash_flags must specify at least the entry sizes and key * comparison semantics (see hash_create()). Flag bits and values specific * to shared-memory hash tables are added here, except that callers may - * choose to specify HASH_PARTITION and/or HASH_FIXED_SIZE. + * choose to specify HASH_PARTITION. * * Note: before Postgres 9.0, this function returned NULL for some failure * cases. Now, it always throws error instead, so callers need not check @@ -336,14 +331,15 @@ ShmemInitHash(const char *name, /* table string name for shmem index */ /* * Hash tables allocated in shared memory have a fixed directory; it can't * grow or other backends wouldn't be able to find it. So, make sure we - * make it big enough to start with. + * make it big enough to start with. We also allocate all the buckets + * upfront, so hash tables cannot grow later. * * The shared memory allocator must be specified too. */ infoP->dsize = infoP->max_dsize = hash_select_dirsize(nelems); infoP->alloc = ShmemHashAlloc; infoP->alloc_arg = NULL; - hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE; + hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE; /* look it up in the shmem index */ location = ShmemInitStruct(name, diff --git a/src/include/storage/shmem.h b/src/include/storage/shmem.h index 0ae609aca8b..2fccbdb534c 100644 --- a/src/include/storage/shmem.h +++ b/src/include/storage/shmem.h @@ -46,8 +46,8 @@ extern void RequestAddinShmemSpace(Size size); /* size constants for the shmem index table */ /* max size of data structure string name */ #define SHMEM_INDEX_KEYSIZE (48) - /* estimated size of the shmem index table (not a hard limit) */ -#define SHMEM_INDEX_SIZE (64) + /* max number of named shmem structures and hash tables */ +#define SHMEM_INDEX_SIZE (256) /* this is a hash bucket in the shmem index table */ typedef struct -- 2.47.3
From 8ce617baf5976c95de9a386e7020f6a1c3504862 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas <[email protected]> Date: Mon, 30 Mar 2026 18:21:06 +0300 Subject: [PATCH v2 6/8] Allocate all parts of shmem hash table from a single contiguous area Previously, the shared header (HASHHDR) and the directory were allocated by the caller, and passed to hash_create(), while the actual elements were allocated separately with ShmemAlloc(). After this commit, all the memory needed by the header, the directory, and all the elements is allocated using a single ShmemInitStruct() call, and the different parts are carved out of that allocation. This way the ShmemIndex entries (and thus pg_shmem_allocations) reflect the size size of the whole hash table. Commit f5930f9a98 attempted this earlier, but it had to be reverted. The new strategy is to let dynahash perform all the allocations with the alloc function, but have the alloc function carve out the parts from the one larger allocation. The shared header and the directory are now also allocated with alloc calls, instead of passing the area for those directly from the caller. Reviewed-by: Tomas Vondra <[email protected]> Discussion: https://www.postgresql.org/message-id/[email protected] --- src/backend/storage/ipc/shmem.c | 71 +++++++++++++++++++++------- src/backend/utils/hash/dynahash.c | 78 +++++++++++++------------------ src/include/utils/hsearch.h | 5 +- src/tools/pgindent/typedefs.list | 1 + 4 files changed, 91 insertions(+), 64 deletions(-) diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c index 47065bb3603..c8171125871 100644 --- a/src/backend/storage/ipc/shmem.c +++ b/src/backend/storage/ipc/shmem.c @@ -112,6 +112,16 @@ static bool firstNumaTouch = true; Datum pg_numa_available(PG_FUNCTION_ARGS); +/* + * A very simple allocator used to carve out different parts of a hash table, + * from a previously allocated contiguous shared memory area. + */ +typedef struct shmem_hash_allocator +{ + char *next; /* start of free space in the area */ + char *end; /* end of the shmem area */ +} shmem_hash_allocator; + /* * InitShmemAllocator() --- set up basic pointers to shared memory. * @@ -126,7 +136,7 @@ InitShmemAllocator(PGShmemHeader *seghdr) Size offset; HASHCTL info; int hash_flags; - size_t size; + shmem_hash_allocator allocator; #ifndef EXEC_BACKEND Assert(!IsUnderPostmaster); @@ -182,15 +192,27 @@ InitShmemAllocator(PGShmemHeader *seghdr) info.dsize = info.max_dsize = hash_select_dirsize(SHMEM_INDEX_SIZE); info.alloc = ShmemHashAlloc; info.alloc_arg = NULL; - hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE; + hash_flags = HASH_ELEM | HASH_STRINGS + | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE; + if (!IsUnderPostmaster) { - size = hash_get_shared_size(&info, hash_flags); - ShmemAllocator->index = (HASHHDR *) ShmemAlloc(size); + size_t size = hash_estimate_size(SHMEM_INDEX_SIZE, info.entrysize); + char *location = ShmemAlloc(size); + + allocator.next = location; + allocator.end = location + size; + info.alloc_arg = &allocator; + + info.hctl = NULL; + hash_flags |= HASH_ALLOC | HASH_FIXED_SIZE; + ShmemAllocator->index = (HASHHDR *) location; } else + { + info.hctl = ShmemAllocator->index; hash_flags |= HASH_ATTACH; - info.hctl = ShmemAllocator->index; + } ShmemIndex = hash_create("ShmemIndex", SHMEM_INDEX_SIZE, &info, hash_flags); Assert(ShmemIndex != NULL); } @@ -233,9 +255,17 @@ ShmemAllocNoError(Size size) static void * ShmemHashAlloc(Size size, void *alloc_arg) { - Size allocated_size; + shmem_hash_allocator *allocator = (shmem_hash_allocator *) alloc_arg; + void *result; - return ShmemAllocRaw(size, &allocated_size); + size = MAXALIGN(size); + + if (allocator->end - allocator->next < size) + return NULL; + result = allocator->next; + allocator->next += size; + + return result; } /* @@ -321,12 +351,14 @@ ShmemAddrIsValid(const void *addr) */ HTAB * ShmemInitHash(const char *name, /* table string name for shmem index */ - int64 nelems, /* size of the table */ + int64 nelems, /* size of the table */ HASHCTL *infoP, /* info about key and bucket size */ int hash_flags) /* info about infoP */ { bool found; + size_t size; void *location; + shmem_hash_allocator allocator; /* * Hash tables allocated in shared memory have a fixed directory; it can't @@ -341,20 +373,27 @@ ShmemInitHash(const char *name, /* table string name for shmem index */ infoP->alloc_arg = NULL; hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE; - /* look it up in the shmem index */ - location = ShmemInitStruct(name, - hash_get_shared_size(infoP, hash_flags), - &found); + size = hash_estimate_size(nelems, infoP->entrysize); + + /* look it up in the shmem index or allocate */ + location = ShmemInitStruct(name, size, &found); /* * if it already exists, attach to it rather than allocate and initialize * new space */ - if (found) + if (!found) + { + allocator.next = (char *) location; + allocator.end = (char *) location + size; + infoP->alloc_arg = &allocator; + } + else + { + /* Pass location of hashtable header to hash_create */ + infoP->hctl = (HASHHDR *) location; hash_flags |= HASH_ATTACH; - - /* Pass location of hashtable header to hash_create */ - infoP->hctl = (HASHHDR *) location; + } return hash_create(name, nelems, infoP, hash_flags); } diff --git a/src/backend/utils/hash/dynahash.c b/src/backend/utils/hash/dynahash.c index 16cf103b8e4..e414691ed03 100644 --- a/src/backend/utils/hash/dynahash.c +++ b/src/backend/utils/hash/dynahash.c @@ -195,6 +195,9 @@ struct HASHHDR int nelem_alloc; /* number of entries to allocate at once */ bool isfixed; /* if true, don't enlarge */ + /* Current directory. In shared tables, this doesn't change */ + HASHSEGMENT *dir; + #ifdef HASH_STATISTICS /* @@ -224,7 +227,7 @@ struct HTAB HashCompareFunc match; /* key comparison function */ HashCopyFunc keycopy; /* key copying function */ HashAllocFunc alloc; /* memory allocator */ - void *alloc_arg; /* opaque argument to pass to allocator function */ + void *alloc_arg; /* opaque argument to pass to alloc function */ MemoryContext hcxt; /* memory context if default allocator used */ char *tabname; /* table name (for error messages) */ bool isshared; /* true if table is in shared memory */ @@ -294,7 +297,7 @@ DynaHashAlloc(Size size, void *alloc_arg) MemoryContext cxt = (MemoryContext) alloc_arg; Assert(MemoryContextIsValid(cxt)); - return MemoryContextAllocExtended(cxt, size, MCXT_ALLOC_NO_OOM); + return MemoryContextAllocExtended(cxt, size, MCXT_ALLOC_NO_OOM); } @@ -374,6 +377,8 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) * hash_destroy very simple. The memory context is made a child of either * a context specified by the caller, or TopMemoryContext if nothing is * specified. + * + * Note that HASH_DIRSIZE and HASH_ALLOC had better be set as well. */ if (flags & HASH_SHARED_MEM) { @@ -485,21 +490,18 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) if (flags & HASH_SHARED_MEM) { - /* - * ctl structure and directory are preallocated for shared memory - * tables. Note that HASH_DIRSIZE and HASH_ALLOC had better be set as - * well. - */ - hashp->hctl = info->hctl; - hashp->dir = (HASHSEGMENT *) (((char *) info->hctl) + sizeof(HASHHDR)); hashp->hcxt = NULL; hashp->isshared = true; /* hash table already exists, we're just attaching to it */ if (flags & HASH_ATTACH) { + hctl = info->hctl; + + hashp->hctl = hctl; + hashp->dir = hctl->dir; + /* make local copies of some heavily-used values */ - hctl = hashp->hctl; hashp->keysize = hctl->keysize; return hashp; @@ -514,14 +516,20 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) hashp->isshared = false; } + /* + * Allocate the header structure. + * + * XXX: In case of a shared memory hash table, other procesess need the + * pointer to the header to re-find the hash table. There is currently no + * explicit way to pass it back from here, the caller relies on the fact + * that this is the first allocation made with the alloc function. That's + * a little ugly, but works for now. + */ + hashp->hctl = (HASHHDR *) hashp->alloc(sizeof(HASHHDR), hashp->alloc_arg); if (!hashp->hctl) - { - hashp->hctl = (HASHHDR *) hashp->alloc(sizeof(HASHHDR), hashp->alloc_arg); - if (!hashp->hctl) - ereport(ERROR, - (errcode(ERRCODE_OUT_OF_MEMORY), - errmsg("out of memory"))); - } + ereport(ERROR, + (errcode(ERRCODE_OUT_OF_MEMORY), + errmsg("out of memory"))); hashp->frozen = false; @@ -724,25 +732,17 @@ init_htab(HTAB *hashp, int64 nelem) nsegs = next_pow2_int(nsegs); /* - * Make sure directory is big enough. If pre-allocated directory is too - * small, choke (caller screwed up). + * Make sure directory is big enough. */ if (nsegs > hctl->dsize) - { - if (!(hashp->dir)) - hctl->dsize = nsegs; - else - return false; - } + hctl->dsize = nsegs; /* Allocate a directory */ - if (!(hashp->dir)) - { - hashp->dir = (HASHSEGMENT *) - hashp->alloc(hctl->dsize * sizeof(HASHSEGMENT), hashp->alloc_arg); - if (!hashp->dir) - return false; - } + hctl->dir = (HASHSEGMENT *) + hashp->alloc(hctl->dsize * sizeof(HASHSEGMENT), hashp->alloc_arg); + if (!hctl->dir) + return false; + hashp->dir = hctl->dir; /* Allocate initial segments */ for (segp = hashp->dir; hctl->nsegs < nsegs; hctl->nsegs++, segp++) @@ -831,19 +831,6 @@ hash_select_dirsize(int64 num_entries) return nDirEntries; } -/* - * Compute the required initial memory allocation for a shared-memory - * hashtable with the given parameters. We need space for the HASHHDR - * and for the (non expansible) directory. - */ -Size -hash_get_shared_size(HASHCTL *info, int flags) -{ - Assert(flags & HASH_DIRSIZE); - Assert(info->dsize == info->max_dsize); - return sizeof(HASHHDR) + info->dsize * sizeof(HASHSEGMENT); -} - /********************** DESTROY ROUTINES ************************/ @@ -1647,6 +1634,7 @@ dir_realloc(HTAB *hashp) { memcpy(p, old_p, old_dirsize); MemSet(((char *) p) + old_dirsize, 0, new_dirsize - old_dirsize); + hashp->hctl->dir = p; hashp->dir = p; hashp->hctl->dsize = new_dsize; diff --git a/src/include/utils/hsearch.h b/src/include/utils/hsearch.h index 03bc1c171cd..b60ae20acc7 100644 --- a/src/include/utils/hsearch.h +++ b/src/include/utils/hsearch.h @@ -80,10 +80,10 @@ typedef struct HASHCTL HashCopyFunc keycopy; /* key copying function */ /* Used if HASH_ALLOC flag is set: */ HashAllocFunc alloc; /* memory allocator */ - void *alloc_arg; /* opaque argument to pass to allocator function */ + void *alloc_arg; /* opaque argument to pass to alloc */ /* Used if HASH_CONTEXT flag is set: */ MemoryContext hcxt; /* memory context to use for allocations */ - /* Used if HASH_SHARED_MEM flag is set: */ + /* Used if HASH_ATTACH flag is set: */ HASHHDR *hctl; /* location of header in shared mem */ } HASHCTL; @@ -150,7 +150,6 @@ extern void hash_seq_term(HASH_SEQ_STATUS *status); extern void hash_freeze(HTAB *hashp); extern Size hash_estimate_size(int64 num_entries, Size entrysize); extern int64 hash_select_dirsize(int64 num_entries); -extern Size hash_get_shared_size(HASHCTL *info, int flags); extern void AtEOXact_HashTables(bool isCommit); extern void AtEOSubXact_HashTables(bool isCommit, int nestDepth); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index e3c1007abdf..d3e7adfafbd 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -4213,6 +4213,7 @@ shm_mq_result shm_toc shm_toc_entry shm_toc_estimator +shmem_hash_allocator shmem_request_hook_type shmem_startup_hook_type sig_atomic_t -- 2.47.3
From d68c213f33234d3aacdc0f9a8e083daf6e172d72 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas <[email protected]> Date: Mon, 30 Mar 2026 18:21:09 +0300 Subject: [PATCH v2 7/8] Remove HASH_DIRSIZE, always use the default algorithm to select it It's not very useful to specify a non-standard directory size. The HASH_DIRSIZE option was only used for shared memory hash tables, and those always used hash_select_dirsize() to choose the size, which in turn just uses the default algorithm anyway. That assumption was ingrained in hash_estimate_size(), too. Reviewed-by: Tomas Vondra <[email protected]> Discussion: https://www.postgresql.org/message-id/[email protected] --- src/backend/storage/ipc/shmem.c | 7 ++-- src/backend/utils/hash/dynahash.c | 56 +++++++------------------------ src/include/utils/hsearch.h | 6 +--- 3 files changed, 15 insertions(+), 54 deletions(-) diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c index c8171125871..2ee7d06e4d3 100644 --- a/src/backend/storage/ipc/shmem.c +++ b/src/backend/storage/ipc/shmem.c @@ -189,11 +189,9 @@ InitShmemAllocator(PGShmemHeader *seghdr) */ info.keysize = SHMEM_INDEX_KEYSIZE; info.entrysize = sizeof(ShmemIndexEnt); - info.dsize = info.max_dsize = hash_select_dirsize(SHMEM_INDEX_SIZE); info.alloc = ShmemHashAlloc; info.alloc_arg = NULL; - hash_flags = HASH_ELEM | HASH_STRINGS - | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE; + hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_FIXED_SIZE; if (!IsUnderPostmaster) { @@ -368,10 +366,9 @@ ShmemInitHash(const char *name, /* table string name for shmem index */ * * The shared memory allocator must be specified too. */ - infoP->dsize = infoP->max_dsize = hash_select_dirsize(nelems); infoP->alloc = ShmemHashAlloc; infoP->alloc_arg = NULL; - hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE; + hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_FIXED_SIZE; size = hash_estimate_size(nelems, infoP->entrysize); diff --git a/src/backend/utils/hash/dynahash.c b/src/backend/utils/hash/dynahash.c index e414691ed03..aedcbde20c4 100644 --- a/src/backend/utils/hash/dynahash.c +++ b/src/backend/utils/hash/dynahash.c @@ -115,10 +115,12 @@ * a whole lot of records per bucket or performance goes down. * * In a hash table allocated in shared memory, the directory cannot be - * expanded because it must stay at a fixed address. The directory size - * should be selected using hash_select_dirsize (and you'd better have - * a good idea of the maximum number of entries!). For non-shared hash - * tables, the initial directory size can be left at the default. + * expanded because it must stay at a fixed address. The directory size is + * chosen at creation based on the initial number of initial elements, so even + * though we support allocating more elements later, performance will suffer + * if the table grows much beyond the initial size. (Currently, shared memory + * hash tables are only created by ShmemInitHash() though, which doesn't + * support growing at all.) */ #define HASH_SEGSIZE 256 #define HASH_SEGSIZE_SHIFT 8 /* must be log2(HASH_SEGSIZE) */ @@ -378,7 +380,7 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) * a context specified by the caller, or TopMemoryContext if nothing is * specified. * - * Note that HASH_DIRSIZE and HASH_ALLOC had better be set as well. + * Note that HASH_ALLOC had better be set as well. */ if (flags & HASH_SHARED_MEM) { @@ -552,15 +554,6 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags) hctl->num_partitions = info->num_partitions; } - /* - * SHM hash tables have fixed directory size passed by the caller. - */ - if (flags & HASH_DIRSIZE) - { - hctl->max_dsize = info->max_dsize; - hctl->dsize = info->dsize; - } - /* remember the entry sizes, too */ hctl->keysize = info->keysize; hctl->entrysize = info->entrysize; @@ -734,8 +727,11 @@ init_htab(HTAB *hashp, int64 nelem) /* * Make sure directory is big enough. */ - if (nsegs > hctl->dsize) - hctl->dsize = nsegs; + hctl->dsize = nsegs; + + /* SHM hash tables have fixed directory. */ + if (hashp->isshared) + hctl->max_dsize = hctl->dsize; /* Allocate a directory */ hctl->dir = (HASHSEGMENT *) @@ -803,34 +799,6 @@ hash_estimate_size(int64 num_entries, Size entrysize) return size; } -/* - * Select an appropriate directory size for a hashtable with the given - * maximum number of entries. - * This is only needed for hashtables in shared memory, whose directories - * cannot be expanded dynamically. - * NB: assumes that all hash structure parameters have default values! - * - * XXX this had better agree with the behavior of init_htab()... - */ -int64 -hash_select_dirsize(int64 num_entries) -{ - int64 nBuckets, - nSegments, - nDirEntries; - - /* estimate number of buckets wanted */ - nBuckets = next_pow2_int64(num_entries); - /* # of segments needed for nBuckets */ - nSegments = next_pow2_int64((nBuckets - 1) / HASH_SEGSIZE + 1); - /* directory entries */ - nDirEntries = DEF_DIRSIZE; - while (nDirEntries < nSegments) - nDirEntries <<= 1; /* dir_alloc doubles dsize at each call */ - - return nDirEntries; -} - /********************** DESTROY ROUTINES ************************/ diff --git a/src/include/utils/hsearch.h b/src/include/utils/hsearch.h index b60ae20acc7..115e97642f5 100644 --- a/src/include/utils/hsearch.h +++ b/src/include/utils/hsearch.h @@ -66,9 +66,6 @@ typedef struct HASHCTL { /* Used if HASH_PARTITION flag is set: */ int64 num_partitions; /* # partitions (must be power of 2) */ - /* Used if HASH_DIRSIZE flag is set: */ - int64 dsize; /* (initial) directory size */ - int64 max_dsize; /* limit to dsize if dir size is limited */ /* Used if HASH_ELEM flag is set (which is now required): */ Size keysize; /* hash key length in bytes */ Size entrysize; /* total user element size in bytes */ @@ -90,7 +87,7 @@ typedef struct HASHCTL /* Flag bits for hash_create; most indicate which parameters are supplied */ #define HASH_PARTITION 0x0001 /* Hashtable is used w/partitioned locking */ /* 0x0002 is unused */ -#define HASH_DIRSIZE 0x0004 /* Set directory size (initial and max) */ +/* 0x0004 is unused */ #define HASH_ELEM 0x0008 /* Set keysize and entrysize (now required!) */ #define HASH_STRINGS 0x0010 /* Select support functions for string keys */ #define HASH_BLOBS 0x0020 /* Select support functions for binary keys */ @@ -149,7 +146,6 @@ extern void *hash_seq_search(HASH_SEQ_STATUS *status); extern void hash_seq_term(HASH_SEQ_STATUS *status); extern void hash_freeze(HTAB *hashp); extern Size hash_estimate_size(int64 num_entries, Size entrysize); -extern int64 hash_select_dirsize(int64 num_entries); extern void AtEOXact_HashTables(bool isCommit); extern void AtEOSubXact_HashTables(bool isCommit, int nestDepth); -- 2.47.3
From b6678b4fb04670d594880855a8bd07f0acb1b7a9 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas <[email protected]> Date: Mon, 30 Mar 2026 18:21:11 +0300 Subject: [PATCH v2 8/8] Make ShmemIndex visible in the pg_shmem_allocations view Reviewed-by: Tomas Vondra <[email protected]> Discussion: https://www.postgresql.org/message-id/[email protected] --- src/backend/storage/ipc/shmem.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c index 2ee7d06e4d3..08dd7ab7a1c 100644 --- a/src/backend/storage/ipc/shmem.c +++ b/src/backend/storage/ipc/shmem.c @@ -137,6 +137,7 @@ InitShmemAllocator(PGShmemHeader *seghdr) HASHCTL info; int hash_flags; shmem_hash_allocator allocator; + Size size = 0; #ifndef EXEC_BACKEND Assert(!IsUnderPostmaster); @@ -195,16 +196,14 @@ InitShmemAllocator(PGShmemHeader *seghdr) if (!IsUnderPostmaster) { - size_t size = hash_estimate_size(SHMEM_INDEX_SIZE, info.entrysize); - char *location = ShmemAlloc(size); + size = hash_estimate_size(SHMEM_INDEX_SIZE, info.entrysize); + ShmemAllocator->index = (HASHHDR *) ShmemAlloc(size); - allocator.next = location; - allocator.end = location + size; + allocator.next = (char *) ShmemAllocator->index; + allocator.end = (char *) ShmemAllocator->index + size; info.alloc_arg = &allocator; - info.hctl = NULL; hash_flags |= HASH_ALLOC | HASH_FIXED_SIZE; - ShmemAllocator->index = (HASHHDR *) location; } else { @@ -213,6 +212,23 @@ InitShmemAllocator(PGShmemHeader *seghdr) } ShmemIndex = hash_create("ShmemIndex", SHMEM_INDEX_SIZE, &info, hash_flags); Assert(ShmemIndex != NULL); + + /* + * Add an entry for the ShmemIndex itself into ShmemIndex, so that it's + * visible in the pg_shmem_allocations view + */ + if (!IsUnderPostmaster) + { + ShmemIndexEnt *result; + bool found; + + result = (ShmemIndexEnt *) + hash_search(ShmemIndex, "ShmemIndex", HASH_ENTER, &found); + Assert(!found); + result->size = size; + result->allocated_size = size; + result->location = ShmemAllocator->index; + } } /* -- 2.47.3
