I'm starting a new thread for this topic that we discussed on the
"Better shared data structure management and resizable shared data
structures" thread:
On 25/03/2026 20:37, Robert Haas wrote:
On Sat, Mar 21, 2026 at 8:14 PM Heikki Linnakangas <[email protected]> wrote:
I wonder if we should set aside a small amount of memory, like 10-20 kB,
in the fixed shmem segment for small structs like that. Currently, such
allocations will usually succeed because we leave some wiggle room, but
the can also be consumed by other things like locks. But we could
reserve a small amount of memory specifically for small allocations in
extensions like this.
Yeah, I don't really understand why we let the lock table use up that
space. I mean, I think it would be useful to have a way to let the
lock table expand without a server restart, and I also suspect that we
could come up with a less-silly data structure than the PROCLOCK hash,
but also if the only thing keeping you from running out of lock space
is the wiggle room, maybe you just need to bump up
max_locks_per_transaction. Like, you could easily burn through the
wiggle room, get an error anyway, and then later find that you also
now can't load certain extensions without a server restart.
Attached patch set tightens up all shared memory hash tables so that
they no longer use the "wiggle room". They are now truly fixed size. You
specify the number of elements in ShmemInitHash(), and that's it.
This also addresses the accounting issue we currently have with hash
tables, that the size reported in pg_shmem_allocations only shows the
size of fixed header and the directory, not the actual hash buckets.
They were previously all lumped together in the "<anonymous>" section.
These patches fix that.
There was an earlier attempt at that at last year [1], but it got
reverted. I hope my approach is less invasive: instead of changing
dynahash.c to use a single allocation directly, I'm providing it an
"alloc" callback that carves out the different parts it needs from the
single contiguous shmem area, which in turn is allocated with
ShmemInitStruct().
[1]
https://www.postgresql.org/message-id/CAH2L28vHzRankszhqz7deXURxKncxfirnuW68zD7%2BhVAqaS5GQ%40mail.gmail.com
- Heikki
From e76e047b47b40c8f3737f69976f481f4050e35da Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 27 Mar 2026 18:36:19 +0200
Subject: [PATCH v1 1/8] Use ShmemInitStruct to allocate shmem for semaphores
This makes them visible visible in pg_shmem_allocations
---
src/backend/port/posix_sema.c | 4 +++-
src/backend/port/sysv_sema.c | 5 ++++-
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/src/backend/port/posix_sema.c b/src/backend/port/posix_sema.c
index e368e5ee7ed..40205b7d400 100644
--- a/src/backend/port/posix_sema.c
+++ b/src/backend/port/posix_sema.c
@@ -196,6 +196,7 @@ void
PGReserveSemaphores(int maxSemas)
{
struct stat statbuf;
+ bool found;
/*
* We use the data directory's inode number to seed the search for free
@@ -216,7 +217,8 @@ PGReserveSemaphores(int maxSemas)
#else
sharedSemas = (PGSemaphore)
- ShmemAlloc(PGSemaphoreShmemSize(maxSemas));
+ ShmemInitStruct("Semaphores", PGSemaphoreShmemSize(maxSemas), &found);
+ Assert(!found);
#endif
numSems = 0;
diff --git a/src/backend/port/sysv_sema.c b/src/backend/port/sysv_sema.c
index 86c4d359ef7..4b2bf84072f 100644
--- a/src/backend/port/sysv_sema.c
+++ b/src/backend/port/sysv_sema.c
@@ -330,6 +330,7 @@ void
PGReserveSemaphores(int maxSemas)
{
struct stat statbuf;
+ bool found;
/*
* We use the data directory's inode number to seed the search for free
@@ -344,7 +345,9 @@ PGReserveSemaphores(int maxSemas)
DataDir)));
sharedSemas = (PGSemaphore)
- ShmemAlloc(PGSemaphoreShmemSize(maxSemas));
+ ShmemInitStruct("Semaphores", PGSemaphoreShmemSize(maxSemas), &found);
+ Assert(!found);
+
numSharedSemas = 0;
maxSharedSemas = maxSemas;
--
2.47.3
From e9f85dad915ce250bf69adf11a95d08e789b3f37 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 27 Mar 2026 22:07:11 +0200
Subject: [PATCH v1 2/8] Remove HASH_SEGMENT option
It's been unused forever.
This isn't needed by the rest of the patches, just something that
caught my eye.
Aleksander Alekseev proposed this a long time ago [0], but Tom Lane
was worried about third-party extensions using it. I believe that to
not be an issue, I tried grepping through all extensions found on
github, and didn't find any references.
[0] https://www.postgresql.org/message-id/20160418180711.55ac82c0@fujitsu
---
src/backend/utils/hash/dynahash.c | 65 +++++++++++--------------------
src/include/utils/hsearch.h | 4 +-
2 files changed, 23 insertions(+), 46 deletions(-)
diff --git a/src/backend/utils/hash/dynahash.c b/src/backend/utils/hash/dynahash.c
index 11f5778eba1..af3f7142afb 100644
--- a/src/backend/utils/hash/dynahash.c
+++ b/src/backend/utils/hash/dynahash.c
@@ -108,11 +108,11 @@
/*
* Constants
*
- * A hash table has a top-level "directory", each of whose entries points
- * to a "segment" of ssize bucket headers. The maximum number of hash
- * buckets is thus dsize * ssize (but dsize may be expansible). Of course,
- * the number of records in the table can be larger, but we don't want a
- * whole lot of records per bucket or performance goes down.
+ * A hash table has a top-level "directory", each of whose entries points to a
+ * "segment" of HASH_SEGSIZE bucket headers. The maximum number of hash
+ * buckets is thus dsize * HASH_SEGSIZE (but dsize may be expansible). Of
+ * course, the number of records in the table can be larger, but we don't want
+ * a whole lot of records per bucket or performance goes down.
*
* In a hash table allocated in shared memory, the directory cannot be
* expanded because it must stay at a fixed address. The directory size
@@ -120,8 +120,8 @@
* a good idea of the maximum number of entries!). For non-shared hash
* tables, the initial directory size can be left at the default.
*/
-#define DEF_SEGSIZE 256
-#define DEF_SEGSIZE_SHIFT 8 /* must be log2(DEF_SEGSIZE) */
+#define HASH_SEGSIZE 256
+#define HASH_SEGSIZE_SHIFT 8 /* must be log2(HASH_SEGSIZE) */
#define DEF_DIRSIZE 256
/* Number of freelists to be used for a partitioned hash table. */
@@ -192,8 +192,6 @@ struct HASHHDR
Size entrysize; /* total user element size in bytes */
int64 num_partitions; /* # partitions (must be power of 2), or 0 */
int64 max_dsize; /* 'dsize' limit if directory is fixed size */
- int64 ssize; /* segment size --- must be power of 2 */
- int sshift; /* segment shift = log2(ssize) */
int nelem_alloc; /* number of entries to allocate at once */
bool isfixed; /* if true, don't enlarge */
@@ -235,8 +233,6 @@ struct HTAB
/* We keep local copies of these fixed values to reduce contention */
Size keysize; /* hash key length in bytes */
- int64 ssize; /* segment size --- must be power of 2 */
- int sshift; /* segment shift = log2(ssize) */
/*
* In a USE_VALGRIND build, non-shared hashtables keep an slist chain of
@@ -499,8 +495,6 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags)
/* make local copies of some heavily-used values */
hctl = hashp->hctl;
hashp->keysize = hctl->keysize;
- hashp->ssize = hctl->ssize;
- hashp->sshift = hctl->sshift;
return hashp;
}
@@ -544,14 +538,6 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags)
hctl->num_partitions = info->num_partitions;
}
- if (flags & HASH_SEGMENT)
- {
- hctl->ssize = info->ssize;
- hctl->sshift = my_log2(info->ssize);
- /* ssize had better be a power of 2 */
- Assert(hctl->ssize == (1L << hctl->sshift));
- }
-
/*
* SHM hash tables have fixed directory size passed by the caller.
*/
@@ -567,8 +553,6 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags)
/* make local copies of heavily-used constant fields */
hashp->keysize = hctl->keysize;
- hashp->ssize = hctl->ssize;
- hashp->sshift = hctl->sshift;
/* Build the hash directory structure */
if (!init_htab(hashp, nelem))
@@ -649,9 +633,6 @@ hdefault(HTAB *hashp)
/* table has no fixed maximum size */
hctl->max_dsize = NO_MAX_DSIZE;
- hctl->ssize = DEF_SEGSIZE;
- hctl->sshift = DEF_SEGSIZE_SHIFT;
-
hctl->isfixed = false; /* can be enlarged */
#ifdef HASH_STATISTICS
@@ -733,7 +714,7 @@ init_htab(HTAB *hashp, int64 nelem)
/*
* Figure number of directory segments needed, round up to a power of 2
*/
- nsegs = (nbuckets - 1) / hctl->ssize + 1;
+ nsegs = (nbuckets - 1) / HASH_SEGSIZE + 1;
nsegs = next_pow2_int(nsegs);
/*
@@ -793,7 +774,7 @@ hash_estimate_size(int64 num_entries, Size entrysize)
/* estimate number of buckets wanted */
nBuckets = next_pow2_int64(num_entries);
/* # of segments needed for nBuckets */
- nSegments = next_pow2_int64((nBuckets - 1) / DEF_SEGSIZE + 1);
+ nSegments = next_pow2_int64((nBuckets - 1) / HASH_SEGSIZE + 1);
/* directory entries */
nDirEntries = DEF_DIRSIZE;
while (nDirEntries < nSegments)
@@ -805,7 +786,7 @@ hash_estimate_size(int64 num_entries, Size entrysize)
size = add_size(size, mul_size(nDirEntries, sizeof(HASHSEGMENT)));
/* segments */
size = add_size(size, mul_size(nSegments,
- MAXALIGN(DEF_SEGSIZE * sizeof(HASHBUCKET))));
+ MAXALIGN(HASH_SEGSIZE * sizeof(HASHBUCKET))));
/* elements --- allocated in groups of choose_nelem_alloc() entries */
elementAllocCnt = choose_nelem_alloc(entrysize);
nElementAllocs = (num_entries - 1) / elementAllocCnt + 1;
@@ -836,7 +817,7 @@ hash_select_dirsize(int64 num_entries)
/* estimate number of buckets wanted */
nBuckets = next_pow2_int64(num_entries);
/* # of segments needed for nBuckets */
- nSegments = next_pow2_int64((nBuckets - 1) / DEF_SEGSIZE + 1);
+ nSegments = next_pow2_int64((nBuckets - 1) / HASH_SEGSIZE + 1);
/* directory entries */
nDirEntries = DEF_DIRSIZE;
while (nDirEntries < nSegments)
@@ -1417,7 +1398,6 @@ hash_seq_search(HASH_SEQ_STATUS *status)
HTAB *hashp;
HASHHDR *hctl;
uint32 max_bucket;
- int64 ssize;
int64 segment_num;
int64 segment_ndx;
HASHSEGMENT segp;
@@ -1457,7 +1437,6 @@ hash_seq_search(HASH_SEQ_STATUS *status)
curBucket = status->curBucket;
hashp = status->hashp;
hctl = hashp->hctl;
- ssize = hashp->ssize;
max_bucket = hctl->max_bucket;
if (curBucket > max_bucket)
@@ -1469,8 +1448,8 @@ hash_seq_search(HASH_SEQ_STATUS *status)
/*
* first find the right segment in the table directory.
*/
- segment_num = curBucket >> hashp->sshift;
- segment_ndx = MOD(curBucket, ssize);
+ segment_num = curBucket >> HASH_SEGSIZE_SHIFT;
+ segment_ndx = MOD(curBucket, HASH_SEGSIZE);
segp = hashp->dir[segment_num];
@@ -1489,7 +1468,7 @@ hash_seq_search(HASH_SEQ_STATUS *status)
hash_seq_term(status);
return NULL; /* search is done */
}
- if (++segment_ndx >= ssize)
+ if (++segment_ndx >= HASH_SEGSIZE)
{
segment_num++;
segment_ndx = 0;
@@ -1566,8 +1545,8 @@ expand_table(HTAB *hashp)
#endif
new_bucket = hctl->max_bucket + 1;
- new_segnum = new_bucket >> hashp->sshift;
- new_segndx = MOD(new_bucket, hashp->ssize);
+ new_segnum = new_bucket >> HASH_SEGSIZE_SHIFT;
+ new_segndx = MOD(new_bucket, HASH_SEGSIZE);
if (new_segnum >= hctl->nsegs)
{
@@ -1606,8 +1585,8 @@ expand_table(HTAB *hashp)
* split at this point. With a different way of reducing the hash value,
* that might not be true!
*/
- old_segnum = old_bucket >> hashp->sshift;
- old_segndx = MOD(old_bucket, hashp->ssize);
+ old_segnum = old_bucket >> HASH_SEGSIZE_SHIFT;
+ old_segndx = MOD(old_bucket, HASH_SEGSIZE);
old_seg = hashp->dir[old_segnum];
new_seg = hashp->dir[new_segnum];
@@ -1684,12 +1663,12 @@ seg_alloc(HTAB *hashp)
HASHSEGMENT segp;
CurrentDynaHashCxt = hashp->hcxt;
- segp = (HASHSEGMENT) hashp->alloc(sizeof(HASHBUCKET) * hashp->ssize);
+ segp = (HASHSEGMENT) hashp->alloc(sizeof(HASHBUCKET) * HASH_SEGSIZE);
if (!segp)
return NULL;
- MemSet(segp, 0, sizeof(HASHBUCKET) * hashp->ssize);
+ MemSet(segp, 0, sizeof(HASHBUCKET) * HASH_SEGSIZE);
return segp;
}
@@ -1786,8 +1765,8 @@ hash_initial_lookup(HTAB *hashp, uint32 hashvalue, HASHBUCKET **bucketptr)
bucket = calc_bucket(hctl, hashvalue);
- segment_num = bucket >> hashp->sshift;
- segment_ndx = MOD(bucket, hashp->ssize);
+ segment_num = bucket >> HASH_SEGSIZE_SHIFT;
+ segment_ndx = MOD(bucket, HASH_SEGSIZE);
segp = hashp->dir[segment_num];
diff --git a/src/include/utils/hsearch.h b/src/include/utils/hsearch.h
index cdc530fb732..0721849c036 100644
--- a/src/include/utils/hsearch.h
+++ b/src/include/utils/hsearch.h
@@ -66,8 +66,6 @@ typedef struct HASHCTL
{
/* Used if HASH_PARTITION flag is set: */
int64 num_partitions; /* # partitions (must be power of 2) */
- /* Used if HASH_SEGMENT flag is set: */
- int64 ssize; /* segment size */
/* Used if HASH_DIRSIZE flag is set: */
int64 dsize; /* (initial) directory size */
int64 max_dsize; /* limit to dsize if dir size is limited */
@@ -90,7 +88,7 @@ typedef struct HASHCTL
/* Flag bits for hash_create; most indicate which parameters are supplied */
#define HASH_PARTITION 0x0001 /* Hashtable is used w/partitioned locking */
-#define HASH_SEGMENT 0x0002 /* Set segment size */
+/* 0x0002 is unused */
#define HASH_DIRSIZE 0x0004 /* Set directory size (initial and max) */
#define HASH_ELEM 0x0008 /* Set keysize and entrysize (now required!) */
#define HASH_STRINGS 0x0010 /* Select support functions for string keys */
--
2.47.3
From 1dbaf4694775e82859b3852730a7f0bd3f7b9150 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 27 Mar 2026 14:27:41 +0200
Subject: [PATCH v1 3/8] Change the signature of dynahash's alloc function
Instead of passing the current memory context to the alloc function
via a shared variable, pass it directly as an argument.
---
src/backend/storage/ipc/shmem.c | 15 +++++++++++--
src/backend/utils/hash/dynahash.c | 35 +++++++++++++++++--------------
src/include/utils/hsearch.h | 3 ++-
3 files changed, 34 insertions(+), 19 deletions(-)
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 3cc38450949..a6fd4c30059 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -100,6 +100,7 @@ typedef struct ShmemAllocatorData
#define ShmemIndexLock (&ShmemAllocator->index_lock)
+static void *ShmemHashAlloc(Size size, void *alloc_arg);
static void *ShmemAllocRaw(Size size, Size *allocated_size);
/* shared memory global variables */
@@ -184,7 +185,8 @@ InitShmemAllocator(PGShmemHeader *seghdr)
info.keysize = SHMEM_INDEX_KEYSIZE;
info.entrysize = sizeof(ShmemIndexEnt);
info.dsize = info.max_dsize = hash_select_dirsize(SHMEM_INDEX_SIZE);
- info.alloc = ShmemAllocNoError;
+ info.alloc = ShmemHashAlloc;
+ info.alloc_arg = NULL;
hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
if (!IsUnderPostmaster)
{
@@ -233,6 +235,14 @@ ShmemAllocNoError(Size size)
return ShmemAllocRaw(size, &allocated_size);
}
+static void *
+ShmemHashAlloc(Size size, void *alloc_arg)
+{
+ Size allocated_size;
+
+ return ShmemAllocRaw(size, &allocated_size);
+}
+
/*
* ShmemAllocRaw -- allocate align chunk and return allocated size
*
@@ -339,7 +349,8 @@ ShmemInitHash(const char *name, /* table string name for shmem index */
* The shared memory allocator must be specified too.
*/
infoP->dsize = infoP->max_dsize = hash_select_dirsize(max_size);
- infoP->alloc = ShmemAllocNoError;
+ infoP->alloc = ShmemHashAlloc;
+ infoP->alloc_arg = NULL;
hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
/* look it up in the shmem index */
diff --git a/src/backend/utils/hash/dynahash.c b/src/backend/utils/hash/dynahash.c
index af3f7142afb..d3dd16a4300 100644
--- a/src/backend/utils/hash/dynahash.c
+++ b/src/backend/utils/hash/dynahash.c
@@ -224,6 +224,7 @@ struct HTAB
HashCompareFunc match; /* key comparison function */
HashCopyFunc keycopy; /* key copying function */
HashAllocFunc alloc; /* memory allocator */
+ void *alloc_arg; /* opaque argument to pass to allocator function */
MemoryContext hcxt; /* memory context if default allocator used */
char *tabname; /* table name (for error messages) */
bool isshared; /* true if table is in shared memory */
@@ -264,7 +265,7 @@ struct HTAB
/*
* Private function prototypes
*/
-static void *DynaHashAlloc(Size size);
+static void *DynaHashAlloc(Size size, void *alloc_arg);
static HASHSEGMENT seg_alloc(HTAB *hashp);
static bool element_alloc(HTAB *hashp, int nelem, int freelist_idx);
static bool dir_realloc(HTAB *hashp);
@@ -287,14 +288,13 @@ static bool has_seq_scans(HTAB *hashp);
/*
* memory allocation support
*/
-static MemoryContext CurrentDynaHashCxt = NULL;
-
static void *
-DynaHashAlloc(Size size)
+DynaHashAlloc(Size size, void *alloc_arg)
{
- Assert(MemoryContextIsValid(CurrentDynaHashCxt));
- return MemoryContextAllocExtended(CurrentDynaHashCxt, size,
- MCXT_ALLOC_NO_OOM);
+ MemoryContext cxt = (MemoryContext) alloc_arg;
+
+ Assert(MemoryContextIsValid(cxt));
+ return MemoryContextAllocExtended(cxt, size, MCXT_ALLOC_NO_OOM);
}
@@ -355,6 +355,7 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags)
{
HTAB *hashp;
HASHHDR *hctl;
+ MemoryContext CurrentDynaHashCxt;
/*
* Hash tables now allocate space for key and data, but you have to say
@@ -473,9 +474,15 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags)
/* And select the entry allocation function, too. */
if (flags & HASH_ALLOC)
+ {
hashp->alloc = info->alloc;
+ hashp->alloc_arg = info->alloc_arg;
+ }
else
+ {
hashp->alloc = DynaHashAlloc;
+ hashp->alloc_arg = CurrentDynaHashCxt;
+ }
if (flags & HASH_SHARED_MEM)
{
@@ -510,7 +517,7 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags)
if (!hashp->hctl)
{
- hashp->hctl = (HASHHDR *) hashp->alloc(sizeof(HASHHDR));
+ hashp->hctl = (HASHHDR *) hashp->alloc(sizeof(HASHHDR), hashp->alloc_arg);
if (!hashp->hctl)
ereport(ERROR,
(errcode(ERRCODE_OUT_OF_MEMORY),
@@ -732,9 +739,8 @@ init_htab(HTAB *hashp, int64 nelem)
/* Allocate a directory */
if (!(hashp->dir))
{
- CurrentDynaHashCxt = hashp->hcxt;
hashp->dir = (HASHSEGMENT *)
- hashp->alloc(hctl->dsize * sizeof(HASHSEGMENT));
+ hashp->alloc(hctl->dsize * sizeof(HASHSEGMENT), hashp->alloc_arg);
if (!hashp->dir)
return false;
}
@@ -1636,8 +1642,7 @@ dir_realloc(HTAB *hashp)
new_dirsize = new_dsize * sizeof(HASHSEGMENT);
old_p = hashp->dir;
- CurrentDynaHashCxt = hashp->hcxt;
- p = (HASHSEGMENT *) hashp->alloc((Size) new_dirsize);
+ p = (HASHSEGMENT *) hashp->alloc((Size) new_dirsize, hashp->alloc_arg);
if (p != NULL)
{
@@ -1662,8 +1667,7 @@ seg_alloc(HTAB *hashp)
{
HASHSEGMENT segp;
- CurrentDynaHashCxt = hashp->hcxt;
- segp = (HASHSEGMENT) hashp->alloc(sizeof(HASHBUCKET) * HASH_SEGSIZE);
+ segp = (HASHSEGMENT) hashp->alloc(sizeof(HASHBUCKET) * HASH_SEGSIZE, hashp->alloc_arg);
if (!segp)
return NULL;
@@ -1703,8 +1707,7 @@ element_alloc(HTAB *hashp, int nelem, int freelist_idx)
#endif
/* Allocate the memory. */
- CurrentDynaHashCxt = hashp->hcxt;
- allocedBlock = hashp->alloc(requestSize);
+ allocedBlock = hashp->alloc(requestSize, hashp->alloc_arg);
if (!allocedBlock)
return false;
diff --git a/src/include/utils/hsearch.h b/src/include/utils/hsearch.h
index 0721849c036..03bc1c171cd 100644
--- a/src/include/utils/hsearch.h
+++ b/src/include/utils/hsearch.h
@@ -41,7 +41,7 @@ typedef void *(*HashCopyFunc) (void *dest, const void *src, Size keysize);
* Note: there is no free function API; can't destroy a hashtable unless you
* use the default allocator.
*/
-typedef void *(*HashAllocFunc) (Size request);
+typedef void *(*HashAllocFunc) (Size request, void *alloc_arg);
/*
* HASHELEMENT is the private part of a hashtable entry. The caller's data
@@ -80,6 +80,7 @@ typedef struct HASHCTL
HashCopyFunc keycopy; /* key copying function */
/* Used if HASH_ALLOC flag is set: */
HashAllocFunc alloc; /* memory allocator */
+ void *alloc_arg; /* opaque argument to pass to allocator function */
/* Used if HASH_CONTEXT flag is set: */
MemoryContext hcxt; /* memory context to use for allocations */
/* Used if HASH_SHARED_MEM flag is set: */
--
2.47.3
From f212dc8e4031b3227f9e7fef2635d3102802f0d6 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 27 Mar 2026 23:53:04 +0200
Subject: [PATCH v1 4/8] Merge init and max size options on shmem hash tables
Replace the separate init and max size options with a single size
option. We didn't make much use of the feature, most callers already
used the same size for both. Allowing the table to grow much beyond
the initial size is bad for performance anyway as we don't support
resizing the hash table's dictionary part.
---
.../pg_stat_statements/pg_stat_statements.c | 2 +-
src/backend/storage/buffer/buf_table.c | 2 +-
src/backend/storage/ipc/shmem.c | 16 ++++----------
src/backend/storage/lmgr/lock.c | 7 +-----
src/backend/storage/lmgr/predicate.c | 3 ---
src/backend/utils/activity/wait_event.c | 22 ++++++++-----------
src/include/storage/shmem.h | 2 +-
7 files changed, 17 insertions(+), 37 deletions(-)
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index 7975476b890..5494d41dca1 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -561,7 +561,7 @@ pgss_shmem_startup(void)
info.keysize = sizeof(pgssHashKey);
info.entrysize = sizeof(pgssEntry);
pgss_hash = ShmemInitHash("pg_stat_statements hash",
- pgss_max, pgss_max,
+ pgss_max,
&info,
HASH_ELEM | HASH_BLOBS);
diff --git a/src/backend/storage/buffer/buf_table.c b/src/backend/storage/buffer/buf_table.c
index 23d85fd32e2..d04ef74b850 100644
--- a/src/backend/storage/buffer/buf_table.c
+++ b/src/backend/storage/buffer/buf_table.c
@@ -60,7 +60,7 @@ InitBufTable(int size)
info.num_partitions = NUM_BUFFER_PARTITIONS;
SharedBufHash = ShmemInitHash("Shared Buffer Lookup Table",
- size, size,
+ size,
&info,
HASH_ELEM | HASH_BLOBS | HASH_PARTITION | HASH_FIXED_SIZE);
}
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index a6fd4c30059..fd2c7c4590a 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -313,14 +313,7 @@ ShmemAddrIsValid(const void *addr)
* table at once. (In practice, all creations are done in the postmaster
* process; child processes should always be attaching to existing tables.)
*
- * max_size is the estimated maximum number of hashtable entries. This is
- * not a hard limit, but the access efficiency will degrade if it is
- * exceeded substantially (since it's used to compute directory size and
- * the hash table buckets will get overfull).
- *
- * init_size is the number of hashtable entries to preallocate. For a table
- * whose maximum size is certain, this should be equal to max_size; that
- * ensures that no run-time out-of-shared-memory failures can occur.
+ * nelems is the maximum number of hashtable entries.
*
* *infoP and hash_flags must specify at least the entry sizes and key
* comparison semantics (see hash_create()). Flag bits and values specific
@@ -333,8 +326,7 @@ ShmemAddrIsValid(const void *addr)
*/
HTAB *
ShmemInitHash(const char *name, /* table string name for shmem index */
- int64 init_size, /* initial table size */
- int64 max_size, /* max size of the table */
+ int64 nelems, /* size of the table */
HASHCTL *infoP, /* info about key and bucket size */
int hash_flags) /* info about infoP */
{
@@ -348,7 +340,7 @@ ShmemInitHash(const char *name, /* table string name for shmem index */
*
* The shared memory allocator must be specified too.
*/
- infoP->dsize = infoP->max_dsize = hash_select_dirsize(max_size);
+ infoP->dsize = infoP->max_dsize = hash_select_dirsize(nelems);
infoP->alloc = ShmemHashAlloc;
infoP->alloc_arg = NULL;
hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
@@ -368,7 +360,7 @@ ShmemInitHash(const char *name, /* table string name for shmem index */
/* Pass location of hashtable header to hash_create */
infoP->hctl = (HASHHDR *) location;
- return hash_create(name, init_size, infoP, hash_flags);
+ return hash_create(name, nelems, infoP, hash_flags);
}
/*
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 234643e4dd7..ae524bb02d6 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -445,8 +445,7 @@ void
LockManagerShmemInit(void)
{
HASHCTL info;
- int64 init_table_size,
- max_table_size;
+ int64 max_table_size;
bool found;
/*
@@ -454,7 +453,6 @@ LockManagerShmemInit(void)
* calculations must agree with LockManagerShmemSize!
*/
max_table_size = NLOCKENTS();
- init_table_size = max_table_size / 2;
/*
* Allocate hash table for LOCK structs. This stores per-locked-object
@@ -465,14 +463,12 @@ LockManagerShmemInit(void)
info.num_partitions = NUM_LOCK_PARTITIONS;
LockMethodLockHash = ShmemInitHash("LOCK hash",
- init_table_size,
max_table_size,
&info,
HASH_ELEM | HASH_BLOBS | HASH_PARTITION);
/* Assume an average of 2 holders per lock */
max_table_size *= 2;
- init_table_size *= 2;
/*
* Allocate hash table for PROCLOCK structs. This stores
@@ -484,7 +480,6 @@ LockManagerShmemInit(void)
info.num_partitions = NUM_LOCK_PARTITIONS;
LockMethodProcLockHash = ShmemInitHash("PROCLOCK hash",
- init_table_size,
max_table_size,
&info,
HASH_ELEM | HASH_FUNCTION | HASH_PARTITION);
diff --git a/src/backend/storage/lmgr/predicate.c b/src/backend/storage/lmgr/predicate.c
index ae0e96aee5f..4c559b2d3ee 100644
--- a/src/backend/storage/lmgr/predicate.c
+++ b/src/backend/storage/lmgr/predicate.c
@@ -1182,7 +1182,6 @@ PredicateLockShmemInit(void)
info.num_partitions = NUM_PREDICATELOCK_PARTITIONS;
PredicateLockTargetHash = ShmemInitHash("PREDICATELOCKTARGET hash",
- max_predicate_lock_targets,
max_predicate_lock_targets,
&info,
HASH_ELEM | HASH_BLOBS |
@@ -1218,7 +1217,6 @@ PredicateLockShmemInit(void)
max_predicate_locks = max_predicate_lock_targets * 2;
PredicateLockHash = ShmemInitHash("PREDICATELOCK hash",
- max_predicate_locks,
max_predicate_locks,
&info,
HASH_ELEM | HASH_FUNCTION |
@@ -1297,7 +1295,6 @@ PredicateLockShmemInit(void)
info.entrysize = sizeof(SERIALIZABLEXID);
SerializableXidHash = ShmemInitHash("SERIALIZABLEXID hash",
- max_serializable_xacts,
max_serializable_xacts,
&info,
HASH_ELEM | HASH_BLOBS |
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index e5a2289f0b0..2b76967776c 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -56,16 +56,14 @@ uint32 *my_wait_event_info = &local_my_wait_event_info;
* For simplicity, we use the same ID counter across types of custom events.
* We could end that anytime the need arises.
*
- * The size of the hash table is based on the assumption that
- * WAIT_EVENT_CUSTOM_HASH_INIT_SIZE is enough for most cases, and it seems
- * unlikely that the number of entries will reach
- * WAIT_EVENT_CUSTOM_HASH_MAX_SIZE.
+ * The size of the hash table is based on the assumption that usually only a
+ * handful of entries are needed, but since it's small in absolute terms
+ * anyway, we leave a generous amount of headroom.
*/
static HTAB *WaitEventCustomHashByInfo; /* find names from infos */
static HTAB *WaitEventCustomHashByName; /* find infos from names */
-#define WAIT_EVENT_CUSTOM_HASH_INIT_SIZE 16
-#define WAIT_EVENT_CUSTOM_HASH_MAX_SIZE 128
+#define WAIT_EVENT_CUSTOM_HASH_SIZE 128
/* hash table entries */
typedef struct WaitEventCustomEntryByInfo
@@ -106,9 +104,9 @@ WaitEventCustomShmemSize(void)
Size sz;
sz = MAXALIGN(sizeof(WaitEventCustomCounterData));
- sz = add_size(sz, hash_estimate_size(WAIT_EVENT_CUSTOM_HASH_MAX_SIZE,
+ sz = add_size(sz, hash_estimate_size(WAIT_EVENT_CUSTOM_HASH_SIZE,
sizeof(WaitEventCustomEntryByInfo)));
- sz = add_size(sz, hash_estimate_size(WAIT_EVENT_CUSTOM_HASH_MAX_SIZE,
+ sz = add_size(sz, hash_estimate_size(WAIT_EVENT_CUSTOM_HASH_SIZE,
sizeof(WaitEventCustomEntryByName)));
return sz;
}
@@ -138,8 +136,7 @@ WaitEventCustomShmemInit(void)
info.entrysize = sizeof(WaitEventCustomEntryByInfo);
WaitEventCustomHashByInfo =
ShmemInitHash("WaitEventCustom hash by wait event information",
- WAIT_EVENT_CUSTOM_HASH_INIT_SIZE,
- WAIT_EVENT_CUSTOM_HASH_MAX_SIZE,
+ WAIT_EVENT_CUSTOM_HASH_SIZE,
&info,
HASH_ELEM | HASH_BLOBS);
@@ -148,8 +145,7 @@ WaitEventCustomShmemInit(void)
info.entrysize = sizeof(WaitEventCustomEntryByName);
WaitEventCustomHashByName =
ShmemInitHash("WaitEventCustom hash by name",
- WAIT_EVENT_CUSTOM_HASH_INIT_SIZE,
- WAIT_EVENT_CUSTOM_HASH_MAX_SIZE,
+ WAIT_EVENT_CUSTOM_HASH_SIZE,
&info,
HASH_ELEM | HASH_STRINGS);
}
@@ -238,7 +234,7 @@ WaitEventCustomNew(uint32 classId, const char *wait_event_name)
/* Allocate a new event Id */
SpinLockAcquire(&WaitEventCustomCounter->mutex);
- if (WaitEventCustomCounter->nextId >= WAIT_EVENT_CUSTOM_HASH_MAX_SIZE)
+ if (WaitEventCustomCounter->nextId >= WAIT_EVENT_CUSTOM_HASH_SIZE)
{
SpinLockRelease(&WaitEventCustomCounter->mutex);
ereport(ERROR,
diff --git a/src/include/storage/shmem.h b/src/include/storage/shmem.h
index 2a9e9becd26..0ae609aca8b 100644
--- a/src/include/storage/shmem.h
+++ b/src/include/storage/shmem.h
@@ -32,7 +32,7 @@ extern void InitShmemAllocator(PGShmemHeader *seghdr);
extern void *ShmemAlloc(Size size);
extern void *ShmemAllocNoError(Size size);
extern bool ShmemAddrIsValid(const void *addr);
-extern HTAB *ShmemInitHash(const char *name, int64 init_size, int64 max_size,
+extern HTAB *ShmemInitHash(const char *name, int64 nelems,
HASHCTL *infoP, int hash_flags);
extern void *ShmemInitStruct(const char *name, Size size, bool *foundPtr);
extern Size add_size(Size s1, Size s2);
--
2.47.3
From 58939da412e9c55e50a13f43861c89dc2be72061 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 27 Mar 2026 22:47:20 +0200
Subject: [PATCH v1 5/8] Prevent shared memory hash tables from growing beyond
initial size
Set HASH_FIXED_SIZE on all shared memory hash tables, to prevent them
from growing after the initial allocation. It was always weirdly
indeterministic that if you e.g. took a lot of locks, consuming all
the unused shared memory for the lock table, you couldn't use that
space for other things anymore, but if you used it for other things
first, then it was no longer available for locks. Better to be strict
and predictable.
Increase SHMEM_INDEX_SIZE because we were already above the old value,
and it's now a hard limit.
XXX Many callers of ShmemInitHash() still pass HASH_FIXED_SIZE, but that's
now unnecessary because ShmemInitHash() will set it anyway. I didn't
go clean them up yet.
---
src/backend/storage/ipc/shmem.c | 20 ++++++++------------
src/include/storage/shmem.h | 4 ++--
2 files changed, 10 insertions(+), 14 deletions(-)
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index fd2c7c4590a..47065bb3603 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -24,9 +24,8 @@
* available to POSTGRES: fixed-size structures, queues and hash
* tables. Fixed-size structures contain things like global variables
* for a module and should never be allocated after the shared memory
- * initialization phase. Hash tables have a fixed maximum size, but
- * their actual size can vary dynamically. When entries are added
- * to the table, more space is allocated. Queues link data structures
+ * initialization phase. Hash tables have a fixed maximum size and
+ * cannot grow beyond that. Queues link data structures
* that have been allocated either within fixed-size structures or as hash
* buckets. Each shared data structure has a string name to identify
* it (assigned in the module that declares it).
@@ -56,11 +55,7 @@
*
* (d) memory allocation model: shared memory can never be
* freed, once allocated. Each hash table has its own free list,
- * so hash buckets can be reused when an item is deleted. However,
- * if one hash table grows very large and then shrinks, its space
- * cannot be redistributed to other tables. We could build a simple
- * hash bucket garbage collector if need be. Right now, it seems
- * unnecessary.
+ * so hash buckets can be reused when an item is deleted.
*/
#include "postgres.h"
@@ -187,7 +182,7 @@ InitShmemAllocator(PGShmemHeader *seghdr)
info.dsize = info.max_dsize = hash_select_dirsize(SHMEM_INDEX_SIZE);
info.alloc = ShmemHashAlloc;
info.alloc_arg = NULL;
- hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
+ hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE;
if (!IsUnderPostmaster)
{
size = hash_get_shared_size(&info, hash_flags);
@@ -318,7 +313,7 @@ ShmemAddrIsValid(const void *addr)
* *infoP and hash_flags must specify at least the entry sizes and key
* comparison semantics (see hash_create()). Flag bits and values specific
* to shared-memory hash tables are added here, except that callers may
- * choose to specify HASH_PARTITION and/or HASH_FIXED_SIZE.
+ * choose to specify HASH_PARTITION.
*
* Note: before Postgres 9.0, this function returned NULL for some failure
* cases. Now, it always throws error instead, so callers need not check
@@ -336,14 +331,15 @@ ShmemInitHash(const char *name, /* table string name for shmem index */
/*
* Hash tables allocated in shared memory have a fixed directory; it can't
* grow or other backends wouldn't be able to find it. So, make sure we
- * make it big enough to start with.
+ * make it big enough to start with. We also allocate all the buckets
+ * upfront, so hash tables cannot grow later.
*
* The shared memory allocator must be specified too.
*/
infoP->dsize = infoP->max_dsize = hash_select_dirsize(nelems);
infoP->alloc = ShmemHashAlloc;
infoP->alloc_arg = NULL;
- hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
+ hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE;
/* look it up in the shmem index */
location = ShmemInitStruct(name,
diff --git a/src/include/storage/shmem.h b/src/include/storage/shmem.h
index 0ae609aca8b..2fccbdb534c 100644
--- a/src/include/storage/shmem.h
+++ b/src/include/storage/shmem.h
@@ -46,8 +46,8 @@ extern void RequestAddinShmemSpace(Size size);
/* size constants for the shmem index table */
/* max size of data structure string name */
#define SHMEM_INDEX_KEYSIZE (48)
- /* estimated size of the shmem index table (not a hard limit) */
-#define SHMEM_INDEX_SIZE (64)
+ /* max number of named shmem structures and hash tables */
+#define SHMEM_INDEX_SIZE (256)
/* this is a hash bucket in the shmem index table */
typedef struct
--
2.47.3
From 8debe072c3123ddbe065d677cf63c091414c03f9 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Sat, 28 Mar 2026 00:32:29 +0200
Subject: [PATCH v1 6/8] Allocate all parts of shmem hash table from a single
contiguous area
Previously, the shared header (HASHHDR) and the directory were
allocated by the caller, and passed to hash_create(), while the actual
elements were allocated separately with ShmemAlloc(). After this
commit, all the memory needed by the header, the directory, and all
the elements is allocated using a single ShmemInitStruct() call, and
the different parts are carved out of that allocation. This way the
ShmemIndex entries (and thus pg_shmem_allocations) reflect the size
size of the whole hash table.
Commit f5930f9a98 attempted this earlier, but it had to be reverted.
The new strategy is to let dynahash perform all the allocations with
the alloc function, but have the alloc function carve out the parts
from the one larger allocation. The shared header and the directory
are now also allocated with alloc calls, instead of passing the area
for those directly from the caller.
---
src/backend/storage/ipc/shmem.c | 71 +++++++++++++++++++++-------
src/backend/utils/hash/dynahash.c | 78 +++++++++++++------------------
src/include/utils/hsearch.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
4 files changed, 91 insertions(+), 64 deletions(-)
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 47065bb3603..c8171125871 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -112,6 +112,16 @@ static bool firstNumaTouch = true;
Datum pg_numa_available(PG_FUNCTION_ARGS);
+/*
+ * A very simple allocator used to carve out different parts of a hash table,
+ * from a previously allocated contiguous shared memory area.
+ */
+typedef struct shmem_hash_allocator
+{
+ char *next; /* start of free space in the area */
+ char *end; /* end of the shmem area */
+} shmem_hash_allocator;
+
/*
* InitShmemAllocator() --- set up basic pointers to shared memory.
*
@@ -126,7 +136,7 @@ InitShmemAllocator(PGShmemHeader *seghdr)
Size offset;
HASHCTL info;
int hash_flags;
- size_t size;
+ shmem_hash_allocator allocator;
#ifndef EXEC_BACKEND
Assert(!IsUnderPostmaster);
@@ -182,15 +192,27 @@ InitShmemAllocator(PGShmemHeader *seghdr)
info.dsize = info.max_dsize = hash_select_dirsize(SHMEM_INDEX_SIZE);
info.alloc = ShmemHashAlloc;
info.alloc_arg = NULL;
- hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE;
+ hash_flags = HASH_ELEM | HASH_STRINGS
+ | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE;
+
if (!IsUnderPostmaster)
{
- size = hash_get_shared_size(&info, hash_flags);
- ShmemAllocator->index = (HASHHDR *) ShmemAlloc(size);
+ size_t size = hash_estimate_size(SHMEM_INDEX_SIZE, info.entrysize);
+ char *location = ShmemAlloc(size);
+
+ allocator.next = location;
+ allocator.end = location + size;
+ info.alloc_arg = &allocator;
+
+ info.hctl = NULL;
+ hash_flags |= HASH_ALLOC | HASH_FIXED_SIZE;
+ ShmemAllocator->index = (HASHHDR *) location;
}
else
+ {
+ info.hctl = ShmemAllocator->index;
hash_flags |= HASH_ATTACH;
- info.hctl = ShmemAllocator->index;
+ }
ShmemIndex = hash_create("ShmemIndex", SHMEM_INDEX_SIZE, &info, hash_flags);
Assert(ShmemIndex != NULL);
}
@@ -233,9 +255,17 @@ ShmemAllocNoError(Size size)
static void *
ShmemHashAlloc(Size size, void *alloc_arg)
{
- Size allocated_size;
+ shmem_hash_allocator *allocator = (shmem_hash_allocator *) alloc_arg;
+ void *result;
- return ShmemAllocRaw(size, &allocated_size);
+ size = MAXALIGN(size);
+
+ if (allocator->end - allocator->next < size)
+ return NULL;
+ result = allocator->next;
+ allocator->next += size;
+
+ return result;
}
/*
@@ -321,12 +351,14 @@ ShmemAddrIsValid(const void *addr)
*/
HTAB *
ShmemInitHash(const char *name, /* table string name for shmem index */
- int64 nelems, /* size of the table */
+ int64 nelems, /* size of the table */
HASHCTL *infoP, /* info about key and bucket size */
int hash_flags) /* info about infoP */
{
bool found;
+ size_t size;
void *location;
+ shmem_hash_allocator allocator;
/*
* Hash tables allocated in shared memory have a fixed directory; it can't
@@ -341,20 +373,27 @@ ShmemInitHash(const char *name, /* table string name for shmem index */
infoP->alloc_arg = NULL;
hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE;
- /* look it up in the shmem index */
- location = ShmemInitStruct(name,
- hash_get_shared_size(infoP, hash_flags),
- &found);
+ size = hash_estimate_size(nelems, infoP->entrysize);
+
+ /* look it up in the shmem index or allocate */
+ location = ShmemInitStruct(name, size, &found);
/*
* if it already exists, attach to it rather than allocate and initialize
* new space
*/
- if (found)
+ if (!found)
+ {
+ allocator.next = (char *) location;
+ allocator.end = (char *) location + size;
+ infoP->alloc_arg = &allocator;
+ }
+ else
+ {
+ /* Pass location of hashtable header to hash_create */
+ infoP->hctl = (HASHHDR *) location;
hash_flags |= HASH_ATTACH;
-
- /* Pass location of hashtable header to hash_create */
- infoP->hctl = (HASHHDR *) location;
+ }
return hash_create(name, nelems, infoP, hash_flags);
}
diff --git a/src/backend/utils/hash/dynahash.c b/src/backend/utils/hash/dynahash.c
index d3dd16a4300..1173304ef0f 100644
--- a/src/backend/utils/hash/dynahash.c
+++ b/src/backend/utils/hash/dynahash.c
@@ -195,6 +195,9 @@ struct HASHHDR
int nelem_alloc; /* number of entries to allocate at once */
bool isfixed; /* if true, don't enlarge */
+ /* Current directory. In shared tables, this doesn't change */
+ HASHSEGMENT *dir;
+
#ifdef HASH_STATISTICS
/*
@@ -224,7 +227,7 @@ struct HTAB
HashCompareFunc match; /* key comparison function */
HashCopyFunc keycopy; /* key copying function */
HashAllocFunc alloc; /* memory allocator */
- void *alloc_arg; /* opaque argument to pass to allocator function */
+ void *alloc_arg; /* opaque argument to pass to alloc function */
MemoryContext hcxt; /* memory context if default allocator used */
char *tabname; /* table name (for error messages) */
bool isshared; /* true if table is in shared memory */
@@ -294,7 +297,7 @@ DynaHashAlloc(Size size, void *alloc_arg)
MemoryContext cxt = (MemoryContext) alloc_arg;
Assert(MemoryContextIsValid(cxt));
- return MemoryContextAllocExtended(cxt, size, MCXT_ALLOC_NO_OOM);
+ return MemoryContextAllocExtended(cxt, size, MCXT_ALLOC_NO_OOM);
}
@@ -374,6 +377,8 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags)
* hash_destroy very simple. The memory context is made a child of either
* a context specified by the caller, or TopMemoryContext if nothing is
* specified.
+ *
+ * Note that HASH_DIRSIZE and HASH_ALLOC had better be set as well.
*/
if (flags & HASH_SHARED_MEM)
{
@@ -486,21 +491,18 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags)
if (flags & HASH_SHARED_MEM)
{
- /*
- * ctl structure and directory are preallocated for shared memory
- * tables. Note that HASH_DIRSIZE and HASH_ALLOC had better be set as
- * well.
- */
- hashp->hctl = info->hctl;
- hashp->dir = (HASHSEGMENT *) (((char *) info->hctl) + sizeof(HASHHDR));
hashp->hcxt = NULL;
hashp->isshared = true;
/* hash table already exists, we're just attaching to it */
if (flags & HASH_ATTACH)
{
+ hctl = info->hctl;
+
+ hashp->hctl = hctl;
+ hashp->dir = hctl->dir;
+
/* make local copies of some heavily-used values */
- hctl = hashp->hctl;
hashp->keysize = hctl->keysize;
return hashp;
@@ -515,14 +517,20 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags)
hashp->isshared = false;
}
+ /*
+ * Allocate the header structure.
+ *
+ * XXX: In case of a shared memory hash table, other procesess need the
+ * pointer to the header to re-find the hash table. There is currently no
+ * explicit way to pass it back from here, the caller relies on the fact
+ * that this is the first allocation made with the alloc function. That's
+ * a little ugly, but works for now.
+ */
+ hashp->hctl = (HASHHDR *) hashp->alloc(sizeof(HASHHDR), hashp->alloc_arg);
if (!hashp->hctl)
- {
- hashp->hctl = (HASHHDR *) hashp->alloc(sizeof(HASHHDR), hashp->alloc_arg);
- if (!hashp->hctl)
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory")));
- }
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory")));
hashp->frozen = false;
@@ -725,25 +733,17 @@ init_htab(HTAB *hashp, int64 nelem)
nsegs = next_pow2_int(nsegs);
/*
- * Make sure directory is big enough. If pre-allocated directory is too
- * small, choke (caller screwed up).
+ * Make sure directory is big enough.
*/
if (nsegs > hctl->dsize)
- {
- if (!(hashp->dir))
- hctl->dsize = nsegs;
- else
- return false;
- }
+ hctl->dsize = nsegs;
/* Allocate a directory */
- if (!(hashp->dir))
- {
- hashp->dir = (HASHSEGMENT *)
- hashp->alloc(hctl->dsize * sizeof(HASHSEGMENT), hashp->alloc_arg);
- if (!hashp->dir)
- return false;
- }
+ hctl->dir = (HASHSEGMENT *)
+ hashp->alloc(hctl->dsize * sizeof(HASHSEGMENT), hashp->alloc_arg);
+ if (!hctl->dir)
+ return false;
+ hashp->dir = hctl->dir;
/* Allocate initial segments */
for (segp = hashp->dir; hctl->nsegs < nsegs; hctl->nsegs++, segp++)
@@ -832,19 +832,6 @@ hash_select_dirsize(int64 num_entries)
return nDirEntries;
}
-/*
- * Compute the required initial memory allocation for a shared-memory
- * hashtable with the given parameters. We need space for the HASHHDR
- * and for the (non expansible) directory.
- */
-Size
-hash_get_shared_size(HASHCTL *info, int flags)
-{
- Assert(flags & HASH_DIRSIZE);
- Assert(info->dsize == info->max_dsize);
- return sizeof(HASHHDR) + info->dsize * sizeof(HASHSEGMENT);
-}
-
/********************** DESTROY ROUTINES ************************/
@@ -1648,6 +1635,7 @@ dir_realloc(HTAB *hashp)
{
memcpy(p, old_p, old_dirsize);
MemSet(((char *) p) + old_dirsize, 0, new_dirsize - old_dirsize);
+ hashp->hctl->dir = p;
hashp->dir = p;
hashp->hctl->dsize = new_dsize;
diff --git a/src/include/utils/hsearch.h b/src/include/utils/hsearch.h
index 03bc1c171cd..b60ae20acc7 100644
--- a/src/include/utils/hsearch.h
+++ b/src/include/utils/hsearch.h
@@ -80,10 +80,10 @@ typedef struct HASHCTL
HashCopyFunc keycopy; /* key copying function */
/* Used if HASH_ALLOC flag is set: */
HashAllocFunc alloc; /* memory allocator */
- void *alloc_arg; /* opaque argument to pass to allocator function */
+ void *alloc_arg; /* opaque argument to pass to alloc */
/* Used if HASH_CONTEXT flag is set: */
MemoryContext hcxt; /* memory context to use for allocations */
- /* Used if HASH_SHARED_MEM flag is set: */
+ /* Used if HASH_ATTACH flag is set: */
HASHHDR *hctl; /* location of header in shared mem */
} HASHCTL;
@@ -150,7 +150,6 @@ extern void hash_seq_term(HASH_SEQ_STATUS *status);
extern void hash_freeze(HTAB *hashp);
extern Size hash_estimate_size(int64 num_entries, Size entrysize);
extern int64 hash_select_dirsize(int64 num_entries);
-extern Size hash_get_shared_size(HASHCTL *info, int flags);
extern void AtEOXact_HashTables(bool isCommit);
extern void AtEOSubXact_HashTables(bool isCommit, int nestDepth);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 712d84128ca..5f6502be030 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4209,6 +4209,7 @@ shm_mq_result
shm_toc
shm_toc_entry
shm_toc_estimator
+shmem_hash_allocator
shmem_request_hook_type
shmem_startup_hook_type
sig_atomic_t
--
2.47.3
From 85cf9e3555458b6ba77e7113317694e3d5aa151f Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 27 Mar 2026 23:25:35 +0200
Subject: [PATCH v1 7/8] Remove HASH_DIRSIZE, always use the default algorithm
to select it
It's not very useful to specify a non-standard directory size, and
always just used the default algorithm for it. hash_estimate_size()
assumed the default directory size anyway.
---
src/backend/storage/ipc/shmem.c | 7 ++--
src/backend/utils/hash/dynahash.c | 54 +++++++------------------------
src/include/utils/hsearch.h | 6 +---
3 files changed, 14 insertions(+), 53 deletions(-)
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index c8171125871..2ee7d06e4d3 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -189,11 +189,9 @@ InitShmemAllocator(PGShmemHeader *seghdr)
*/
info.keysize = SHMEM_INDEX_KEYSIZE;
info.entrysize = sizeof(ShmemIndexEnt);
- info.dsize = info.max_dsize = hash_select_dirsize(SHMEM_INDEX_SIZE);
info.alloc = ShmemHashAlloc;
info.alloc_arg = NULL;
- hash_flags = HASH_ELEM | HASH_STRINGS
- | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE;
+ hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_FIXED_SIZE;
if (!IsUnderPostmaster)
{
@@ -368,10 +366,9 @@ ShmemInitHash(const char *name, /* table string name for shmem index */
*
* The shared memory allocator must be specified too.
*/
- infoP->dsize = infoP->max_dsize = hash_select_dirsize(nelems);
infoP->alloc = ShmemHashAlloc;
infoP->alloc_arg = NULL;
- hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE;
+ hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_FIXED_SIZE;
size = hash_estimate_size(nelems, infoP->entrysize);
diff --git a/src/backend/utils/hash/dynahash.c b/src/backend/utils/hash/dynahash.c
index 1173304ef0f..378213fda6c 100644
--- a/src/backend/utils/hash/dynahash.c
+++ b/src/backend/utils/hash/dynahash.c
@@ -115,10 +115,12 @@
* a whole lot of records per bucket or performance goes down.
*
* In a hash table allocated in shared memory, the directory cannot be
- * expanded because it must stay at a fixed address. The directory size
- * should be selected using hash_select_dirsize (and you'd better have
- * a good idea of the maximum number of entries!). For non-shared hash
- * tables, the initial directory size can be left at the default.
+ * expanded because it must stay at a fixed address. The directory size is
+ * chosen at creation based on the initial number of initial elements, so even
+ * though we support allocating more elements later, performance will suffer
+ * if the table grows much beyond the initial size. (Currently, shared memory
+ * hash tables are only created by ShmemInitHash() though, which doesn't
+ * support growing at all.)
*/
#define HASH_SEGSIZE 256
#define HASH_SEGSIZE_SHIFT 8 /* must be log2(HASH_SEGSIZE) */
@@ -553,15 +555,6 @@ hash_create(const char *tabname, int64 nelem, const HASHCTL *info, int flags)
hctl->num_partitions = info->num_partitions;
}
- /*
- * SHM hash tables have fixed directory size passed by the caller.
- */
- if (flags & HASH_DIRSIZE)
- {
- hctl->max_dsize = info->max_dsize;
- hctl->dsize = info->dsize;
- }
-
/* remember the entry sizes, too */
hctl->keysize = info->keysize;
hctl->entrysize = info->entrysize;
@@ -735,8 +728,11 @@ init_htab(HTAB *hashp, int64 nelem)
/*
* Make sure directory is big enough.
*/
- if (nsegs > hctl->dsize)
- hctl->dsize = nsegs;
+ hctl->dsize = nsegs;
+
+ /* SHM hash tables have fixed directory. */
+ if (hashp->isshared)
+ hctl->max_dsize = hctl->dsize;
/* Allocate a directory */
hctl->dir = (HASHSEGMENT *)
@@ -804,34 +800,6 @@ hash_estimate_size(int64 num_entries, Size entrysize)
return size;
}
-/*
- * Select an appropriate directory size for a hashtable with the given
- * maximum number of entries.
- * This is only needed for hashtables in shared memory, whose directories
- * cannot be expanded dynamically.
- * NB: assumes that all hash structure parameters have default values!
- *
- * XXX this had better agree with the behavior of init_htab()...
- */
-int64
-hash_select_dirsize(int64 num_entries)
-{
- int64 nBuckets,
- nSegments,
- nDirEntries;
-
- /* estimate number of buckets wanted */
- nBuckets = next_pow2_int64(num_entries);
- /* # of segments needed for nBuckets */
- nSegments = next_pow2_int64((nBuckets - 1) / HASH_SEGSIZE + 1);
- /* directory entries */
- nDirEntries = DEF_DIRSIZE;
- while (nDirEntries < nSegments)
- nDirEntries <<= 1; /* dir_alloc doubles dsize at each call */
-
- return nDirEntries;
-}
-
/********************** DESTROY ROUTINES ************************/
diff --git a/src/include/utils/hsearch.h b/src/include/utils/hsearch.h
index b60ae20acc7..115e97642f5 100644
--- a/src/include/utils/hsearch.h
+++ b/src/include/utils/hsearch.h
@@ -66,9 +66,6 @@ typedef struct HASHCTL
{
/* Used if HASH_PARTITION flag is set: */
int64 num_partitions; /* # partitions (must be power of 2) */
- /* Used if HASH_DIRSIZE flag is set: */
- int64 dsize; /* (initial) directory size */
- int64 max_dsize; /* limit to dsize if dir size is limited */
/* Used if HASH_ELEM flag is set (which is now required): */
Size keysize; /* hash key length in bytes */
Size entrysize; /* total user element size in bytes */
@@ -90,7 +87,7 @@ typedef struct HASHCTL
/* Flag bits for hash_create; most indicate which parameters are supplied */
#define HASH_PARTITION 0x0001 /* Hashtable is used w/partitioned locking */
/* 0x0002 is unused */
-#define HASH_DIRSIZE 0x0004 /* Set directory size (initial and max) */
+/* 0x0004 is unused */
#define HASH_ELEM 0x0008 /* Set keysize and entrysize (now required!) */
#define HASH_STRINGS 0x0010 /* Select support functions for string keys */
#define HASH_BLOBS 0x0020 /* Select support functions for binary keys */
@@ -149,7 +146,6 @@ extern void *hash_seq_search(HASH_SEQ_STATUS *status);
extern void hash_seq_term(HASH_SEQ_STATUS *status);
extern void hash_freeze(HTAB *hashp);
extern Size hash_estimate_size(int64 num_entries, Size entrysize);
-extern int64 hash_select_dirsize(int64 num_entries);
extern void AtEOXact_HashTables(bool isCommit);
extern void AtEOSubXact_HashTables(bool isCommit, int nestDepth);
--
2.47.3
From 87063f3d4e1d632fd117746adc2bb748d8630754 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 27 Mar 2026 23:34:59 +0200
Subject: [PATCH v1 8/8] Make ShmemIndex visible in the pg_shmem_allocations
view
---
src/backend/storage/ipc/shmem.c | 28 ++++++++++++++++++++++------
1 file changed, 22 insertions(+), 6 deletions(-)
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 2ee7d06e4d3..08dd7ab7a1c 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -137,6 +137,7 @@ InitShmemAllocator(PGShmemHeader *seghdr)
HASHCTL info;
int hash_flags;
shmem_hash_allocator allocator;
+ Size size = 0;
#ifndef EXEC_BACKEND
Assert(!IsUnderPostmaster);
@@ -195,16 +196,14 @@ InitShmemAllocator(PGShmemHeader *seghdr)
if (!IsUnderPostmaster)
{
- size_t size = hash_estimate_size(SHMEM_INDEX_SIZE, info.entrysize);
- char *location = ShmemAlloc(size);
+ size = hash_estimate_size(SHMEM_INDEX_SIZE, info.entrysize);
+ ShmemAllocator->index = (HASHHDR *) ShmemAlloc(size);
- allocator.next = location;
- allocator.end = location + size;
+ allocator.next = (char *) ShmemAllocator->index;
+ allocator.end = (char *) ShmemAllocator->index + size;
info.alloc_arg = &allocator;
-
info.hctl = NULL;
hash_flags |= HASH_ALLOC | HASH_FIXED_SIZE;
- ShmemAllocator->index = (HASHHDR *) location;
}
else
{
@@ -213,6 +212,23 @@ InitShmemAllocator(PGShmemHeader *seghdr)
}
ShmemIndex = hash_create("ShmemIndex", SHMEM_INDEX_SIZE, &info, hash_flags);
Assert(ShmemIndex != NULL);
+
+ /*
+ * Add an entry for the ShmemIndex itself into ShmemIndex, so that it's
+ * visible in the pg_shmem_allocations view
+ */
+ if (!IsUnderPostmaster)
+ {
+ ShmemIndexEnt *result;
+ bool found;
+
+ result = (ShmemIndexEnt *)
+ hash_search(ShmemIndex, "ShmemIndex", HASH_ENTER, &found);
+ Assert(!found);
+ result->size = size;
+ result->allocated_size = size;
+ result->location = ShmemAllocator->index;
+ }
}
/*
--
2.47.3
