On Tue, 7 Jul 2026 at 20:14, Heikki Linnakangas <[email protected]> wrote: > > On 07/07/2026 21:41, Dhruv Aron wrote: > > At Databricks, we’ve found that the existing dynahash table structure is > > leaving performance gains on the table when it comes to shared buffer > > lookups: the multi-level structure (directory, segment, bucket chain, > > freelist) appears excessive for the shared buffers and could be > > simplified to boost performance and lower memory overhead. As such, we > > are proposing a specialized hash table just for this purpose and would > > appreciate feedback on this approach. > > Yeah, the dynahash has features that we just don't need in the buffer > lookup table... > > The indirection with the directory is actually unnecessary for all the > shared memory hash tables, since none of them can be resized. I've > wondered if we should try to eliminate that from dynahash for all shmem > hash tables. But the buffer lookup table is very performance-critical, > so if there are any performance gains to be had, it's indeed probably > worthwhile to have a separate implementation just for it. > > > bufmgr.c also changed slightly to prevent a race condition. > > Hmm, we're now holding the buffer header lock much longer than before, > in InvalidateBuffer(). It's a spinlock, it really should not be held for > more than a few instructions. BufTableDelete() is very fast in the new > implementation, but still. Could we perhaps do some of > BufTableDelete()'s work ahead of time, before we acquire the buffer > header lock? Or maybe it's not a problem, in which case some kind of a > worst case scenario benchmark to show that would be nice. Maybe test how > it behaves when you have a lot of hash collisions, I think that'd make > BufTableDelete() more expensive. > > > My testing > > (helper script also attached) indicates that all three standard hash > > table operations (insert, lookup, and delete) generally execute > > significantly faster than the existing PG18 dynahash counterparts: > > Nice! > > I wonder how big the impact is with real world workloads. I've certainly > seen the buffer table lookups consume a fair share of CPU time, so I'd > assume that it shows up. > > > Another data point: > > unpatched master, with shared_buffers='128 MB': > > postgres=# select * from pg_shmem_allocations where name like 'Shared > Buffer%' order by name ; > name | off | size | allocated_size > ----------------------------+-----------+--------+---------------- > Shared Buffer Lookup Table | 141607040 | 926000 | 926108 > (1 row) > > With this patch: > > postgres=# select * from pg_shmem_allocations where name like 'Shared > Buffer%' order by name ; > name | off | size | allocated_size > ------------------------------+-----------+--------+---------------- > Shared Buffer Lookup Buckets | 141607040 | 65536 | 65644 > Shared Buffer Lookup Entries | 141672576 | 393216 | 393216 > (2 rows) > > So the new hash table takes much less memory. That's nice because you > can then fit more in CPU caches.
Where did this thread come from? I can't see any conversation beyond this single email, and as such, can't see any patch either. Thom
