Hi,
While running ANALYZE across a database with 500,000+ tables (~350 columns each) in a single client session, we hit with an error: ERROR: invalid memory alloc request size 1073741824 This isn't just "the catcache got big and used a lot of RAM" - it's a specific, deterministic ceiling in RehashCatCache()/RehashCatCacheLists() that any sufficiently large catcache will hit, regardless of how much memory is available on the machine. What We Observed ----------------- After ANALYZE had processed ~505,180 tables (each contains 350 cols approx) in one session, a memory context dump showed: === Memory Contexts after vacuuming 505180 tables === CacheMemoryContext 26 GB 26 GB 1 Attopt cache 120 MB 118 MB 1 PgStat Shared Ref 35 MB 35 MB 1 Relcache by OID 24 MB 19 MB 1 PgStat Shared Ref Hash 24 MB 24 MB 1 ... Eventually a subsequent allocation fails with: ERROR: invalid memory alloc request size 1073741824 Root cause ---------- catcache.c's CatalogCacheCreateEntry() triggers a rehash whenever the load factor exceeds 2: if (cache->cc_ntup > cache->cc_nbuckets * 2) RehashCatCache(cache); RehashCatCache() doubles the bucket count and allocates the new bucket array with the *regular* allocator: newnbuckets = cp->cc_nbuckets * 2; newbucket = (dlist_head *) MemoryContextAllocZero(CacheMemoryContext, newnbuckets * sizeof(dlist_head)); MemoryContextAllocZero() enforces MaxAllocSize (0x3fffffff = 1,073,741,823 bytes - "1 gigabyte minus 1", see memutils.h). sizeof (dlist_head) is 16 bytes on 64-bit platforms. Buckets are powers of two, so the bucket array size doubles:..., 512MB, then exactly 1GB. That final doubling requests: 67,108,864 buckets * 16 bytes = 1,073,741,824 bytes which is exactly 1 byte over MaxAllocSize, so the allocation is refused and the backend hard-errors with "invalid memory alloc request size 1073741824" - the exact message and exact number we observed. Given the load factor of 2, this doubling occurs once a catcache accumulates roughly 67.1 million live tuples. For ATTNUM specifically (examine_attribute() in analyze.c calls SearchSysCache2(ATTNUM, ...) once per column of every relation ANALYZE processes), that means any session whose ANALYZE/autovacuum touches enough distinct relations to push pg_attribute entries past ~67M will hit this. Why this matters beyond "reduce your table count" -------------------------------------------------- - It fails outright rather than degrading gracefully or scaling with available RAM. A machine with terabytes of free memory will still hit this at exactly the same tuple count, because the ceiling is MaxAllocSize, not physical memory. - Felt patching a single call site as a partial fix. We tested replacing SearchSysCache2() with a direct systable_beginscan() in examine_attribute(), which will avoids populating the ATTNUM catcache from that one code path. That slows down how fast the cache fills, but any other code path that still uses the catcache heavily (in this backend or others) will eventually hit the same ceiling. Open Questions / Possible Fixes -------------------------------- 1. Have RehashCatCache()/RehashCatCacheLists() allocate the bucket array with MemoryContextAllocHuge() instead of MemoryContextAllocZero(). Is there a known reason this wasn't already done, or would a patch along these lines be welcome? 2. If there's a reason the 1GB ceiling must stay in place for this allocation, then is splitting ANALYZE into multiple sessions/backends, each covering a limited number of tables (so no single backend's catcache grows past the threshold), the only real workaround available today? Or is there some other supported way to Invalidate/reset a backend's catcache growth (short of restarting the connection) that we're missing? The reproduction scripts we used are attached below. Thanks & Regards, Reshmithaa B Member Technical Staff ZOHO Corporation
create.sh
Description: Binary data
test.sh
Description: Binary data
