Hi Matthias, > This is a part of the work I discussed in my PGConf.DEV talk of this year [0]. > > Many of Index's allocated fields in RelationData have the same (or > practically the same) contents for many indexes. E.g. a btree index on > a bigint column will always have the same contents in rd_opfamily, > rd_opcintype, and rd_support, and (given the same opclass options in > each column) will have equivalent rd_supportinfo.
I'm glad to see work on reducing relcache's memory footprint. I've seen production workloads with 10-100k tables and many hundred thousand indexes. > The attached patchset adds a deduplication layer into the relcache, > which makes sure we only allocate one set of (rd_opfamily, > rd_opcintype, rd_support, rd_supportinfo) for indexes with equivalent > key definitions (so, a matching number of key attributes, opclasses, > and AM). Why did you specifically worked on deduplicating the index fields in RelationData? Is that consuming most out of all of RelationData? > Additionally, it includes a patch by Andres (polished by me) that adds > a proxy context, which reduces the the overhead of small and > long-lived allocations in (what we expect to be) small memory contexts > by forwarding the allocations to malloc (after wrapping the struct). > > Earlier versions of the patch adjusted aset.c to accept smaller memory > context sizes, but I abandoned that approach in favour of Andres' > ProxyContext -- it can outsource most the complexities of memory > management to the system allocator. > > Patches in this patchset: > 0001/0002: prepare relcache for deduplication. > 0003: implements the deduplication > 0004: Andres' ProxyContext patch > 0005: Use proxy context in relcache for 'index data' On the testing end: if this effort makes more progress, I could run this against the aforementioned database to get some real-world numbers. One more micro optimization that we could do is better packing RelationData. On my AMD64 system sizeof(RelationData) == 488 bytes. By reordering the members we could get it down to 440 bytes which is about 10% savings. -- David Geier
