On 2.12.2014 02:52, Tom Lane wrote: > Tomas Vondra <t...@fuzzy.cz> writes: >> On 2.12.2014 01:33, Tom Lane wrote: >>> What I suspect you're looking at here is the detritus of creation >>> of a huge number of memory contexts. mcxt.c keeps its own state >>> about existing contents in TopMemoryContext. So, if we posit that >>> those contexts weren't real small, there's certainly room to >>> believe that there was some major memory bloat going on recently. > >> Aha! MemoryContextCreate allocates the memory for the new context >> from TopMemoryContext explicitly, so that it survives resets of the >> parent context. Is that what you had in mind by keeping state about >> existing contexts? > > Right. > >> That'd probably explain the TopMemoryContext size, because >> array_agg() creates separate context for each group. So if you have >> 1M groups, you have 1M contexts. Although I don's see how the size >> of those contexts would matter? > > Well, if they're each 6K, that's your 6GB right there.
FWIW, I tried to repeat the test with the array_agg() patch, submitted to the next CF: http://www.postgresql.org/message-id/5479faea.3070...@fuzzy.cz and the memory context stats after the first iteration look like this: master (unpatched) ------------------ TopMemoryContext: 204878128 total in 25011 blocks; 204010192 free (750034 chunks); 867936 used patched with array_agg ---------------------- TopMemoryContext: 78128 total in 11 blocks; 10144 free (34 chunks); 67984 used Also, the RSS issue (which was the initial subject of this thread) pretty much disappeared. So indeed, this seems to be caused by the 'islands' referenced by TopMemoryContext. I wonder whether it'd be worth looking for places doing something similar to array_agg(), potentially creating many contexts in parallel, and maybe modifying them as in the patch. regards Tomas -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
