On Fri, 20 Sept 2024 at 05:03, Masahiko Sawada <sawada.m...@gmail.com> wrote: > I've done other benchmarking tests while changing the memory block > sizes from 8kB to 8MB. I measured the execution time of logical > decoding of one transaction that inserted 10M rows. I set > logical_decoding_work_mem large enough to avoid spilling behavior. In > this scenario, we allocate many memory chunks while decoding the > transaction and resulting in calling more malloc() in smaller memory > block sizes. Here are results (an average of 3 executions):
I was interested in seeing the memory consumption with the test that was causing an OOM due to the GenerationBlock fragmentation. You saw bad results with 8MB blocks and good results with 8KB blocks. The measure that's interesting here is the MemoryContextMemAllocated() for the GenerationContext in question. > The fact that we're using rb->size and txn->size to choose the > transactions to evict could make this idea less attractive. Even if we > defragment transactions, rb->size and txn->size don't change. > Therefore, it doesn't mean we can avoid evicting transactions due to > full of logical_decoding_work_mem, but just mean the amount of > allocated memory might have been reduced. I had roughly imagined that you'd add an extra field to store txn->size when the last fragmentation was done and only defrag the transaction when the ReorderBufferTXN->size is, say, double the last size when the changes were last defragmented. Of course, if the first defragmentation was enough to drop MemoryContextMemAllocated() below the concerning threshold above logical_decoding_work_mem then further defrags wouldn't be done, at least, until such times as the MemoryContextMemAllocated() became concerning again. If you didn't want to a full Size variable for the defragmentation size, you could just store a uint8 to store which power of 2 ReorderBufferTXN->size was when it was last defragmented. There are plenty of holds in that struct that could be filled without enlarging the struct. In general, it's a bit annoying to have to code around this GenerationContext fragmentation issue. Unfortunately, AllocSet could also suffer from fragmentation issues too if lots of chunks end up on freelists that are never reused, so using another context type might just create a fragmentation issue for a different workload. David
