On Thu, Feb 23, 2017 at 7:15 PM, Niklas Edmundsson <ni...@acc.umu.se> wrote: > On Thu, 23 Feb 2017, Yann Ylavic wrote: > >>> Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just >>> defines a new buffer size for use with the file bucket. It's a little >>> less >>> than 64K, I assume to make room for an allocation header: >>> >>> #define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64) >> >> >> Actually I'm not very pleased with this solution (or the final one >> that would make this size open / configurable). >> The issue is potentially the huge (order big-n) allocations which >> finally may hurt the system (fragmentation, OOM...). > > > Is this a real or theoretical problem?
Both. Fragmentation is a hard issue, but a constant is: the more you ask for big allocs, the more likely you won't be serviced one day or another (or another task will be killed for that, until yours). Long living (or closed to memory limits) systems suffer from this, no matter what allocator, small and large allocations fragment the memory (httpd is likely not the only program on the system), the only "remedy" is small order allocations (2^order pages, a "sane" order being lower than 2, hence order-1 on a system with 4K pages is 8K bytes). > > Our large-file cache module does 128k allocs to get a sane block size when > copying files to the cache. The only potential drawback we noticed was httpd > processes becoming bloated due to the default MaxMemFree 2048, so we're > running with MaxMemFree 256 now. With MaxMemFree 256 (KB per allocator), each APR allocator reclaims but mainly returns a lot more memory chunks to the system's allocator, which does a better job to in recycling many differently sized chunks than APR's (which is pretty basic in this regard, it's role is more about quickly recycling common sizes). With MaxMemFree 2048 (2MB), there is more builtin handling of "exotic" chunks, which may be painful. That might be something else, though... > > Granted, doing alloc/free for all outgoing data means way more alloc/free:s, > so we might just miss the issues because cache fills aren't as common. That's why reuse of common and reasonably sized chunks in the APR allocator can help. > > However, for large file performance I really don't buy into the notion that > it's a good idea to break everything into tiny puny blocks. The potential > for wasting CPU cycles on this micro-management is rather big... I don't think that a readv/writev of 16 iovecs of 8KB is (noticeably) slower than read/write of contiguous 128K, both might well end up in a scaterlist for the kernel/hardware. > > I do find iovecs useful, it the small blocks that gets me into skeptic > mode... Small blocks is not for networking, it's for internal use only. And remember that TLS records are 16K max anyway, give 128KB to openssl's SSL_write it will output 8 chunks of 16KB. > > > Kinda related: We also have the support for larger page sizes with modern > CPUs. Has anyone investigated if it makes sense allocating memory pools in > chunks that fit those large pages? I think PPC64 have 64K pages already. APR pools are already based on the page size IIRC. Regards, Yann.
