On Mar 24, 2011, at 11:53 AM, Martin Sustrik wrote: > Hi all, > >>> The memory allocator should be extensible (think of an allocator >>> strategy); > > I see there's interest in this feature, so let's figure out how to > implement it. > > 3. There are at least two use cases here. AFAIU, Douglas want to > custom-allocate the message bodies while Chunk want to custom-allocate > everything. We should think a bit more about whether one use case > subsumes the other or whether they are two separate use cases with two > separate solutions.
True, I am looking for a global replacement for all malloc/free calls. Even if the existing ones were just replaced by a macro that was easily modified at compile time that would make me happy. > 4. As a side note: Chuck, given the latest experiments we've done, it > doesn't look like the fragmentation of the heap is the problem. Recall > that all the chunks in question were either 8kB or 12kB long, which is a > multiply of page size. I still suspect that glibc just don't return the > allocated memory back to the system... if that is so, no custom > user-space allocator is going to help. I don't agree. While the experiments showed that the yqueue memory was being deallocated as expected, it may have still been causing some kind of fragmentation that prevented the memory from being returned back to the OS. When I was reading up on jemalloc [1] I noticed a passage early in the text regarding allocator-induced fragmentation. If the allocator let's too much fragmentation occur, this "leak" could make the Facebook application heaps grow beyond the bounds of physical memory and cause problems. My use-case is very similar in the sense that I have distributed applications that run for days or weeks at a time wherein a lot of sockets are created and destroyed over that time period. There appears to something about the way 0mq is allocating and freeing memory that creates this fragmentation and slow heap growth over time. If jemalloc can solve it, then I'm very interested. Note that even though some of the 0mq allocations are 8kB or 12kB, unless they are page aligned then even these sizes could cause fragmentation. If even one byte on a page is being held by the process, it cannot be relinquished to the OS hence the heap fragmentation. [1] http://on.fb.me/f5bVt5 _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
