On Jul 17, 2014, at 1:40 PM, Jason Evans <[email protected]> wrote:
> On Jul 17, 2014, at 10:29 AM, D'Alessandro, Luke K <[email protected]> > wrote: >> We’re using jemalloc to manage network pinned memory using the custom chunk >> allocator/deallocator functionality. Essentially we decorate the default >> chunk allocator with memory registration calls. This mostly works great. >> >> We just discovered though, that sometimes jemalloc will call pages_purge() >> on our chunks without telling us, which calls madvise(). >> >> [...] >> >> This break our network registration, because the network card has already >> captured the virtual->physical mapping for the pages, and we’re not able to >> (nor do we want to) reregister mappings—it’s expensive and complex to keep >> track of. >> >> We can mlock()/munlock() on some systems, but many of the systems we run on >> aren’t under our control and don’t like it when you mlock(). >> >> How important is this functionality? Can it be disabled at configure time? >> Would it make sense to add a dynamic configuration string to turn this off >> on a per-arena basis? > > Assuming you're using jemalloc exclusively for managing the pinned memory, > there's no problem at all with disabling the purging. You can do the with > MALLOC_CONF=lg_dirty_mult:-1 (or just change LG_DIRTY_MULT_DEFAULT in the > source code, given that you already have other changes in place). Thanks Jason, the environment variable works great. I don’t have any other changes in place, just using git master and the `arena.*.chunk.{alloc,dealloc}` functionality to register a function that forwards to the original {alloc,dalloc} and then uses IB verbs registration/deregistration on the returned chunks. I don’t really want to to use jemalloc exclusively for managing the pinned memory—in fact, it gets used to back all of the normal malloc/free calls in the code. We use allocx() dallocx() with the pinned arena directly for network managed memory. Will the MALLOC_CONF cause us problems with the rest of the our runtime? I could link something else first, to deal with those routines if this is a bad thing to disable in general (-lc?). > I'm planning to change dirty page purging quite a bit for jemalloc 4, in > order to add more effective hysteresis. The tuning mechanisms will change, > but I expect it will still be possible to turn purging off completely. Cool. It would be nice to have allocx() dallocx() take a “cache” reference instead of an arena reference, with the cache configured with arenas to handle cache misses or something to deal with multithreaded-accesses. Other than that we really like using the library and as long as our network memory doesn’t move between cores frequently, this works well. Luke _______________________________________________ jemalloc-discuss mailing list [email protected] http://www.canonware.com/mailman/listinfo/jemalloc-discuss
