2012/10/6 Justin Lebar <[email protected]>: >> It is equivalent, as long as you can ensure that _all_ allocations go >> through your wrappered malloc. > > For the purposes of the live-block-walker, does it matter if we miss > the occasional malloc()/free()?
It matters for just being able to run at all without crashing: a mismatch between wrappered and non-wrappered functions would cause a crash or corruption like allocator mismatches typically do. Benoit > > On Sat, Oct 6, 2012 at 11:51 AM, Benoit Jacob <[email protected]> > wrote: >> It is equivalent, as long as you can ensure that _all_ allocations go >> through your wrappered malloc. The terror of anyone making such a >> wrappered malloc is that some allocations might escape it, such as: >> allocations made by libraries that your application uses, etc. Not >> being familiar with the field, I just found it easier to be confident >> that all allocations would be correctly wrappered by doing it in the >> memory allocator itself, since if any allocation did not go through >> it, we would have allocator mismatches anyway. >> >> Here is an example of what scared me away from trying to do it in the >> application: this is a comment in jemalloc 3.0's src/jemalloc.c: >> >> #if ((is_malloc(je_malloc) == 1) && defined(__GLIBC__) && >> !defined(__UCLIBC__)) >> /* >> * glibc provides the RTLD_DEEPBIND flag for dlopen which can make it >> possible >> * to inconsistently reference libc's malloc(3)-compatible functions >> * (https://bugzilla.mozilla.org/show_bug.cgi?id=493541). >> * >> * These definitions interpose hooks in glibc. The functions are actually >> * passed an extra argument for the caller return address, which will be >> * ignored. >> */ >> JEMALLOC_EXPORT void (* const __free_hook)(void *ptr) = je_free; >> JEMALLOC_EXPORT void *(* const __malloc_hook)(size_t size) = je_malloc; >> JEMALLOC_EXPORT void *(* const __realloc_hook)(void *ptr, size_t size) = >> je_realloc; >> JEMALLOC_EXPORT void *(* const __memalign_hook)(size_t alignment, size_t >> size) = >> je_memalign; >> #endif >> >> Benoit >> >> 2012/10/6 Salvatore Sanfilippo <[email protected]>: >>> Hello, >>> >>> what is the advantage of this approach compared to doing it entirely >>> in the application code just wrapping malloc/realloc/free? >>> Basically wrappered_malloc() allocates a bit more space accordingly to >>> the metadata to store, store this metadata at the start of the block, >>> and then returns the pointer advanced to the start of the empty space. >>> >>> Doing it in the context of the application makes it >>> malloc-implementation agnostic that can be an advantage. >>> >>> Regards, >>> Salvatore >>> >>> On Sat, Oct 6, 2012 at 7:28 AM, Benoit Jacob <[email protected]> >>> wrote: >>>> Hello, >>>> >>>> The attached patch instruments jemalloc 3.0, adding the ability to >>>> enumerate all live blocks. >>>> >>>> Currently, the only information stored about blocks is their (payload) >>>> address and size, but the plan is to also store their allocation call >>>> stack. >>>> >>>> This is achieved by allocating larger blocks than requested, and using >>>> the extra space to store doubly linked list elements. >>>> >>>> This is provided just in case it might be useful to anyone, not >>>> considered ready for inclusion in jemalloc. It's been tested for 15 >>>> minutes in a Firefox build. >>>> >>>> Details about the overhead, and how it could be reduced: >>>> * Memory overhead is, per block: 32 bytes on 32-bit systems, 48 bytes >>>> on 64-bit systems (assuming size_t == uintptr_t). Could easily be >>>> reduced to 16 bytes in both cases (by using a XOR linked list and >>>> assuming that no block exceeds 4G). >>>> * Slowness overhead is essentially an additional mutex to lock on >>>> every malloc/free call. Could be solved in various ways, either >>>> copying what jemalloc does internally, or by using a lock-free list. >>>> >>>> If you want to test it out, currently the only built-in way to output >>>> the list of blocks is to call je_dump_list_of_blocks(void) e.g. from >>>> your debugger. See its code to see the relevant calls. Sample output >>>> from Firefox: >>>> >>>> ...snip... >>>> Block 193965: real block = 0x7fffba218580, payload = 0x7fffba2185b0, >>>> payload size = 64 >>>> Block 193966: real block = 0x7fffc02dd000, payload = 0x7fffc02dd030, >>>> payload size = 1024 >>>> Block 193967: real block = 0x7fffc2053ce0, payload = 0x7fffc2053d10, >>>> payload size = 24 >>>> Block 193968: real block = 0x7fffc5b4ed80, payload = 0x7fffc5b4edb0, >>>> payload size = 80 >>>> Block 193969: real block = 0x7fffd119e240, payload = 0x7fffd119e270, >>>> payload size = 64 >>>> Block 193970: real block = 0x7fffcfb4aa60, payload = 0x7fffcfb4aa90, >>>> payload size = 24 >>>> Block 193971: real block = 0x7fffbb85e1f0, payload = 0x7fffbb85e220, >>>> payload size = 24 >>>> Block 193972: real block = 0x7fffe20a62e0, payload = 0x7fffe20a6310, >>>> payload size = 32 >>>> >>>> End enumeration of 193973 jemalloc blocks. >>>> >>>> Cheers, >>>> Benoit >>>> >>>> _______________________________________________ >>>> jemalloc-discuss mailing list >>>> [email protected] >>>> http://www.canonware.com/mailman/listinfo/jemalloc-discuss >>>> >>> >>> >>> >>> -- >>> Salvatore 'antirez' Sanfilippo >>> open source developer - VMware >>> http://invece.org >>> >>> Beauty is more important in computing than anywhere else in technology >>> because software is so complicated. Beauty is the ultimate defence >>> against complexity. >>> — David Gelernter >> _______________________________________________ >> jemalloc-discuss mailing list >> [email protected] >> http://www.canonware.com/mailman/listinfo/jemalloc-discuss _______________________________________________ jemalloc-discuss mailing list [email protected] http://www.canonware.com/mailman/listinfo/jemalloc-discuss
