On 2/14/14, 6:53 PM, Jason Evans wrote:
On Feb 14, 2014, at 1:52 PM, Evan Wies<e...@neomantra.net> wrote:
Is it possible to allocate memory in a defined region, specifically anywhere
above the first 2GB?
I see that mallocx with ALLOCM_ALIGN allows one to specify a base address
aligned to a power of 2. Can I grab a bunch of memory that way and somehow
tell an arena to use that space and then always mallocx with that arena?
I'm in the midst of writing a LuaJIT [1] binding to jemalloc, and have
successfully allocated memory aligned to 2^31 but can't figure out the next
step -- short of writing my own allocator to dole out jemalloc-allocated memory.
Context:
LuaJIT, a Lua Just-In-Time compiler, can itself only allocate memory in the lower portions of
memory (<1GB, <2GB, <4GB depending on platform). One can get around this limitation by
using its FFI to allocate memory for FFI-bound C structures, however allocating these in the lower
regions effectively "steals" available memory from the LuaJIT VM for Lua objects. I
want to ensure maximum available memory for the LuaJIT by allocating all my objects outside this
area.
By default jemalloc prefers mmap() over sbrk() when allocating memory, which
for 64-bit systems means that it won't allocate the low memory LuaJIT wants
until mmap() fails. On 32-bit systems, I think there are non-trivial
differences between operating systems in terms of where mmap() will place
anonymous mappings; IIRC FreeBSD uses a fixed (but tunable) amount of memory
for sbrk(), whereas Linux allows sbrk() and mmap() to collide (but I don't
remember where code gets mapped, so the practical result may be similar to
FreeBSD). I suspect that jemalloc's preference for mmap() will be sufficient
for all but the most memory-hungry Lua applications, even on 32-bit systems.
Take a look at where jemalloc is actually allocating memory from; it's probably
above where code is mapped.
If you want to get hacky, you can build jemalloc with munmap() usage disabled,
temporarily sbrk() all the low memory LuaJIT could possibly want, then
temporarily allocate all the malloc()ed memory you could possibly want, then
release both. The result will be that jemalloc has a pool of cached chunks
that can't have come from low memory.
Jason
Jason,
Thanks for this prompt reply. I did as you suggested and the default
behavior works well, especially on Linux. I put my research up here:
https://gist.github.com/neomantra/9122165
Here's the LuaJIT binding to jemalloc:
https://github.com/neomantra/luajit-jemalloc
I've switched most of my allocations to jemalloc and have relieved
myself of most out-of-memory errors (for LuaJIT) and improved (shorter)
run times. I'm pretty psyched about this :)
Regards,
Evan
_______________________________________________
jemalloc-discuss mailing list
jemalloc-discuss@canonware.com
http://www.canonware.com/mailman/listinfo/jemalloc-discuss
