I'm using ARROW_DEFAULT_MEMORY_POOL=system
Based on a review of memory_pool.cc I expect this to become posix_memalign
calls on Linux
When I call posiix_memalign in a .so that I created and linked with my app,
using LD_PRELOAD=/usr/local/lib/libmimalloc.so to run the app, these calls
get forwarded to mi_posix_memalign (because I threw a prinf in there and
re-built mimalloc)... note, I'm not talking about Arrow's built-in mimalloc.
Maybe Arrow's mimalloc is keeping the LD_PRELOAD of my custom mimalloc from
taking effect? How is mimalloc included in Arrow? When I
call arrow::mimalloc_memory_pool() I do get an Ok status, so it is in the
build I'm using from `apt`
-John
On Tue, Jun 14, 2022 at 10:37 AM Weston Pace <weston.p...@gmail.com> wrote:
> Sorry, that should have said "when Arrow builds jemalloc". Here is
> the command we send down (from ThirdPartyToolchain.cmake):
>
> ```
> JEMALLOC_CONFIGURE_COMMAND
> "--prefix=${JEMALLOC_PREFIX}"
> "--libdir=${JEMALLOC_LIB_DIR}"
> "--with-jemalloc-prefix=je_arrow_"
> "--with-private-namespace=je_arrow_private_"
> "--without-export"
> "--disable-shared"
> # Don't override operator new()
> "--disable-cxx"
> "--disable-libdl"
> # See https://github.com/jemalloc/jemalloc/issues/1237
> "--disable-initial-exec-tls"
> ${EP_LOG_OPTIONS})
> list(APPEND
> ```
>
> On Tue, Jun 14, 2022 at 5:35 AM Weston Pace <weston.p...@gmail.com> wrote:
> >
> > I can try and give a more detailed answer later in the week but the
> > gist of it is that Arrow manages all "buffer allocations" with a
> > memory pool. These are the allocations for the actual data in the
> > arrays. These are the allocations that use the memory pool configured
> > by ARROW_DEFAULT_MEMORY_POOL.
> >
> > To avoid interfering with the user's allocations Arrow does not
> > configure the system allocator at all. So when Arrow builds it alters
> > it slightly (using cmake variables I think) to be specific to Arrow.
> > This might make it a bit tricky to get debug symbols for jemalloc but
> > you could always build Arrow in debug mode and intercept the methods
> > in memory_pool.cc if your focus is tracking allocations.
> >
> > Arrow still uses the system allocator for all non-buffer allocations.
> > So, for example, when reading in a large IPC file, the majority of the
> > data will be allocated by Arrow's memory pool. However, the schema,
> > and the wrapper array object itself will be allocated by the system
> > allocator. This is probably why switching the system allocator to
> > jemalloc shows some, but not all, Arrow allocations happening there.
> >
> > On Tue, Jun 14, 2022 at 5:28 AM John Muehlhausen <j...@jgm.org> wrote:
> > >
> > > A code review has demonstrated that Arrow uses posix_memalign ... I do
> > > believe mimalloc preload is "catching" this but I didn't tool it with
> my
> > > customization. Still interested in any guidance on the other points
> > > raised, and sorry for some of this being noise.
> > >
> > > -John
> > >
> > > On Tue, Jun 14, 2022 at 9:06 AM John Muehlhausen <j...@jgm.org> wrote:
> > >
> > > > Hello,
> > > >
> > > > This comment is regarding installation with `apt` on ubuntu 18.04 ...
> > > > `libarrow-dev/bionic,now 8.0.0-1 amd64`
> > > >
> > > > I'm a bit confused about the memory pool situation:
> > > >
> > > > * I run with `ARROW_DEFAULT_MEMORY_POOL=system` and check that
> > > > `arrow::default_memory_pool()->backend_name() ==
> > > > arrow::system_memory_pool()->backend_name()`
> > > >
> > > > * I then LD_PRELOAD a customized (*) mimalloc according to the
> directions
> > > > at the mimalloc git repo and things like `strm->Reset(INT32_MAX);`
> seem not
> > > > to be hitting it... I figured that is a big enough chunk to jostle
> it into
> > > > doing something... `BufferOutputStream::Create(INT32_MAX)` is also
> not
> > > > intercepted by mimalloc. Is the "system" pool somehow going around
> the
> > > > typical allocation interfaces on linux? I built my own .so and
> linked it
> > > > to the app and malloc() is getting intercepted.
> > > >
> > > > * `arrow::mimalloc_memory_pool(&mmmp);` does return something... but
> > > > apparently not "my" mimalloc ... statically linked?
> > > >
> > > > * what is going on in Arrow with constructor (pre-main())
> allocations?
> > > > Some of this does hit my LD_PRELOADed mimalloc
> > > >
> > > > * any way to get symbols for the apt-installed libs or would I need
> to
> > > > build from source to get backtrace with symbols? (for chasing down
> sources
> > > > of allocations)
> > > >
> > > > * what is the C++ lib equivalent of the following from the Python
> code? I
> > > > figure I could stop trying to understand the built-in/default
> allocators if
> > > > I could just replace them... but this may also intersect with my
> question
> > > > about constructors. Maybe I'd have to make sure my constructor runs
> first
> > > > to perform the switch-a-roo before anything else tries to use the
> default
> > > > pool?
> > > >
> > > > ```
> > > > namespace py {
> > > >
> > > > static std::mutex memory_pool_mutex;
> > > > static MemoryPool* default_python_pool = nullptr;
> > > >
> > > > void set_default_memory_pool(MemoryPool* pool) {
> > > > std::lock_guard<std::mutex> guard(memory_pool_mutex);
> > > > default_python_pool = pool;
> > > > }
> > > > ```
> > > >
> > > >
> > > > (*) the mimalloc customization: the main app has a weak reference
> that
> > > > ends up defined by the LD_PRELOAD mimalloc, where the function
> so-supplied
> > > > allows the app to install a function pointer (back to the main app)
> that
> > > > gets called (if defined) at various interesting points in mimalloc
> > > >
> > > >
> > > > Thanks,
> > > > John
> > > >
>
