westonpace opened a new pull request #10240: URL: https://github.com/apache/arrow/pull/10240
Currently Arrow builds that enable jemalloc also enable background threads. There is no way to disable this without recompiling arrow. * MALLOC_CONF and /etc/malloc.conf are ignored as they are lower precedence than global variable initialization * Disabling dirty page purging via `jemalloc_set_decay_ms` does not terminate the background threads * Switching to a different allocator (e.g. python's `set_default_memory_pool`) does not terminate the background threads. This PR adds a capability to do so. In a future PR it may be worth looking into some mechanism to prevent the threads from starting at all (e.g. migrate away from global variable initialization) but there is no good lifecycle hook for this and adding a run-once initial configuration step when the allocator is first accessed adds complexity and potential race cases. * As best I can tell this method should be thread safe but I can't find any good jemalloc documentation on this so I just left it vague. I figure most users will be shutting down the background threads before they start using Arrow anyways. * I confirmed that shutting down the background threads did indeed shutdown the threads and that the valgrind potential leak goes away (FYI, the leak is a valgrind error as valgrind is not properly waiting for detached threads to shut down, more on that here: https://bugs.kde.org/show_bug.cgi?id=415141) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
