As of jemalloc 5, jemalloc default build can not be used in libraries that are dlopened. However, libmxnet.so is dlopened by Python (ctypes). To use MXNet with jemalloc 5, users must not link to system libjemalloc.so but must rather link to a libjemalloc compiled with special parameters to allow dlopen to work. See https://github.com/jemalloc/jemalloc/issues/937
jemalloc 5 is distributed as part of Ubuntu 18.10 and higher, as well as Debian Stable. Users on these systems will be unable to use MXNet after compiling with USE_JEMALLOC=ON on systems with libjemalloc-dev, which is the default setting. Thus in https://github.com/apache/incubator-mxnet/pull/17324 I suggest to disable Jemalloc by default in the source build. Auto-detecting the version of jemalloc is not helpful, because over time less and less systems will come with a working version of jemalloc. Please go ahead and approve the PR if you agree. The better solution is to build jemalloc as part of our build. See https://github.com/apache/incubator-mxnet/pull/17121 But as CMake build of jemalloc is not yet integrated upstream, this approach currently relies on a development branch of jemalloc. Thus it was suggested to revisit the approach once CMake integration is stable upstream.
