On 31.01.2020 22:10, Andrew Doran wrote:
> On Fri, Jan 31, 2020 at 06:55:00PM -0000, Christos Zoulas wrote:
>> In article <724af477-010b-9ddf-6ece-e23d7cf59...@gmx.com>,
>> Kamil Rytarowski  <n...@gmx.com> wrote:
>>> -=-=-=-=-=-
>>> -=-=-=-=-=-
>>> On 31.01.2020 03:38, Christos Zoulas wrote:
>>>> And it is fixed now.
>>>> christos
>>> OK. I am going to submit a bug report upstream and get some feedback
>>> what is the way forward here, delaying initialization.
>> I think that the way forward (on our side) is to do away with libpthread,
>> merge it with libc and kill all the stub nonsense.
> Agreed.
> pthread__init() does some expensive stuff like _lwp_ctl().  I think we can
> safely & without hacks defer a lot of that till the first pthread_create().
> Andrew

This libc-libpthread split/merge is a red herring.

The problem here is with a mutual dependencies between POSIX threads
library and malloc library.

I did some investigation and here are my findings:

1. jemalloc abuses initialization and initializes self very early, with
a constructor:

 * If an application creates a thread before doing any allocation in the
 * thread, then calls fork(2) in the main thread followed by memory
 * in the child process, a race can occur that results in deadlock
within the
 * child: the main thread may have forked while the created thread had
 * partially initialized the allocator.  Ordinarily jemalloc prevents
 * fork/malloc races via the following functions it registers during
 * initialization using pthread_atfork(), but of course that does no good if
 * the allocator isn't fully initialized at fork time.  The following
 * constructor is a partial solution to this problem.  It may still be
 * to trigger the deadlock described above, but doing so would involve
 * via a library constructor that runs before jemalloc's runs.
static void

jemalloc_constructor(void) {

Relevant commit:

commit 20f1fc95adb35ea63dc61f47f2b0ffbd37d39f32
Author: Jason Evans <j...@fb.com>
Date:   Tue Oct 9 14:46:22 2012 -0700

    Fix fork(2)-related deadlocks.

    Add a library constructor for jemalloc that initializes the allocator.
    This fixes a race that could occur if threads were created by the main
    thread prior to any memory allocation, followed by fork(2), and then
    memory allocation in the child process.

    Fix the prefork/postfork functions to acquire/release the ctl, prof, and
    rtree mutexes.  This fixes various fork() child process deadlocks, but
    one possible deadlock remains (intentionally) unaddressed: prof
    backtracing can acquire runtime library mutexes, so deadlock is still
    possible if heap profiling is enabled during fork().  This deadlock is
    known to be a real issue in at least the case of libgcc-based

    Reported by tfengjun.

2. FreeBSD added a hack and an internal pthread_mutex_init() version
called: _pthread_mutex_init_calloc_cb().. it passes a callback pointer
to jemalloc's tiny calloc().

This is very ugly and I consider it as the wrong way of boostraping malloc.

3. There is a problem inside libpthread. It as designed to not malloc()
early to not trigger malloc initialization, however it was broken as it
calls at_fork functions:

#0  malloc (size=size@entry=16) at
#1  0x0000776b71d71a44 in af_alloc () at
#2  af_alloc () at /usr/src/lib/libc/gen/pthread_atfork.c:74
#3  _pthread_atfork (prepare=prepare@entry=0x0, parent=parent@entry=0x0,
    child=child@entry=0x776b7220b3e5 <pthread__fork_callback>)
    at /usr/src/lib/libc/gen/pthread_atfork.c:121
#4  0x0000776b7220cacd in pthread__init () at
#5  0x0000776b71d7a585 in _libc_init () at

These at_fork routines caused also issues with false-positives in Leak
Sanitizer. I had to pacify the sanitizer and disable tracking of its

This patch removes '#if 0' hack from src/lib/libpthread and switches
at_fork to mmap()+munmap().


This test disabled the constructor hack:


With these changes everything seems to work.

In order to avoid the FreeBSD specific hack with the constructor and
initialize jemalloc always during libc bootstrap I propose the following

 - add __libc_malloc_init() call in _libc_init()
 - redirect __libc_malloc_init() to jemalloc__init() with jemalloc
 - otherwise redirect to an empty stub

Here is a patch that does everything and works fine for me.


There are no longer jemalloc calls before being ready and jemalloc is
still initialized always but in its proper time.

(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /tmp/a.out

Breakpoint 2, jemalloc__init () at
3209            malloc_init();
(gdb) c

Breakpoint 3, malloc_init_hard () at
1533    malloc_init_hard(void) {
(gdb) bt
#0  malloc_init_hard () at
#1  0x00007f7ff763f9c4 in ?? () from /usr/lib/libc.so.12
#2  0x00007f7ff7ef9400 in ?? ()
#3  0x00007f7ff763ac99 in _init () from /usr/lib/libc.so.12
#4  0x0000000000000000 in ?? ()

Attachment: signature.asc
Description: OpenPGP digital signature

Reply via email to