Jason, thank you for explanation! We linked libquickfix against jemalloc and it helped because of changed initialization order.
On Tue, Jan 21, 2014 at 10:12 PM, Jason Evans <[email protected]> wrote: > On Jan 20, 2014, at 1:03 AM, Evgeniy Ivanov <[email protected]> wrote: >> I got a following deadlock: >> >> #0 __lll_lock_wait_private () at >> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 >> #1 0x00007f6f72cc048f in _L_lock_16 () from >> /home/eiva/prefix/glibc-2.16/lib/libc.so.6 >> #2 0x00007f6f72cc02d1 in __new_exitfn (listp=0x7f6f73030c80 <lock>) >> at cxa_atexit.c:78 >> #3 0x00007f6f72cc0424 in __internal_atexit (func=0x7f6f7371aad0 >> <prof_fdump>, arg=0x0, d=0x7f6f73932570, listp=<optimized out>) at >> cxa_atexit.c:34 >> #4 0x00007f6f7371a9d7 in prof_boot2 () at >> /export/home/eiva/git/thd/src/thirdparty/jemalloc/3.4.1/src/src/prof.c:1212 >> #5 0x00007f6f736f3c4e in malloc_init_hard () at >> /export/home/eiva/git/thd/src/thirdparty/jemalloc/3.4.1/src/src/jemalloc.c:790 >> #6 0x00007f6f736effc4 in malloc_init () at >> /export/home/eiva/git/thd/src/thirdparty/jemalloc/3.4.1/src/src/jemalloc.c:306 >> #7 0x00007f6f736f0698 in calloc (num=1, size=1040) at >> /export/home/eiva/git/thd/src/thirdparty/jemalloc/3.4.1/src/src/jemalloc.c:1036 >> #8 0x00007f6f72cc03ba in __new_exitfn (listp=0x7f6f7302f5a8 >> <__exit_funcs>) at cxa_atexit.c:100 >> #9 0x00007f6f72cc0424 in __internal_atexit (func=0x3972062ae0 >> <std::ios_base::Init::~Init()>, arg=0x7f6f736ea9c8 <_ZStL8__ioinit>, >> d=0x7f6f736df170, listp=<optimized out>) at cxa_atexit.c:34 >> #10 0x00007f6f7342336e in __cxx_global_var_init () at >> /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/iostream:72 >> #11 0x00007f6f73423389 in global constructors keyed to a() () at >> /tb/builds/thd/sbn/2.6/src/thirdparty/quickfix/1.12.4/src/src/C++/SocketInitiator.cpp:54 >> #12 0x00007f6f7348aa36 in __do_global_ctors_aux () from >> //home/eiva/git/tb/build.x86_64-unknown-linux/platform/lib64/libquickfix.so.10 >> #13 0x00007f6f733cdb43 in _init () from >> //home/eiva/git/tb/build.x86_64-unknown-linux/platform/lib64/libquickfix.so.10 >> #14 0x00007ffff9915058 in ?? () >> #15 0x00007f6f73953849 in call_init (l=0x7f6f73b65540, >> argc=1936584824, argv=0x7ffff9915048, env=0x7ffff9915058) at >> dl-init.c:69 >> #16 0x00007f6f73953987 in _dl_init (main_map=0x7f6f73b691c8, argc=1, >> argv=0x7ffff9915048, env=0x7ffff9915058) at dl-init.c:133 >> #17 0x00007f6f73945b2a in _dl_start_user () from >> /home/eiva/prefix/glibc-2.16/lib/ld-2.16.so >> #18 0x0000000000000001 in ?? () >> #19 0x00007ffff9917072 in ?? () >> #20 0x0000000000000000 in ?? () >> >> It is always reproducible with libquickfix-1.12.4 linked against my >> application (libc-2.12 and libc-2.16). >> MALLOC_CONF="prof:true" >> >> Is it issue in jemalloc or in glibc? > > libquickfix is registering an atexit() function inside a library initializer, > prior to any other allocation having triggered jemalloc initialization. > During jemalloc initialization, there are two potential atexit() calls, one > for heap profiling and one for printing stats at exit. The deadlock here is > occurring due to glibc's atexit() code holding a lock during the calloc() > call that triggers jemalloc initialization, and jemalloc trying to call back > into the atexit() code. > > There's little jemalloc can do about this except to avoid atexit(). However, > you may still be able to use heap profiling in conjunction with libquickfix. > I vaguely recall that it is possible to specify precedences for library > initializers; you can probably free(malloc(1)) inside a high priority library > initializer inside your own application code and cause jemalloc to be > initialized prior to libquickfix, in which case the recursive atexit() > deadlock will be avoided. > > Jason -- Cheers, Evgeniy _______________________________________________ jemalloc-discuss mailing list [email protected] http://www.canonware.com/mailman/listinfo/jemalloc-discuss
