Re: Use of chunksize before initialization
On Mon, Mar 23, 2015 at 01:50:26PM +0200, Ivan A. Kosarev wrote: > > On 03/21/2015 11:31 PM, Konstantin Belousov wrote: > > On Sat, Mar 21, 2015 at 11:20:26AM +0200, Ivan A. Kosarev wrote: > >> On 03/21/2015 03:02 AM, Konstantin Belousov wrote: > >>> On Fri, Mar 20, 2015 at 03:59:52PM +0200, Ivan A. Kosarev wrote: > #12 0x0008011b428d in malloc_init_hard () at jemalloc_jemalloc.c:698 > #13 malloc_init () at jemalloc_jemalloc.c:296 > #14 0x000801243ea2 in ?? () from /lib/libc.so.7 > #15 0x0008006a5400 in ?? () > #16 0x00080089e5b0 in ?? () from /libexec/ld-elf.so.1 > #17 0x7fffe0b0 in ?? () > #18 0x000801139d06 in _init () from /lib/libc.so.7 > #19 0x7fffe0b0 in ?? () > >>> The backtrace is strange. Did you compiled malloc with the debugging > >>> symbols, while keep rest of libc without -g ? > >> I've just added the -g flag to CC_FLAGS in the Makefile and made sure to > >> install an unstripped version of the .so . I could investigate more on > >> why the early calls omit debug symbols, if it does any matter. > > I want to understand at what stage of the initialization the access happens. > > This is why I want to see the complete backtrace. > > It is jemalloc_constructor() that calls malloc_init(), so it should be > called directly by the loader. Do you mean rtld by loader ? Dynamic linker does not explicitely call into libc. The possible situations when such call occurs, is either execution of the initializers, or calls into the libthr-provided rtld locks, where libthr itself calls malloc to allocate the lock's backing store. The appearance of _init on the unparsed stack is indicative, but not conclusive, since gdb shows the nearest dynamic symbol, which could be _init just by chance. I need to know exact sequence of events causing the problem. Or, in other words, I cannot debug by retelling. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Use of chunksize before initialization
On 03/21/2015 11:31 PM, Konstantin Belousov wrote: On Sat, Mar 21, 2015 at 11:20:26AM +0200, Ivan A. Kosarev wrote: On 03/21/2015 03:02 AM, Konstantin Belousov wrote: On Fri, Mar 20, 2015 at 03:59:52PM +0200, Ivan A. Kosarev wrote: #12 0x0008011b428d in malloc_init_hard () at jemalloc_jemalloc.c:698 #13 malloc_init () at jemalloc_jemalloc.c:296 #14 0x000801243ea2 in ?? () from /lib/libc.so.7 #15 0x0008006a5400 in ?? () #16 0x00080089e5b0 in ?? () from /libexec/ld-elf.so.1 #17 0x7fffe0b0 in ?? () #18 0x000801139d06 in _init () from /lib/libc.so.7 #19 0x7fffe0b0 in ?? () The backtrace is strange. Did you compiled malloc with the debugging symbols, while keep rest of libc without -g ? I've just added the -g flag to CC_FLAGS in the Makefile and made sure to install an unstripped version of the .so . I could investigate more on why the early calls omit debug symbols, if it does any matter. I want to understand at what stage of the initialization the access happens. This is why I want to see the complete backtrace. It is jemalloc_constructor() that calls malloc_init(), so it should be called directly by the loader. -- ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Use of chunksize before initialization
On Sat, Mar 21, 2015 at 11:20:26AM +0200, Ivan A. Kosarev wrote: > > On 03/21/2015 03:02 AM, Konstantin Belousov wrote: > > On Fri, Mar 20, 2015 at 03:59:52PM +0200, Ivan A. Kosarev wrote: > >> #12 0x0008011b428d in malloc_init_hard () at jemalloc_jemalloc.c:698 > >> #13 malloc_init () at jemalloc_jemalloc.c:296 > >> #14 0x000801243ea2 in ?? () from /lib/libc.so.7 > >> #15 0x0008006a5400 in ?? () > >> #16 0x00080089e5b0 in ?? () from /libexec/ld-elf.so.1 > >> #17 0x7fffe0b0 in ?? () > >> #18 0x000801139d06 in _init () from /lib/libc.so.7 > >> #19 0x7fffe0b0 in ?? () > > The backtrace is strange. Did you compiled malloc with the debugging > > symbols, while keep rest of libc without -g ? > > I've just added the -g flag to CC_FLAGS in the Makefile and made sure to > install an unstripped version of the .so . I could investigate more on > why the early calls omit debug symbols, if it does any matter. I want to understand at what stage of the initialization the access happens. This is why I want to see the complete backtrace. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Use of chunksize before initialization
On 03/21/2015 03:02 AM, Konstantin Belousov wrote: On Fri, Mar 20, 2015 at 03:59:52PM +0200, Ivan A. Kosarev wrote: #12 0x0008011b428d in malloc_init_hard () at jemalloc_jemalloc.c:698 #13 malloc_init () at jemalloc_jemalloc.c:296 #14 0x000801243ea2 in ?? () from /lib/libc.so.7 #15 0x0008006a5400 in ?? () #16 0x00080089e5b0 in ?? () from /libexec/ld-elf.so.1 #17 0x7fffe0b0 in ?? () #18 0x000801139d06 in _init () from /lib/libc.so.7 #19 0x7fffe0b0 in ?? () The backtrace is strange. Did you compiled malloc with the debugging symbols, while keep rest of libc without -g ? I've just added the -g flag to CC_FLAGS in the Makefile and made sure to install an unstripped version of the .so . I could investigate more on why the early calls omit debug symbols, if it does any matter. Does it happen always, on only for the early initialization of the mutexes ? I'm not sure I understand the whole logic of the initialization process, but we could put a statement initializing the chunksize variable to 0 to the beginning of malloc_init_hard() and see if the assertion (or any other before it) fails. Since my suspicion is that the variable get random values at base_boot(), the presence of the failure depends on random factors. For a simple one-line program calling malloc() it is known to not to fail, of course. I should be able to to more tests on Mon. It might be related to r276630. Can you test on, say, 10.1 ? The Tsan tests mentioned below that cause mass (alignment != 0) failures are known to work fine on 10.1. : jemalloc_chunk.c:152: Failed assertion: "alignment != 0" Here's more of failures of this kind around: http://lab.llvm.org:8011/builders/sanitizer_x86_64-freebsd/builds/4758/steps/make-check-tsan/logs/stdio Can you please let me know if the analysis is correct and there's something to fix about initialization of the variable? Backtrace looks valid. Thanks. -- ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Use of chunksize before initialization
On Fri, Mar 20, 2015 at 03:59:52PM +0200, Ivan A. Kosarev wrote:
> Hello everybody,
>
> The malloc_init_hard() function defined in jemalloc_jemalloc.c, FreeBSD
> 11 r277486 reads:
>
> static bool
> malloc_init_hard(void)
> {
> ...
> if (base_boot()) {
> malloc_mutex_unlock(&init_lock);
> eturn (true);
> }
>
> if (chunk_boot()) {
> malloc_mutex_unlock(&init_lock);
> return (true);
> }
> ...
>
> The second call initializes the 'chunksize' global variable defined in
> jemalloc_chunk.c:
>
> bool
> chunk_boot(void)
> {
> /* Set variables according to the value of opt_lg_chunk. */
> chunksize = (ZU(1) << opt_lg_chunk);
> assert(chunksize >= PAGE);
> ...
>
> However, it seems the first call to base_boot() depends on that variable
> already:
>
> (gdb) bt
> #0 thr_kill () at thr_kill.S:3
> #1 0x000801241408 in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:51
> #2 0x0041d817 in __interceptor_raise () at
> /usr/home/ik/llvm/llvm.current/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:2097
> #3 0x00080123f969 in abort () at /usr/src/lib/libc/stdlib/abort.c:65
> #4 0x0041c5d9 in __interceptor_abort () at
> /usr/home/ik/llvm/llvm.current/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1851
> #5 0x0008011a8d64 in __je_chunk_alloc (size=,
> alignment=, base=, zero=,
> dss_prec=dss_prec_disabled) at jemalloc_chunk.c:150
> #6 0x0008011a9bfc in base_pages_alloc (minsize=128) at
> jemalloc_base.c:35
> #7 __je_base_alloc (size=) at jemalloc_base.c:57
> #8 0x0008011a9c96 in __je_base_calloc (number=,
> size=6) at jemalloc_base.c:74
> #9 0x0008008ae548 in mutex_init (calloc_cb=0x0, mutex= out>, mutex_attr=) at
> /usr/src/lib/libthr/thread/thr_mutex.c:145
> #10 _pthread_mutex_init_calloc_cb (mutex=0x801487c90, calloc_cb=0x0) at
> /usr/src/lib/libthr/thread/thr_mutex.c:229
> #11 0x0008011a18da in __je_malloc_mutex_init (mutex=0x18744) at
> jemalloc_mutex.c:97
> #12 0x0008011b428d in malloc_init_hard () at jemalloc_jemalloc.c:698
> #13 malloc_init () at jemalloc_jemalloc.c:296
> #14 0x000801243ea2 in ?? () from /lib/libc.so.7
> #15 0x0008006a5400 in ?? ()
> #16 0x00080089e5b0 in ?? () from /libexec/ld-elf.so.1
> #17 0x7fffe0b0 in ?? ()
> #18 0x000801139d06 in _init () from /lib/libc.so.7
> #19 0x7fffe0b0 in ?? ()
The backtrace is strange. Did you compiled malloc with the debugging
symbols, while keep rest of libc without -g ?
Does it happen always, on only for the early initialization of the
mutexes ? It might be related to r276630. Can you test on, say, 10.1 ?
>
> Note that base_pages() calls chunk_alloc() with chucksize used as the
> alignment value:
>
> static bool
> base_pages_alloc(size_t minsize)
> {
> ...
> base_pages = chunk_alloc(csize, chunksize, true, &zero,
> chunk_dss_prec_get());
> ...
>
> and the latter tests it against zero:
>
> void *
> chunk_alloc(size_t size, size_t alignment, bool base, bool *zero,
> dss_prec_t dss_prec)
> {
> ...
> assert(alignment != 0);
>
>
> so we sometimes we end up with:
>
> : jemalloc_chunk.c:152: Failed assertion: "alignment != 0"
>
> Here's more of failures of this kind around:
>
> http://lab.llvm.org:8011/builders/sanitizer_x86_64-freebsd/builds/4758/steps/make-check-tsan/logs/stdio
>
> Can you please let me know if the analysis is correct and there's
> something to fix about initialization of the variable?
>
Backtrace looks valid.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Use of chunksize before initialization
Hello everybody,
The malloc_init_hard() function defined in jemalloc_jemalloc.c, FreeBSD
11 r277486 reads:
static bool
malloc_init_hard(void)
{
...
if (base_boot()) {
malloc_mutex_unlock(&init_lock);
eturn (true);
}
if (chunk_boot()) {
malloc_mutex_unlock(&init_lock);
return (true);
}
...
The second call initializes the 'chunksize' global variable defined in
jemalloc_chunk.c:
bool
chunk_boot(void)
{
/* Set variables according to the value of opt_lg_chunk. */
chunksize = (ZU(1) << opt_lg_chunk);
assert(chunksize >= PAGE);
...
However, it seems the first call to base_boot() depends on that variable
already:
(gdb) bt
#0 thr_kill () at thr_kill.S:3
#1 0x000801241408 in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:51
#2 0x0041d817 in __interceptor_raise () at
/usr/home/ik/llvm/llvm.current/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:2097
#3 0x00080123f969 in abort () at /usr/src/lib/libc/stdlib/abort.c:65
#4 0x0041c5d9 in __interceptor_abort () at
/usr/home/ik/llvm/llvm.current/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1851
#5 0x0008011a8d64 in __je_chunk_alloc (size=,
alignment=, base=, zero=,
dss_prec=dss_prec_disabled) at jemalloc_chunk.c:150
#6 0x0008011a9bfc in base_pages_alloc (minsize=128) at
jemalloc_base.c:35
#7 __je_base_alloc (size=) at jemalloc_base.c:57
#8 0x0008011a9c96 in __je_base_calloc (number=,
size=6) at jemalloc_base.c:74
#9 0x0008008ae548 in mutex_init (calloc_cb=0x0, mutex=out>, mutex_attr=) at
/usr/src/lib/libthr/thread/thr_mutex.c:145
#10 _pthread_mutex_init_calloc_cb (mutex=0x801487c90, calloc_cb=0x0) at
/usr/src/lib/libthr/thread/thr_mutex.c:229
#11 0x0008011a18da in __je_malloc_mutex_init (mutex=0x18744) at
jemalloc_mutex.c:97
#12 0x0008011b428d in malloc_init_hard () at jemalloc_jemalloc.c:698
#13 malloc_init () at jemalloc_jemalloc.c:296
#14 0x000801243ea2 in ?? () from /lib/libc.so.7
#15 0x0008006a5400 in ?? ()
#16 0x00080089e5b0 in ?? () from /libexec/ld-elf.so.1
#17 0x7fffe0b0 in ?? ()
#18 0x000801139d06 in _init () from /lib/libc.so.7
#19 0x7fffe0b0 in ?? ()
Note that base_pages() calls chunk_alloc() with chucksize used as the
alignment value:
static bool
base_pages_alloc(size_t minsize)
{
...
base_pages = chunk_alloc(csize, chunksize, true, &zero,
chunk_dss_prec_get());
...
and the latter tests it against zero:
void *
chunk_alloc(size_t size, size_t alignment, bool base, bool *zero,
dss_prec_t dss_prec)
{
...
assert(alignment != 0);
so we sometimes we end up with:
: jemalloc_chunk.c:152: Failed assertion: "alignment != 0"
Here's more of failures of this kind around:
http://lab.llvm.org:8011/builders/sanitizer_x86_64-freebsd/builds/4758/steps/make-check-tsan/logs/stdio
Can you please let me know if the analysis is correct and there's
something to fix about initialization of the variable?
Thanks.
--
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
