Re: [Nfs-ganesha-devel] crash in jemalloc leading to a deadlock.

2017-11-16 Thread Malahal Naineni 
Yes. we should handle postrotate signal.

On Wed, Nov 15, 2017 at 11:53 PM, Frank Filz <ffilz...@mindspring.com>
wrote:

> If we keep the logging fd open, then we need a signal to tell it that
> logrotate has occurred…
>
>
>
> Frank
>
>
>
> *From:* Malahal Naineni [mailto:mala...@gmail.com]
> *Sent:* Tuesday, November 14, 2017 9:44 PM
> *To:* Frank Filz <ffilz...@mindspring.com>
> *Cc:* d...@redhat.com; nfs-ganesha-devel@lists.sourceforge.net
>
> *Subject:* Re: [Nfs-ganesha-devel] crash in jemalloc leading to a
> deadlock.
>
>
>
> Silly glibc, they should have provided backtrace_symbols_func() to take a
> callback function. Using backtrace_symbols_func() is a bit harder for us
> here. Here is a thought:
>
>
>
> 1. Our logger should NOT open/re-open for every message. It should just
> open once.
>
> 2. Our logger should have an interface to provide such fd using API or
> global symbol.
>
> 3. There are some tricks to get "fd" with syslog tracing as well
>
> 4. Then pass such fd from step 2 to backtrace_symbols_fd()
>
>
>
> Step1 might be optional as open() should mostly succeed though.
>
>
>
>
>
> On Thu, Nov 9, 2017 at 7:53 PM, Frank Filz <ffilz...@mindspring.com>
> wrote:
>
> That might be a good solution, though what fd would we use? Can we safely
> open an fd during a sighandler?
>
>
>
> Frank
>
>
>
> *From:* Malahal Naineni [mailto:mala...@gmail.com]
> *Sent:* Wednesday, November 8, 2017 11:51 PM
> *To:* d...@redhat.com
> *Cc:* nfs-ganesha-devel@lists.sourceforge.net
> *Subject:* Re: [Nfs-ganesha-devel] crash in jemalloc leading to a
> deadlock.
>
>
>
> backtrace_symbols_fd() takes the same buffer and size arguments as
> backtrace_symbols(), but instead of returning an array of strings to the
> caller, it writes the strings, one per line, to the file descriptor fd.
> backtrace_symbols_fd() does not call malloc(3), and so can be employed in
> situations where the latter function might fail.
>
>
>
> On Thu, Nov 9, 2017 at 12:24 AM, Daniel Gryniewicz <d...@redhat.com>
> wrote:
>
> Allocating in a backtrace seems like a very bad idea.  If there's ever a
> crash during an allocation, it is guaranteed to deadlock.
>
> Daniel
>
>
>
> On 11/08/2017 01:43 PM, Pradeep wrote:
>
> I'm using Ganesha 2.6 dev.12 with jemalloc-3.6.0 and hitting a case
> where jemalloc seem to be holding a lock and crashing. In Ganesha's
> gsh_backtrace(), we try to allocate memory and that hangs (ended up in
> deadlock). Have you seen this before? Perhaps it is a good idea not to
> allocate memory in backtrace path?
>
>
> #0 0x7f49b51ff1bd in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x7f49b51fad02 in _L_lock_791 () from /lib64/libpthread.so.0
> #2 0x7f49b51fac08 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x7f49b65d12dc in arena_bin_malloc_hard () from
> /lib64/libjemalloc.so.1
> #4 0x7f49b65d1516 in je_arena_tcache_fill_small () from
> /lib64/libjemalloc.so.1
> #5 0x7f49b65ea6ff in je_tcache_alloc_small_hard () from
> /lib64/libjemalloc.so.1
> #6 0x7f49b65ca14f in malloc () from /lib64/libjemalloc.so.1
> #7 0x7f49b6c5a785 in _dl_scope_free () from /lib64/ld-linux-x86-64.so.2
> #8 0x7f49b6c55841 in _dl_map_object_deps () from
> /lib64/ld-linux-x86-64.so.2
> #9 0x7f49b6c5ba4b in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
> #10 0x7f49b6c57364 in _dl_catch_error () from
> /lib64/ld-linux-x86-64.so.2
> #11 0x7f49b6c5b35b in _dl_open () from /lib64/ld-linux-x86-64.so.2
> #12 0x7f49b48f5ff2 in do_dlopen () from /lib64/libc.so.6
> #13 0x7f49b6c57364 in _dl_catch_error () from
> /lib64/ld-linux-x86-64.so.2
> #14 0x7f49b48f60b2 in __libc_dlopen_mode () from /lib64/libc.so.6
> #15 0x7f49b48cf595 in init () from /lib64/libc.so.6
> #16 0x7f49b51fdbb0 in pthread_once () from /lib64/libpthread.so.0
> #17 0x7f49b48cf6ac in backtrace () from /lib64/libc.so.6
> #18 0x0045193d in gsh_backtrace () at
> /usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:228
> #19 0x004519fe in crash_handler (signo=11,
> info=0x7f49b155db70, ctx=0x7f49b155da40) at
> /usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:244
> #20 
> #21 0x7f49b65d0c61 in arena_purge () from /lib64/libjemalloc.so.1
> #22 0x7f49b65d218d in je_arena_dalloc_large () from
> /lib64/libjemalloc.so.1
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.

Re: [Nfs-ganesha-devel] crash in jemalloc leading to a deadlock.

2017-11-15 Thread Frank Filz 
If we keep the logging fd open, then we need a signal to tell it that logrotate 
has occurred…



Frank



From: Malahal Naineni [mailto:mala...@gmail.com]
Sent: Tuesday, November 14, 2017 9:44 PM
To: Frank Filz <ffilz...@mindspring.com>
Cc: d...@redhat.com; nfs-ganesha-devel@lists.sourceforge.net
Subject: Re: [Nfs-ganesha-devel] crash in jemalloc leading to a deadlock.



Silly glibc, they should have provided backtrace_symbols_func() to take a 
callback function. Using backtrace_symbols_func() is a bit harder for us here. 
Here is a thought:



1. Our logger should NOT open/re-open for every message. It should just open 
once.

2. Our logger should have an interface to provide such fd using API or global 
symbol.

3. There are some tricks to get "fd" with syslog tracing as well

4. Then pass such fd from step 2 to backtrace_symbols_fd()



Step1 might be optional as open() should mostly succeed though.





On Thu, Nov 9, 2017 at 7:53 PM, Frank Filz <ffilz...@mindspring.com 
<mailto:ffilz...@mindspring.com> > wrote:

That might be a good solution, though what fd would we use? Can we safely open 
an fd during a sighandler?



Frank



From: Malahal Naineni [mailto:mala...@gmail.com <mailto:mala...@gmail.com> ]
Sent: Wednesday, November 8, 2017 11:51 PM
To: d...@redhat.com <mailto:d...@redhat.com>
Cc: nfs-ganesha-devel@lists.sourceforge.net 
<mailto:nfs-ganesha-devel@lists.sourceforge.net>
Subject: Re: [Nfs-ganesha-devel] crash in jemalloc leading to a deadlock.



backtrace_symbols_fd() takes the same buffer and size arguments as 
backtrace_symbols(), but instead of returning an array of strings to the 
caller, it writes the strings, one per line, to the file descriptor fd. 
backtrace_symbols_fd() does not call malloc(3), and so can be employed in 
situations where the latter function might fail.



On Thu, Nov 9, 2017 at 12:24 AM, Daniel Gryniewicz <d...@redhat.com 
<mailto:d...@redhat.com> > wrote:

Allocating in a backtrace seems like a very bad idea.  If there's ever a crash 
during an allocation, it is guaranteed to deadlock.

Daniel



On 11/08/2017 01:43 PM, Pradeep wrote:

I'm using Ganesha 2.6 dev.12 with jemalloc-3.6.0 and hitting a case
where jemalloc seem to be holding a lock and crashing. In Ganesha's
gsh_backtrace(), we try to allocate memory and that hangs (ended up in
deadlock). Have you seen this before? Perhaps it is a good idea not to
allocate memory in backtrace path?


#0 0x7f49b51ff1bd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x7f49b51fad02 in _L_lock_791 () from /lib64/libpthread.so.0
#2 0x7f49b51fac08 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x7f49b65d12dc in arena_bin_malloc_hard () from /lib64/libjemalloc.so.1
#4 0x7f49b65d1516 in je_arena_tcache_fill_small () from
/lib64/libjemalloc.so.1
#5 0x7f49b65ea6ff in je_tcache_alloc_small_hard () from
/lib64/libjemalloc.so.1
#6 0x7f49b65ca14f in malloc () from /lib64/libjemalloc.so.1
#7 0x7f49b6c5a785 in _dl_scope_free () from /lib64/ld-linux-x86-64.so.2
#8 0x7f49b6c55841 in _dl_map_object_deps () from /lib64/ld-linux-x86-64.so.2
#9 0x7f49b6c5ba4b in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#10 0x7f49b6c57364 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#11 0x7f49b6c5b35b in _dl_open () from /lib64/ld-linux-x86-64.so.2
#12 0x7f49b48f5ff2 in do_dlopen () from /lib64/libc.so.6
#13 0x7f49b6c57364 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#14 0x7f49b48f60b2 in __libc_dlopen_mode () from /lib64/libc.so.6
#15 0x7f49b48cf595 in init () from /lib64/libc.so.6
#16 0x7f49b51fdbb0 in pthread_once () from /lib64/libpthread.so.0
#17 0x7f49b48cf6ac in backtrace () from /lib64/libc.so.6
#18 0x0045193d in gsh_backtrace () at
/usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:228
#19 0x004519fe in crash_handler (signo=11,
info=0x7f49b155db70, ctx=0x7f49b155da40) at
/usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:244
#20 
#21 0x7f49b65d0c61 in arena_purge () from /lib64/libjemalloc.so.1
#22 0x7f49b65d218d in je_arena_dalloc_large () from /lib64/libjemalloc.so.1

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net 
<mailto:Nfs-ganesha-devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net 
<mailto:Nfs

Re: [Nfs-ganesha-devel] crash in jemalloc leading to a deadlock.

2017-11-14 Thread Malahal Naineni 
Silly glibc, they should have provided backtrace_symbols_func() to take a
callback function. Using backtrace_symbols_func() is a bit harder for us
here. Here is a thought:

1. Our logger should NOT open/re-open for every message. It should just
open once.
2. Our logger should have an interface to provide such fd using API or
global symbol.
3. There are some tricks to get "fd" with syslog tracing as well
4. Then pass such fd from step 2 to backtrace_symbols_fd()

Step1 might be optional as open() should mostly succeed though.


On Thu, Nov 9, 2017 at 7:53 PM, Frank Filz <ffilz...@mindspring.com> wrote:

> That might be a good solution, though what fd would we use? Can we safely
> open an fd during a sighandler?
>
>
>
> Frank
>
>
>
> *From:* Malahal Naineni [mailto:mala...@gmail.com]
> *Sent:* Wednesday, November 8, 2017 11:51 PM
> *To:* d...@redhat.com
> *Cc:* nfs-ganesha-devel@lists.sourceforge.net
> *Subject:* Re: [Nfs-ganesha-devel] crash in jemalloc leading to a
> deadlock.
>
>
>
> backtrace_symbols_fd() takes the same buffer and size arguments as
> backtrace_symbols(), but instead of returning an array of strings to the
> caller, it writes the strings, one per line, to the file descriptor fd.
> backtrace_symbols_fd() does not call malloc(3), and so can be employed in
> situations where the latter function might fail.
>
>
>
> On Thu, Nov 9, 2017 at 12:24 AM, Daniel Gryniewicz <d...@redhat.com>
> wrote:
>
> Allocating in a backtrace seems like a very bad idea.  If there's ever a
> crash during an allocation, it is guaranteed to deadlock.
>
> Daniel
>
>
>
> On 11/08/2017 01:43 PM, Pradeep wrote:
>
> I'm using Ganesha 2.6 dev.12 with jemalloc-3.6.0 and hitting a case
> where jemalloc seem to be holding a lock and crashing. In Ganesha's
> gsh_backtrace(), we try to allocate memory and that hangs (ended up in
> deadlock). Have you seen this before? Perhaps it is a good idea not to
> allocate memory in backtrace path?
>
>
> #0 0x7f49b51ff1bd in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x7f49b51fad02 in _L_lock_791 () from /lib64/libpthread.so.0
> #2 0x7f49b51fac08 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x7f49b65d12dc in arena_bin_malloc_hard () from
> /lib64/libjemalloc.so.1
> #4 0x7f49b65d1516 in je_arena_tcache_fill_small () from
> /lib64/libjemalloc.so.1
> #5 0x7f49b65ea6ff in je_tcache_alloc_small_hard () from
> /lib64/libjemalloc.so.1
> #6 0x7f49b65ca14f in malloc () from /lib64/libjemalloc.so.1
> #7 0x7f49b6c5a785 in _dl_scope_free () from /lib64/ld-linux-x86-64.so.2
> #8 0x7f49b6c55841 in _dl_map_object_deps () from
> /lib64/ld-linux-x86-64.so.2
> #9 0x7f49b6c5ba4b in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
> #10 0x7f49b6c57364 in _dl_catch_error () from
> /lib64/ld-linux-x86-64.so.2
> #11 0x7f49b6c5b35b in _dl_open () from /lib64/ld-linux-x86-64.so.2
> #12 0x7f49b48f5ff2 in do_dlopen () from /lib64/libc.so.6
> #13 0x7f49b6c57364 in _dl_catch_error () from
> /lib64/ld-linux-x86-64.so.2
> #14 0x7f49b48f60b2 in __libc_dlopen_mode () from /lib64/libc.so.6
> #15 0x7f49b48cf595 in init () from /lib64/libc.so.6
> #16 0x7f49b51fdbb0 in pthread_once () from /lib64/libpthread.so.0
> #17 0x7f49b48cf6ac in backtrace () from /lib64/libc.so.6
> #18 0x0045193d in gsh_backtrace () at
> /usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:228
> #19 0x004519fe in crash_handler (signo=11,
> info=0x7f49b155db70, ctx=0x7f49b155da40) at
> /usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:244
> #20 
> #21 0x7f49b65d0c61 in arena_purge () from /lib64/libjemalloc.so.1
> #22 0x7f49b65d218d in je_arena_dalloc_large () from
> /lib64/libjemalloc.so.1
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=email

Re: [Nfs-ganesha-devel] crash in jemalloc leading to a deadlock.

2017-11-08 Thread Malahal Naineni 
backtrace_symbols_fd() takes the same buffer and size arguments as
backtrace_symbols(), but instead of returning an array of strings to the
caller, it writes the strings, one per line, to the file descriptor fd.
backtrace_symbols_fd() does not call malloc(3), and so can be employed in
situations where the latter function might fail.

On Thu, Nov 9, 2017 at 12:24 AM, Daniel Gryniewicz  wrote:

> Allocating in a backtrace seems like a very bad idea.  If there's ever a
> crash during an allocation, it is guaranteed to deadlock.
>
> Daniel
>
>
> On 11/08/2017 01:43 PM, Pradeep wrote:
>
>> I'm using Ganesha 2.6 dev.12 with jemalloc-3.6.0 and hitting a case
>> where jemalloc seem to be holding a lock and crashing. In Ganesha's
>> gsh_backtrace(), we try to allocate memory and that hangs (ended up in
>> deadlock). Have you seen this before? Perhaps it is a good idea not to
>> allocate memory in backtrace path?
>>
>>
>> #0 0x7f49b51ff1bd in __lll_lock_wait () from /lib64/libpthread.so.0
>> #1 0x7f49b51fad02 in _L_lock_791 () from /lib64/libpthread.so.0
>> #2 0x7f49b51fac08 in pthread_mutex_lock () from /lib64/libpthread.so.0
>> #3 0x7f49b65d12dc in arena_bin_malloc_hard () from
>> /lib64/libjemalloc.so.1
>> #4 0x7f49b65d1516 in je_arena_tcache_fill_small () from
>> /lib64/libjemalloc.so.1
>> #5 0x7f49b65ea6ff in je_tcache_alloc_small_hard () from
>> /lib64/libjemalloc.so.1
>> #6 0x7f49b65ca14f in malloc () from /lib64/libjemalloc.so.1
>> #7 0x7f49b6c5a785 in _dl_scope_free () from
>> /lib64/ld-linux-x86-64.so.2
>> #8 0x7f49b6c55841 in _dl_map_object_deps () from
>> /lib64/ld-linux-x86-64.so.2
>> #9 0x7f49b6c5ba4b in dl_open_worker () from
>> /lib64/ld-linux-x86-64.so.2
>> #10 0x7f49b6c57364 in _dl_catch_error () from
>> /lib64/ld-linux-x86-64.so.2
>> #11 0x7f49b6c5b35b in _dl_open () from /lib64/ld-linux-x86-64.so.2
>> #12 0x7f49b48f5ff2 in do_dlopen () from /lib64/libc.so.6
>> #13 0x7f49b6c57364 in _dl_catch_error () from
>> /lib64/ld-linux-x86-64.so.2
>> #14 0x7f49b48f60b2 in __libc_dlopen_mode () from /lib64/libc.so.6
>> #15 0x7f49b48cf595 in init () from /lib64/libc.so.6
>> #16 0x7f49b51fdbb0 in pthread_once () from /lib64/libpthread.so.0
>> #17 0x7f49b48cf6ac in backtrace () from /lib64/libc.so.6
>> #18 0x0045193d in gsh_backtrace () at
>> /usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:228
>> #19 0x004519fe in crash_handler (signo=11,
>> info=0x7f49b155db70, ctx=0x7f49b155da40) at
>> /usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:244
>> #20 
>> #21 0x7f49b65d0c61 in arena_purge () from /lib64/libjemalloc.so.1
>> #22 0x7f49b65d218d in je_arena_dalloc_large () from
>> /lib64/libjemalloc.so.1
>>
>> 
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> ___
>> Nfs-ganesha-devel mailing list
>> Nfs-ganesha-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>>
>>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] crash in jemalloc leading to a deadlock.

2017-11-08 Thread Daniel Gryniewicz 
Allocating in a backtrace seems like a very bad idea.  If there's ever a 
crash during an allocation, it is guaranteed to deadlock.


Daniel

On 11/08/2017 01:43 PM, Pradeep wrote:

I'm using Ganesha 2.6 dev.12 with jemalloc-3.6.0 and hitting a case
where jemalloc seem to be holding a lock and crashing. In Ganesha's
gsh_backtrace(), we try to allocate memory and that hangs (ended up in
deadlock). Have you seen this before? Perhaps it is a good idea not to
allocate memory in backtrace path?


#0 0x7f49b51ff1bd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x7f49b51fad02 in _L_lock_791 () from /lib64/libpthread.so.0
#2 0x7f49b51fac08 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x7f49b65d12dc in arena_bin_malloc_hard () from /lib64/libjemalloc.so.1
#4 0x7f49b65d1516 in je_arena_tcache_fill_small () from
/lib64/libjemalloc.so.1
#5 0x7f49b65ea6ff in je_tcache_alloc_small_hard () from
/lib64/libjemalloc.so.1
#6 0x7f49b65ca14f in malloc () from /lib64/libjemalloc.so.1
#7 0x7f49b6c5a785 in _dl_scope_free () from /lib64/ld-linux-x86-64.so.2
#8 0x7f49b6c55841 in _dl_map_object_deps () from /lib64/ld-linux-x86-64.so.2
#9 0x7f49b6c5ba4b in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#10 0x7f49b6c57364 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#11 0x7f49b6c5b35b in _dl_open () from /lib64/ld-linux-x86-64.so.2
#12 0x7f49b48f5ff2 in do_dlopen () from /lib64/libc.so.6
#13 0x7f49b6c57364 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#14 0x7f49b48f60b2 in __libc_dlopen_mode () from /lib64/libc.so.6
#15 0x7f49b48cf595 in init () from /lib64/libc.so.6
#16 0x7f49b51fdbb0 in pthread_once () from /lib64/libpthread.so.0
#17 0x7f49b48cf6ac in backtrace () from /lib64/libc.so.6
#18 0x0045193d in gsh_backtrace () at
/usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:228
#19 0x004519fe in crash_handler (signo=11,
info=0x7f49b155db70, ctx=0x7f49b155da40) at
/usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:244
#20 
#21 0x7f49b65d0c61 in arena_purge () from /lib64/libjemalloc.so.1
#22 0x7f49b65d218d in je_arena_dalloc_large () from /lib64/libjemalloc.so.1

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] crash in jemalloc leading to a deadlock.

2017-11-08 Thread Frank Filz 
> I'm using Ganesha 2.6 dev.12 with jemalloc-3.6.0 and hitting a case where
> jemalloc seem to be holding a lock and crashing. In Ganesha's
> gsh_backtrace(), we try to allocate memory and that hangs (ended up in
> deadlock). Have you seen this before? Perhaps it is a good idea not to
> allocate memory in backtrace path?

Hmm, sadly backtrace_symbols is a library function, and yea, it allocates
memory...

Not sure there's much we can do about that. It would be interesting to
understand what is causing jemalloc to throw a signal. We probably managed
to corrupt memory... You might run under ASAN and see if it highlights
anything.

When we detect out of memory, we abort() which is not catchable and thus no
backtrace, so at least that path is safe...

Frank

> #0 0x7f49b51ff1bd in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x7f49b51fad02 in _L_lock_791 () from /lib64/libpthread.so.0
> #2 0x7f49b51fac08 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x7f49b65d12dc in arena_bin_malloc_hard () from
> /lib64/libjemalloc.so.1
> #4 0x7f49b65d1516 in je_arena_tcache_fill_small () from
> /lib64/libjemalloc.so.1
> #5 0x7f49b65ea6ff in je_tcache_alloc_small_hard () from
> /lib64/libjemalloc.so.1
> #6 0x7f49b65ca14f in malloc () from /lib64/libjemalloc.so.1
> #7 0x7f49b6c5a785 in _dl_scope_free () from
/lib64/ld-linux-x86-64.so.2
> #8 0x7f49b6c55841 in _dl_map_object_deps () from /lib64/ld-linux-x86-
> 64.so.2
> #9 0x7f49b6c5ba4b in dl_open_worker () from /lib64/ld-linux-x86-
> 64.so.2
> #10 0x7f49b6c57364 in _dl_catch_error () from /lib64/ld-linux-x86-
> 64.so.2
> #11 0x7f49b6c5b35b in _dl_open () from /lib64/ld-linux-x86-64.so.2
> #12 0x7f49b48f5ff2 in do_dlopen () from /lib64/libc.so.6
> #13 0x7f49b6c57364 in _dl_catch_error () from /lib64/ld-linux-x86-
> 64.so.2
> #14 0x7f49b48f60b2 in __libc_dlopen_mode () from /lib64/libc.so.6
> #15 0x7f49b48cf595 in init () from /lib64/libc.so.6
> #16 0x7f49b51fdbb0 in pthread_once () from /lib64/libpthread.so.0
> #17 0x7f49b48cf6ac in backtrace () from /lib64/libc.so.6
> #18 0x0045193d in gsh_backtrace () at
> /usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:228
> #19 0x004519fe in crash_handler (signo=11, info=0x7f49b155db70,
> ctx=0x7f49b155da40) at
> /usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:244
> #20 
> #21 0x7f49b65d0c61 in arena_purge () from /lib64/libjemalloc.so.1
> #22 0x7f49b65d218d in je_arena_dalloc_large () from
> /lib64/libjemalloc.so.1
> 
>

--
> Check out the vibrant tech community on one of the world's most engaging
> tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel