If we keep the logging fd open, then we need a signal to tell it that logrotate
has occurred…
Frank
From: Malahal Naineni [mailto:mala...@gmail.com]
Sent: Tuesday, November 14, 2017 9:44 PM
To: Frank Filz <ffilz...@mindspring.com>
Cc: d...@redhat.com; nfs-ganesha-devel@lists.sourceforge.net
Subject: Re: [Nfs-ganesha-devel] crash in jemalloc leading to a deadlock.
Silly glibc, they should have provided backtrace_symbols_func() to take a
callback function. Using backtrace_symbols_func() is a bit harder for us here.
Here is a thought:
1. Our logger should NOT open/re-open for every message. It should just open
once.
2. Our logger should have an interface to provide such fd using API or global
symbol.
3. There are some tricks to get "fd" with syslog tracing as well
4. Then pass such fd from step 2 to backtrace_symbols_fd()
Step1 might be optional as open() should mostly succeed though.
On Thu, Nov 9, 2017 at 7:53 PM, Frank Filz <ffilz...@mindspring.com
<mailto:ffilz...@mindspring.com> > wrote:
That might be a good solution, though what fd would we use? Can we safely open
an fd during a sighandler?
Frank
From: Malahal Naineni [mailto:mala...@gmail.com <mailto:mala...@gmail.com> ]
Sent: Wednesday, November 8, 2017 11:51 PM
To: d...@redhat.com <mailto:d...@redhat.com>
Cc: nfs-ganesha-devel@lists.sourceforge.net
<mailto:nfs-ganesha-devel@lists.sourceforge.net>
Subject: Re: [Nfs-ganesha-devel] crash in jemalloc leading to a deadlock.
backtrace_symbols_fd() takes the same buffer and size arguments as
backtrace_symbols(), but instead of returning an array of strings to the
caller, it writes the strings, one per line, to the file descriptor fd.
backtrace_symbols_fd() does not call malloc(3), and so can be employed in
situations where the latter function might fail.
On Thu, Nov 9, 2017 at 12:24 AM, Daniel Gryniewicz <d...@redhat.com
<mailto:d...@redhat.com> > wrote:
Allocating in a backtrace seems like a very bad idea. If there's ever a crash
during an allocation, it is guaranteed to deadlock.
Daniel
On 11/08/2017 01:43 PM, Pradeep wrote:
I'm using Ganesha 2.6 dev.12 with jemalloc-3.6.0 and hitting a case
where jemalloc seem to be holding a lock and crashing. In Ganesha's
gsh_backtrace(), we try to allocate memory and that hangs (ended up in
deadlock). Have you seen this before? Perhaps it is a good idea not to
allocate memory in backtrace path?
#0 0x00007f49b51ff1bd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f49b51fad02 in _L_lock_791 () from /lib64/libpthread.so.0
#2 0x00007f49b51fac08 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f49b65d12dc in arena_bin_malloc_hard () from /lib64/libjemalloc.so.1
#4 0x00007f49b65d1516 in je_arena_tcache_fill_small () from
/lib64/libjemalloc.so.1
#5 0x00007f49b65ea6ff in je_tcache_alloc_small_hard () from
/lib64/libjemalloc.so.1
#6 0x00007f49b65ca14f in malloc () from /lib64/libjemalloc.so.1
#7 0x00007f49b6c5a785 in _dl_scope_free () from /lib64/ld-linux-x86-64.so.2
#8 0x00007f49b6c55841 in _dl_map_object_deps () from /lib64/ld-linux-x86-64.so.2
#9 0x00007f49b6c5ba4b in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#10 0x00007f49b6c57364 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#11 0x00007f49b6c5b35b in _dl_open () from /lib64/ld-linux-x86-64.so.2
#12 0x00007f49b48f5ff2 in do_dlopen () from /lib64/libc.so.6
#13 0x00007f49b6c57364 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#14 0x00007f49b48f60b2 in __libc_dlopen_mode () from /lib64/libc.so.6
#15 0x00007f49b48cf595 in init () from /lib64/libc.so.6
#16 0x00007f49b51fdbb0 in pthread_once () from /lib64/libpthread.so.0
#17 0x00007f49b48cf6ac in backtrace () from /lib64/libc.so.6
#18 0x000000000045193d in gsh_backtrace () at
/usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:228
#19 0x00000000004519fe in crash_handler (signo=11,
info=0x7f49b155db70, ctx=0x7f49b155da40) at
/usr/src/debug/nfs-ganesha-2.6-dev.12/MainNFSD/nfs_init.c:244
#20 <signal handler called>
#21 0x00007f49b65d0c61 in arena_purge () from /lib64/libjemalloc.so.1
#22 0x00007f49b65d218d in je_arena_dalloc_large () from /lib64/libjemalloc.so.1
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
<mailto:Nfs-ganesha-devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
<mailto:Nfs-ganesha-devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=icon>
Virus-free.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=link>
www.avast.com
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
