This list has been deprecated. Please subscribe to the new devel list at
lists.nfs-ganesha.org.
Hi Malahal
I've set up ABRT so I'm now getting coredumps for the crashes. I've
installed debuginfo package for nfs-ganesha and libntirpc.
I'd be really grateful if you could give me some guidance on debugging this.
Some info on the latest crash:
The following was echoed to the kernel log:
traps: ganesha.nfsd[28589] general protection ip:7fcf2421dded
> sp:7fcd9d4d03a0 error:0 in libntirpc.so.1.6.3[7fcf2420d000+3d000]
>
Last lines of output from # gdb /usr/bin/ganesha.nfsd coredump:
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/ganesha.nfsd -L
/var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.c'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fcf2421dded in opr_rbtree_insert (head=head@entry=0x7fcef800c528,
node=node@entry=0x7fce68004750) at
/usr/src/debug/ntirpc-1.6.3/src/rbtree.c:271
271 switch (head->cmpf(node, parent)) {
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.x86_64 dbus-libs-1.10.24-7.el7.x86_64
elfutils-libelf-0.170-4.el7.x86_64 elfutils-libs-0.170-4.el7.x86_64
glibc-2.17-222.el7.x86_64 gssproxy-0.7.0-17.el7.x86_64
keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-19.el7.x86_64
libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-52.el7.x86_64
libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-12.el7_5.x86_64
libgcc-4.8.5-28.el7_5.1.x86_64 libgcrypt-1.5.3-14.el7.x86_64
libgpg-error-1.12-3.el7.x86_64 libnfsidmap-0.25-19.el7.x86_64
libselinux-2.5-12.el7.x86_64 libuuid-2.23.2-52.el7.x86_64
lz4-1.7.5-2.el7.x86_64 pcre-8.32-17.el7.x86_64
systemd-libs-219-57.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64
zlib-1.2.7-17.el7.x86_64
Output from bt:
(gdb) bt
#0 0x00007fcf2421dded in opr_rbtree_insert (head=head@entry=0x7fcef800c528,
node=node@entry=0x7fce68004750) at
/usr/src/debug/ntirpc-1.6.3/src/rbtree.c:271
#1 0x00007fcf24218eac in clnt_req_setup (cc=cc@entry=0x7fce68004720,
timeout=...) at /usr/src/debug/ntirpc-1.6.3/src/clnt_generic.c:515
#2 0x000055d62490347f in nsm_unmonitor (host=host@entry=0x7fce00018ea0) at
/usr/src/debug/nfs-ganesha-2.6.3/src/Protocols/NLM/nsm.c:219
#3 0x000055d6249425cf in dec_nsm_client_ref (client=0x7fce00018ea0) at
/usr/src/debug/nfs-ganesha-2.6.3/src/SAL/nlm_owner.c:857
#4 0x000055d624942f61 in free_nlm_client (client=0x7fce00017500) at
/usr/src/debug/nfs-ganesha-2.6.3/src/SAL/nlm_owner.c:1039
#5 0x000055d6249431d3 in dec_nlm_client_ref (client=0x7fce00017500) at
/usr/src/debug/nfs-ganesha-2.6.3/src/SAL/nlm_owner.c:1130
#6 0x000055d6249439ae in free_nlm_owner (owner=owner@entry=0x7fce00024bc0)
at /usr/src/debug/nfs-ganesha-2.6.3/src/SAL/nlm_owner.c:1314
#7 0x000055d624929a48 in free_state_owner (owner=0x7fce00024bc0) at
/usr/src/debug/nfs-ganesha-2.6.3/src/SAL/state_misc.c:818
#8 0x000055d624929dc0 in dec_state_owner_ref (owner=0x7fce00024bc0) at
/usr/src/debug/nfs-ganesha-2.6.3/src/SAL/state_misc.c:968
#9 0x000055d6248ff173 in nlm4_Unlock (args=0x7fce68003b98,
req=0x7fce68003490, res=0x7fce68000d70) at
/usr/src/debug/nfs-ganesha-2.6.3/src/Protocols/NLM/nlm_Unlock.c:127
#10 0x000055d6248c0f0f in nfs_rpc_process_request (reqdata=0x7fce68003490)
at /usr/src/debug/nfs-ganesha-2.6.3/src/MainNFSD/nfs_worker_thread.c:1329
#11 0x000055d6248c02ba in nfs_rpc_decode_request (xprt=0x7fcef011b600,
xdrs=0x7fce68001480)
at
/usr/src/debug/nfs-ganesha-2.6.3/src/MainNFSD/nfs_rpc_dispatcher_thread.c:1341
#12 0x00007fcf2422dbcd in svc_rqst_xprt_task (wpe=0x7fcef011b818) at
/usr/src/debug/ntirpc-1.6.3/src/svc_rqst.c:751
#13 0x00007fcf2422df2a in svc_rqst_epoll_events (n_events=<optimized out>,
sr_rec=0x55d6253b3fd0) at /usr/src/debug/ntirpc-1.6.3/src/svc_rqst.c:923
#14 svc_rqst_epoll_loop (sr_rec=<optimized out>) at
/usr/src/debug/ntirpc-1.6.3/src/svc_rqst.c:996
#15 svc_rqst_run_task (wpe=0x55d6253b3fd0) at
/usr/src/debug/ntirpc-1.6.3/src/svc_rqst.c:1032
#16 0x00007fcf2423671a in work_pool_thread (arg=0x55d6282753f0) at
/usr/src/debug/ntirpc-1.6.3/src/work_pool.c:176
#17 0x00007fcf2465ce25 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fcf23d28bad in clone () from /lib64/libc.so.6
Thanks for your assistance so far on this
David
On Fri, Sep 28, 2018 at 8:06 PM David C <dcsysengin...@gmail.com> wrote:
> Thanks, Malahal. I'll get the coredumps enabled. I've had a few more
> crashes today, hopefully they'll shed some light on the issue.
>
> On Fri, Sep 28, 2018 at 1:20 PM Malahal Naineni <mala...@gmail.com> wrote:
>
>> You need to enable coredumps for ganesha. Here are some instructions!
>> Step2 is NOT needed as your packages are signed:
>>
>> https://ganltc.github.io/setup-to-take-ganesha-coredumps.html
>>
>> On Fri, Sep 28, 2018 at 4:38 PM David C <dcsysengin...@gmail.com> wrote:
>>
>>> This list has been deprecated. Please subscribe to the new devel list at
>>> lists.nfs-ganesha.org.
>>> Hi All
>>>
>>> CentOS 7.5
>>> nfs-ganesha-2.6.3-1.el7.x86_64
>>> nfs-ganesha-vfs-2.6.3-1.el7.x86_64
>>> libntirpc-1.6.3-1.el7.x86_64
>>>
>>> My Ganesha service crashed and the following was echoed to my kernel log:
>>>
>>> ganesha.nfsd[28752]: segfault at 0 ip (null) sp
>>>> 00007ff9a2af8458 error 14 in ganesha.nfsd[559170ef3000+1a4000]
>>>>
>>>
>>> Nothing in my ganesha.log
>>>
>>> These are the log settings from my ganesha.conf:
>>>
>>> LOG {
>>>> ## Default log level for all components
>>>> Default_Log_Level = DEBUG;
>>>>
>>>> ## Configure per-component log levels.
>>>> #Components {
>>>> #FSAL = INFO;
>>>> #NFS4 = EVENT;
>>>> #}
>>>>
>>>> ## Where to log
>>>> Facility {
>>>> name = FILE;
>>>> destination = "/var/log/ganesha.log";
>>>> enable = active;
>>>> }
>>>> }
>>>>
>>>
>>> This is an example of one of my exports (they're all Nfsv3 with VFS
>>> FSAL):
>>>
>>> EXPORT
>>>> {
>>>> Export_Id = 80;
>>>> Path = /mnt/dir;
>>>> Pseudo = /mnt/dir;
>>>> Access_Type = RW;
>>>> Protocols = 3;
>>>> Transports = TCP;
>>>> Squash = no_root_squash;
>>>> Disable_ACL=False;
>>>> Filesystem_Id = 101.1;
>>>> CLIENT {
>>>> Clients = *;
>>>> Squash = None;
>>>> Access_Type = RW;
>>>> }
>>>> FSAL {
>>>> Name = VFS;
>>>> }
>>>> }
>>>>
>>>>
>>> The exports are mounted on CentOS 7.4 clients with autofs-5.0.7 and
>>> nfs-utils-1.3.0-0.48.el7_4.x86_64
>>>
>>> This crashed occurred approx 2 hours after I increased the number of
>>> clients accessing the server by approx five clients, don't know if that's
>>> related
>>>
>>> Could someone help me troubleshoot this please?
>>>
>>> Many thanks
>>> David
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Nfs-ganesha-devel mailing list
>>> Nfs-ganesha-devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>>>
>>
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
