This list has been deprecated. Please subscribe to the new devel list at
lists.nfs-ganesha.org.
Looking at the code head->cmpf should be "clnt_req_xid_cmpf" function
address. Your gdb didn't show that, but I don't know how that could happen
with the V2.6.3 code though. @Dan, any insights for this issue?
On Mon, Oct 1, 2018 at 2:22 PM David C <dcsysengin...@gmail.com> wrote:
> Hi Malahal
>
> Result of that command:
>
> (gdb) p head->cmpf
> $1 = (opr_rbtree_cmpf_t) 0x31fb0b405ba000b7
>
> Thanks,
>
> On Mon, Oct 1, 2018 at 5:55 AM Malahal Naineni <mala...@gmail.com> wrote:
>
>> Looks like the head is messed up. Run these in gdb and let us know the
>> second commands output. 1. "frame 0" 2. "p head->cmpf". I believe,
>> head->cmpf function is NULL or bad leading to this segfault. I haven't seen
>> this crash before and never used Ganesha 2.6 version.
>>
>> Regards, Malahal.
>>
>> On Mon, Oct 1, 2018 at 1:25 AM David C <dcsysengin...@gmail.com> wrote:
>>
>>> Hi Malahal
>>>
>>> I've set up ABRT so I'm now getting coredumps for the crashes. I've
>>> installed debuginfo package for nfs-ganesha and libntirpc.
>>>
>>> I'd be really grateful if you could give me some guidance on debugging
>>> this.
>>>
>>> Some info on the latest crash:
>>>
>>> The following was echoed to the kernel log:
>>>
>>> traps: ganesha.nfsd[28589] general protection ip:7fcf2421dded
>>>> sp:7fcd9d4d03a0 error:0 in libntirpc.so.1.6.3[7fcf2420d000+3d000]
>>>>
>>>
>>> Last lines of output from # gdb /usr/bin/ganesha.nfsd coredump:
>>>
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>>> Core was generated by `/usr/bin/ganesha.nfsd -L
>>> /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.c'.
>>> Program terminated with signal 11, Segmentation fault.
>>> #0 0x00007fcf2421dded in opr_rbtree_insert (head=head@entry=0x7fcef800c528,
>>> node=node@entry=0x7fce68004750) at
>>> /usr/src/debug/ntirpc-1.6.3/src/rbtree.c:271
>>> 271 switch (head->cmpf(node, parent)) {
>>> Missing separate debuginfos, use: debuginfo-install
>>> bzip2-libs-1.0.6-13.el7.x86_64 dbus-libs-1.10.24-7.el7.x86_64
>>> elfutils-libelf-0.170-4.el7.x86_64 elfutils-libs-0.170-4.el7.x86_64
>>> glibc-2.17-222.el7.x86_64 gssproxy-0.7.0-17.el7.x86_64
>>> keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-19.el7.x86_64
>>> libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-52.el7.x86_64
>>> libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-12.el7_5.x86_64
>>> libgcc-4.8.5-28.el7_5.1.x86_64 libgcrypt-1.5.3-14.el7.x86_64
>>> libgpg-error-1.12-3.el7.x86_64 libnfsidmap-0.25-19.el7.x86_64
>>> libselinux-2.5-12.el7.x86_64 libuuid-2.23.2-52.el7.x86_64
>>> lz4-1.7.5-2.el7.x86_64 pcre-8.32-17.el7.x86_64
>>> systemd-libs-219-57.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64
>>> zlib-1.2.7-17.el7.x86_64
>>>
>>> Output from bt:
>>>
>>> (gdb) bt
>>> #0 0x00007fcf2421dded in opr_rbtree_insert (head=head@entry=0x7fcef800c528,
>>> node=node@entry=0x7fce68004750) at
>>> /usr/src/debug/ntirpc-1.6.3/src/rbtree.c:271
>>> #1 0x00007fcf24218eac in clnt_req_setup (cc=cc@entry=0x7fce68004720,
>>> timeout=...) at /usr/src/debug/ntirpc-1.6.3/src/clnt_generic.c:515
>>> #2 0x000055d62490347f in nsm_unmonitor (host=host@entry=0x7fce00018ea0)
>>> at /usr/src/debug/nfs-ganesha-2.6.3/src/Protocols/NLM/nsm.c:219
>>> #3 0x000055d6249425cf in dec_nsm_client_ref (client=0x7fce00018ea0) at
>>> /usr/src/debug/nfs-ganesha-2.6.3/src/SAL/nlm_owner.c:857
>>> #4 0x000055d624942f61 in free_nlm_client (client=0x7fce00017500) at
>>> /usr/src/debug/nfs-ganesha-2.6.3/src/SAL/nlm_owner.c:1039
>>> #5 0x000055d6249431d3 in dec_nlm_client_ref (client=0x7fce00017500) at
>>> /usr/src/debug/nfs-ganesha-2.6.3/src/SAL/nlm_owner.c:1130
>>> #6 0x000055d6249439ae in free_nlm_owner (owner=owner@entry=0x7fce00024bc0)
>>> at /usr/src/debug/nfs-ganesha-2.6.3/src/SAL/nlm_owner.c:1314
>>> #7 0x000055d624929a48 in free_state_owner (owner=0x7fce00024bc0) at
>>> /usr/src/debug/nfs-ganesha-2.6.3/src/SAL/state_misc.c:818
>>> #8 0x000055d624929dc0 in dec_state_owner_ref (owner=0x7fce00024bc0) at
>>> /usr/src/debug/nfs-ganesha-2.6.3/src/SAL/state_misc.c:968
>>> #9 0x000055d6248ff173 in nlm4_Unlock (args=0x7fce68003b98,
>>> req=0x7fce68003490, res=0x7fce68000d70) at
>>> /usr/src/debug/nfs-ganesha-2.6.3/src/Protocols/NLM/nlm_Unlock.c:127
>>> #10 0x000055d6248c0f0f in nfs_rpc_process_request
>>> (reqdata=0x7fce68003490) at
>>> /usr/src/debug/nfs-ganesha-2.6.3/src/MainNFSD/nfs_worker_thread.c:1329
>>> #11 0x000055d6248c02ba in nfs_rpc_decode_request (xprt=0x7fcef011b600,
>>> xdrs=0x7fce68001480)
>>> at
>>> /usr/src/debug/nfs-ganesha-2.6.3/src/MainNFSD/nfs_rpc_dispatcher_thread.c:1341
>>> #12 0x00007fcf2422dbcd in svc_rqst_xprt_task (wpe=0x7fcef011b818) at
>>> /usr/src/debug/ntirpc-1.6.3/src/svc_rqst.c:751
>>> #13 0x00007fcf2422df2a in svc_rqst_epoll_events (n_events=<optimized
>>> out>, sr_rec=0x55d6253b3fd0) at
>>> /usr/src/debug/ntirpc-1.6.3/src/svc_rqst.c:923
>>> #14 svc_rqst_epoll_loop (sr_rec=<optimized out>) at
>>> /usr/src/debug/ntirpc-1.6.3/src/svc_rqst.c:996
>>> #15 svc_rqst_run_task (wpe=0x55d6253b3fd0) at
>>> /usr/src/debug/ntirpc-1.6.3/src/svc_rqst.c:1032
>>> #16 0x00007fcf2423671a in work_pool_thread (arg=0x55d6282753f0) at
>>> /usr/src/debug/ntirpc-1.6.3/src/work_pool.c:176
>>> #17 0x00007fcf2465ce25 in start_thread () from /lib64/libpthread.so.0
>>> #18 0x00007fcf23d28bad in clone () from /lib64/libc.so.6
>>>
>>> Thanks for your assistance so far on this
>>> David
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Sep 28, 2018 at 8:06 PM David C <dcsysengin...@gmail.com> wrote:
>>>
>>>> Thanks, Malahal. I'll get the coredumps enabled. I've had a few more
>>>> crashes today, hopefully they'll shed some light on the issue.
>>>>
>>>> On Fri, Sep 28, 2018 at 1:20 PM Malahal Naineni <mala...@gmail.com>
>>>> wrote:
>>>>
>>>>> You need to enable coredumps for ganesha. Here are some instructions!
>>>>> Step2 is NOT needed as your packages are signed:
>>>>>
>>>>> https://ganltc.github.io/setup-to-take-ganesha-coredumps.html
>>>>>
>>>>> On Fri, Sep 28, 2018 at 4:38 PM David C <dcsysengin...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> This list has been deprecated. Please subscribe to the new devel list
>>>>>> at lists.nfs-ganesha.org.
>>>>>> Hi All
>>>>>>
>>>>>> CentOS 7.5
>>>>>> nfs-ganesha-2.6.3-1.el7.x86_64
>>>>>> nfs-ganesha-vfs-2.6.3-1.el7.x86_64
>>>>>> libntirpc-1.6.3-1.el7.x86_64
>>>>>>
>>>>>> My Ganesha service crashed and the following was echoed to my kernel
>>>>>> log:
>>>>>>
>>>>>> ganesha.nfsd[28752]: segfault at 0 ip (null) sp
>>>>>>> 00007ff9a2af8458 error 14 in ganesha.nfsd[559170ef3000+1a4000]
>>>>>>>
>>>>>>
>>>>>> Nothing in my ganesha.log
>>>>>>
>>>>>> These are the log settings from my ganesha.conf:
>>>>>>
>>>>>> LOG {
>>>>>>> ## Default log level for all components
>>>>>>> Default_Log_Level = DEBUG;
>>>>>>>
>>>>>>> ## Configure per-component log levels.
>>>>>>> #Components {
>>>>>>> #FSAL = INFO;
>>>>>>> #NFS4 = EVENT;
>>>>>>> #}
>>>>>>>
>>>>>>> ## Where to log
>>>>>>> Facility {
>>>>>>> name = FILE;
>>>>>>> destination = "/var/log/ganesha.log";
>>>>>>> enable = active;
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>
>>>>>> This is an example of one of my exports (they're all Nfsv3 with VFS
>>>>>> FSAL):
>>>>>>
>>>>>> EXPORT
>>>>>>> {
>>>>>>> Export_Id = 80;
>>>>>>> Path = /mnt/dir;
>>>>>>> Pseudo = /mnt/dir;
>>>>>>> Access_Type = RW;
>>>>>>> Protocols = 3;
>>>>>>> Transports = TCP;
>>>>>>> Squash = no_root_squash;
>>>>>>> Disable_ACL=False;
>>>>>>> Filesystem_Id = 101.1;
>>>>>>> CLIENT {
>>>>>>> Clients = *;
>>>>>>> Squash = None;
>>>>>>> Access_Type = RW;
>>>>>>> }
>>>>>>> FSAL {
>>>>>>> Name = VFS;
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>> The exports are mounted on CentOS 7.4 clients with autofs-5.0.7 and
>>>>>> nfs-utils-1.3.0-0.48.el7_4.x86_64
>>>>>>
>>>>>> This crashed occurred approx 2 hours after I increased the number of
>>>>>> clients accessing the server by approx five clients, don't know if that's
>>>>>> related
>>>>>>
>>>>>> Could someone help me troubleshoot this please?
>>>>>>
>>>>>> Many thanks
>>>>>> David
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Nfs-ganesha-devel mailing list
>>>>>> Nfs-ganesha-devel@lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>>>>>>
>>>>>
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
