These to be a use-after-free on an owner (refcount bug, likely?).
Daniel
On 11/02/2016 01:47 AM, Eric Eastman wrote:
> While testing V2.4.1 to check if I was still seeing the
> "assert(refcount > 0)" failure I reported on V2.4.0.3, which is still
> occurring with 2.4.1, I am also seeing cases of segfault errors.
> Three of the segfaults I have seen in my testing of V2.4.1 had
> identical backtraces, and the forth was at a different point in the
> code. From the command "dmesg -T | grep segfault"
>
> [Tue Nov 1 06:33:28 2016] ganesha.nfsd[33814]: segfault at 0 ip
> 00000000004cdcad sp 00007fae1df92160 error 6 in
> ganesha.nfsd[400000+1bb000]
> [Tue Nov 1 07:02:17 2016] ganesha.nfsd[37754]: segfault at 0 ip
> 00000000004cdcad sp 00007f3b9c366160 error 6 in
> ganesha.nfsd[400000+1bb000]
> [Tue Nov 1 11:17:07 2016] ganesha.nfsd[39697]: segfault at a0 ip
> 00007f4b0ed4f414 sp 00007f4ab11422b0 error 4 in
> libpthread-2.19.so[7f4b0ed45000+19000]
> [Tue Nov 1 14:32:18 2016] ganesha.nfsd[41935]: segfault at 0 ip
> 00000000004cdcad sp 00007f968ad86160 error 6 in
> ganesha.nfsd[400000+1bb000]
>
> As I was debugging the assert issue, I had the system configured to
> capture core dumps. All three of the "error 6" backtraces looked the
> same:
>
> ...
> Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/opt/keeper/bin/ganesha.nfsd -F -L
> /var/log/nfs-ganesha.log'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00000000004cdcad in __glist_add (left=0x0, right=0x7fadcc1ca3a8,
> elt=0x7fadc94e0f50) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/include/gsh_list.h:78
> 78 left->next = elt;
>
> (gdb) bt
> #0 0x00000000004cdcad in __glist_add (left=0x0, right=0x7fadcc1ca3a8,
> elt=0x7fadc94e0f50) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/include/gsh_list.h:78
> #1 0x00000000004cdce9 in glist_add_tail (head=0x7fadcc1ca3a8,
> elt=0x7fadc94e0f50) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/include/gsh_list.h:86
> #2 0x00000000004cef3b in state_add_impl (obj=0x7fadb2a41638,
> state_type=STATE_TYPE_SHARE, state_data=0x7fae1df923d0,
> owner_input=0x7fadcc1ca200, state=0x7fae1df92da0,
> refer=0x7fae1df924f0) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/SAL/nfs4_state.c:213
> #3 0x000000000047333c in open4_ex (arg=0x7fadace1a8a8,
> data=0x7fae1df92e80, res_OPEN4=0x7fadce951948,
> clientid=0x7fadc8819600, owner=0x7fadcc1ca200,
> file_state=0x7fae1df92da0, new_state=0x7fae1df92d68) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1535
> #4 0x000000000047432e in nfs4_op_open (op=0x7fadace1a8a0,
> data=0x7fae1df92e80, resp=0x7fadce951940) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1844
> #5 0x000000000045e90d in nfs4_Compound (arg=0x7fadac9d24e8,
> req=0x7fadac9d2328, res=0x7fadc9466ec0) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
> #6 0x000000000044b9a8 in nfs_rpc_execute (reqdata=0x7fadac9d2300) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
> #7 0x000000000044c317 in worker_run (ctx=0x7fae53fd8680) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
> #8 0x000000000050b288 in fridgethr_start_routine (arg=0x7fae53fd8680)
> at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/support/fridgethr.c:550
> #9 0x00007fae59b65182 in start_thread (arg=0x7fae1df94700) at
> pthread_create.c:312
> #10 0x00007fae5943b47d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>
> The "error 4" bt:
> ...
> [New LWP 39788]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/opt/keeper/bin/ganesha.nfsd -F -L
> /var/log/nfs-ganesha.log'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 __GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:66
> 66 ../nptl/pthread_mutex_lock.c: No such file or directory.
>
> (gdb) bt
> #0 __GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:66
> #1 0x00000000004d7114 in nfs4_Check_Stateid (stateid=0x7f4a779e50a8,
> fsal_obj=0x7f4a6c303038, state=0x7f4ab11424f0, data=0x7f4ab1142e80,
> flags=63, owner_seqid=0, check_seqid=false, tag=0x569d7b "WRITE") at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/SAL/nfs4_state_id.c:1076
> #2 0x00000000004852b8 in nfs4_write (op=0x7f4a779e50a0,
> data=0x7f4ab1142e80, resp=0x7f4a75e14dc0, io=FSAL_IO_WRITE, info=0x0)
> at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/Protocols/NFS/nfs4_op_write.c:213
> #3 0x0000000000485d1b in nfs4_op_write (op=0x7f4a779e50a0,
> data=0x7f4ab1142e80, resp=0x7f4a75e14dc0) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/Protocols/NFS/nfs4_op_write.c:482
> #4 0x000000000045e90d in nfs4_Compound (arg=0x7f4a779e4268,
> req=0x7f4a779e40a8, res=0x7f4a7d4c5c40) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
> #5 0x000000000044b9a8 in nfs_rpc_execute (reqdata=0x7f4a779e4080) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
> #6 0x000000000044c317 in worker_run (ctx=0x7f4b093e9100) at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
> #7 0x000000000050b288 in fridgethr_start_routine (arg=0x7f4b093e9100)
> at
> /home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/support/fridgethr.c:550
> #8 0x00007f4b0ed4d182 in start_thread (arg=0x7f4ab1144700) at
> pthread_create.c:312
> #9 0x00007f4b0e62347d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>
> I have the cores, so if you need a backtrace of all threads or other
> info, just let me know.
>
> The test I was running was the same test as given in the email with
> the subject: "assert in dec_state_owner_ref() with V2.4.0.3" which has
> multiple worker processes creating 1000 uniquely named directories and
> in each creates 11 256K files. It then does a "rm -rf" on the
> directory tree and starts over.
>
> Information on the test system and versions:
>
> Ganesha version:
> # ganesha.nfsd -v
> nfs-ganesha compiled on Oct 30 2016 at 22:46:56
> Release = V2.4.1
> Release comment = GANESHA file server is 64 bits compliant and
> supports NFS v3,4.0,4.1 (pNFS) and 9P
> Git HEAD = a146801b8e29580697391fb7f165ae9ead023894
> Git Describe = V2.4.1-0-ga146801
>
> OS is Ubuntu 14.04 with a 4.8.4 x86_64 kernel on a 8 processor system.
> Backstore file systems is ext4.
>
> The client is mounting the NFS server with the fstab line:
> ede-c2-gw01:/var/top /C2-NFS4 nfs4 rw,hard,noauto,vers=4.2 0 0
>
> # cat ganesha.conf
> LOG {
> components {
> ALL = INFO;
> }
> }
> EXPORT_DEFAULTS {
> SecType = none, sys;
> Protocols = 3, 4;
> Transports = TCP;
> }
> EXPORT {
> Export_ID = 43;
> Path = /var/top;
> Pseudo = /var/top;
> Access_Type = RW;
> Squash = No_Root_Squash;
> FSAL {
> Name = VFS;
> }
> }
>
> Cmake command:
> # cmake -DCMAKE_INSTALL_PREFIX=/opt/keeper -DALLOCATOR=jemalloc
> -DUSE_ADMIN_TOOLS=ON -DUSE_DBUS=ON ../src
>
> Thanks,
> Eric
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> _______________________________________________
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
