While testing V2.4.1 to check if I was still seeing the
"assert(refcount > 0)" failure I reported on V2.4.0.3, which is still
occurring with 2.4.1, I am also seeing cases of segfault errors.
Three of the segfaults I have seen in my testing of V2.4.1 had
identical backtraces, and the forth was at a different point in the
code. From the command "dmesg -T | grep segfault"
[Tue Nov 1 06:33:28 2016] ganesha.nfsd[33814]: segfault at 0 ip
00000000004cdcad sp 00007fae1df92160 error 6 in
ganesha.nfsd[400000+1bb000]
[Tue Nov 1 07:02:17 2016] ganesha.nfsd[37754]: segfault at 0 ip
00000000004cdcad sp 00007f3b9c366160 error 6 in
ganesha.nfsd[400000+1bb000]
[Tue Nov 1 11:17:07 2016] ganesha.nfsd[39697]: segfault at a0 ip
00007f4b0ed4f414 sp 00007f4ab11422b0 error 4 in
libpthread-2.19.so[7f4b0ed45000+19000]
[Tue Nov 1 14:32:18 2016] ganesha.nfsd[41935]: segfault at 0 ip
00000000004cdcad sp 00007f968ad86160 error 6 in
ganesha.nfsd[400000+1bb000]
As I was debugging the assert issue, I had the system configured to
capture core dumps. All three of the "error 6" backtraces looked the
same:
...
Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/keeper/bin/ganesha.nfsd -F -L
/var/log/nfs-ganesha.log'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000004cdcad in __glist_add (left=0x0, right=0x7fadcc1ca3a8,
elt=0x7fadc94e0f50) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/include/gsh_list.h:78
78 left->next = elt;
(gdb) bt
#0 0x00000000004cdcad in __glist_add (left=0x0, right=0x7fadcc1ca3a8,
elt=0x7fadc94e0f50) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/include/gsh_list.h:78
#1 0x00000000004cdce9 in glist_add_tail (head=0x7fadcc1ca3a8,
elt=0x7fadc94e0f50) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/include/gsh_list.h:86
#2 0x00000000004cef3b in state_add_impl (obj=0x7fadb2a41638,
state_type=STATE_TYPE_SHARE, state_data=0x7fae1df923d0,
owner_input=0x7fadcc1ca200, state=0x7fae1df92da0,
refer=0x7fae1df924f0) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/SAL/nfs4_state.c:213
#3 0x000000000047333c in open4_ex (arg=0x7fadace1a8a8,
data=0x7fae1df92e80, res_OPEN4=0x7fadce951948,
clientid=0x7fadc8819600, owner=0x7fadcc1ca200,
file_state=0x7fae1df92da0, new_state=0x7fae1df92d68) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1535
#4 0x000000000047432e in nfs4_op_open (op=0x7fadace1a8a0,
data=0x7fae1df92e80, resp=0x7fadce951940) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1844
#5 0x000000000045e90d in nfs4_Compound (arg=0x7fadac9d24e8,
req=0x7fadac9d2328, res=0x7fadc9466ec0) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
#6 0x000000000044b9a8 in nfs_rpc_execute (reqdata=0x7fadac9d2300) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
#7 0x000000000044c317 in worker_run (ctx=0x7fae53fd8680) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
#8 0x000000000050b288 in fridgethr_start_routine (arg=0x7fae53fd8680)
at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/support/fridgethr.c:550
#9 0x00007fae59b65182 in start_thread (arg=0x7fae1df94700) at
pthread_create.c:312
#10 0x00007fae5943b47d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
The "error 4" bt:
...
[New LWP 39788]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/keeper/bin/ganesha.nfsd -F -L
/var/log/nfs-ganesha.log'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:66
66 ../nptl/pthread_mutex_lock.c: No such file or directory.
(gdb) bt
#0 __GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:66
#1 0x00000000004d7114 in nfs4_Check_Stateid (stateid=0x7f4a779e50a8,
fsal_obj=0x7f4a6c303038, state=0x7f4ab11424f0, data=0x7f4ab1142e80,
flags=63, owner_seqid=0, check_seqid=false, tag=0x569d7b "WRITE") at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/SAL/nfs4_state_id.c:1076
#2 0x00000000004852b8 in nfs4_write (op=0x7f4a779e50a0,
data=0x7f4ab1142e80, resp=0x7f4a75e14dc0, io=FSAL_IO_WRITE, info=0x0)
at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/Protocols/NFS/nfs4_op_write.c:213
#3 0x0000000000485d1b in nfs4_op_write (op=0x7f4a779e50a0,
data=0x7f4ab1142e80, resp=0x7f4a75e14dc0) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/Protocols/NFS/nfs4_op_write.c:482
#4 0x000000000045e90d in nfs4_Compound (arg=0x7f4a779e4268,
req=0x7f4a779e40a8, res=0x7f4a7d4c5c40) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
#5 0x000000000044b9a8 in nfs_rpc_execute (reqdata=0x7f4a779e4080) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
#6 0x000000000044c317 in worker_run (ctx=0x7f4b093e9100) at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
#7 0x000000000050b288 in fridgethr_start_routine (arg=0x7f4b093e9100)
at
/home/keeper/work/ganesha/2.4.1-debug/ganeshabuilder/nfs-ganesha/src/support/fridgethr.c:550
#8 0x00007f4b0ed4d182 in start_thread (arg=0x7f4ab1144700) at
pthread_create.c:312
#9 0x00007f4b0e62347d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
I have the cores, so if you need a backtrace of all threads or other
info, just let me know.
The test I was running was the same test as given in the email with
the subject: "assert in dec_state_owner_ref() with V2.4.0.3" which has
multiple worker processes creating 1000 uniquely named directories and
in each creates 11 256K files. It then does a "rm -rf" on the
directory tree and starts over.
Information on the test system and versions:
Ganesha version:
# ganesha.nfsd -v
nfs-ganesha compiled on Oct 30 2016 at 22:46:56
Release = V2.4.1
Release comment = GANESHA file server is 64 bits compliant and
supports NFS v3,4.0,4.1 (pNFS) and 9P
Git HEAD = a146801b8e29580697391fb7f165ae9ead023894
Git Describe = V2.4.1-0-ga146801
OS is Ubuntu 14.04 with a 4.8.4 x86_64 kernel on a 8 processor system.
Backstore file systems is ext4.
The client is mounting the NFS server with the fstab line:
ede-c2-gw01:/var/top /C2-NFS4 nfs4 rw,hard,noauto,vers=4.2 0 0
# cat ganesha.conf
LOG {
components {
ALL = INFO;
}
}
EXPORT_DEFAULTS {
SecType = none, sys;
Protocols = 3, 4;
Transports = TCP;
}
EXPORT {
Export_ID = 43;
Path = /var/top;
Pseudo = /var/top;
Access_Type = RW;
Squash = No_Root_Squash;
FSAL {
Name = VFS;
}
}
Cmake command:
# cmake -DCMAKE_INSTALL_PREFIX=/opt/keeper -DALLOCATOR=jemalloc
-DUSE_ADMIN_TOOLS=ON -DUSE_DBUS=ON ../src
Thanks,
Eric
------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
