I have created a simple worker bash script that if run in parallel 6
times on a single NFS mounted client causes the assert problem with
Ganesha NFS. In a 6 hour run, I saw between 2 to 7 asserts per hour
(my upstart script restarted Ganesha NFS after each assert) . The
start and worker scripts are at the bottom of this post. I was using
NFS 4.2 on the client mount. The Ganesha NFS backstore file system was
ext4 and I was using the VFS FSAL.
Each worker process creates 1000 uniquely named directories and in
each creates 11 256K files. It then does a "rm -rf" on the directory
tree and starts over.
Setup:
The Ganesha NFS server and the NFS client are both bare metal servers
with 8 cores, 48GB RAM, running Ubuntu 14.04 using a single 1GB
Ethernet port for the NFS connection (no bonding). They are both
using the 4.8.2 Linux kernel.
The nfs /etc/fstab entry on the client:
ede-c2-gw01:/var/top /C2-NFS4 nfs4 rw,hard,noauto,vers=4.2 0 0
Info on the nfs mount from the client:
# grep nfs /proc/mounts
ede-c2-gw01:/var/top /C2-NFS4 nfs4
rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.16.2.131,local_lock=none,addr=172.16.2.130
0 0
The ganesha.conf file on the server:
LOG {
components {
ALL = INFO;
}
}
EXPORT_DEFAULTS {
SecType = none, sys;
Protocols = 3, 4;
Transports = TCP;
}
EXPORT {
Export_ID = 43;
Path = /var/top;
Pseudo = /var/top;
Access_Type = RW;
Squash = No_Root_Squash;
FSAL {
Name = VFS;
}
}
Ganesha version:
# ganesha.nfsd -v
nfs-ganesha compiled on Oct 19 2016 at 20:08:06
Release = V2.4.0.3
Release comment = GANESHA file server is 64 bits compliant and
supports NFS v3,4.0,4.1 (pNFS) and 9P
Git HEAD = 0f55a9a97a4bf232fb0e42542e4ca7491fbf84ce
Git Describe = V2.4.0.3-0-g0f55a9a
I have made 1 minor modification to the version of ganesha.nfsd used
in this test to the file "src/SAL/state_misc.c" in the
dec_state_owner_ref() function, to have it log the value of refcount
if it is less then 0, before the "assert(refcount > 0)" call as shown:
if (! (refcount > 0)) {
LogInfo(COMPONENT_STATE, "EDE going to assert, refcount
now=%" PRId32 " {%s}", refcount, str);
}
assert(refcount > 0);
In multiple cases, but not all cases, the log file has shown more then
1 thread in this code section just before the assert is called. Here
is one example from my Ganesha NFS log file:
26/10/2016 20:24:52 : epoch 58114720 : ede-c2-gw01 :
ganesha.nfsd-2167691[reaper] dec_state_owner_ref :STATE :INFO :EDE
going to assert, refcount now=-1 {}
26/10/2016 20:24:52 : epoch 58114720 : ede-c2-gw01 :
ganesha.nfsd-2167691[work-137] dec_state_owner_ref :STATE :INFO :EDE
going to assert, refcount now=-1 { s?a?}
26/10/2016 20:24:52 : epoch 58114720 : ede-c2-gw01 :
ganesha.nfsd-2167691[work-248] dec_state_owner_ref :STATE :INFO :EDE
going to assert, refcount now=-1 { ?*?}
26/10/2016 20:24:52 : epoch 58114720 : ede-c2-gw01 :
ganesha.nfsd-2167691[work-100] dec_state_owner_ref :STATE :INFO :EDE
going to assert, refcount now=-1 { ?t?}
The start script used to start the 6 worker processes, named start.sh:
#!/bin/bash
./rm-tree-test.sh $1 &> log.1 &
./rm-tree-test.sh $1 &> log.2 &
./rm-tree-test.sh $1 &> log.3 &
./rm-tree-test.sh $1 &> log.4 &
./rm-tree-test.sh $1 &> log.5 &
./rm-tree-test.sh $1 &> log.6 &
The worker script, named rm-tree-test.sh:
#!/bin/bash -x
# usage: ./rm-tree-test.sh NFS_MOUNT_POINT
# example: ./rm-tree-test.sh /NFS_TEST
# Set number of work directories, number of files per dir and file size
DIRS=1000
FILES=11
FILESIZE="256K"
if [ -z "$1" ]
then
echo "Must provide the mountpoint, aborting"
exit 2
fi
mountpoint -q $1
if [ $? -eq 0 ]
then
echo "$1 is the mount point we will be using"
else
echo "$1 is not a mount point, aborting"
exit 2
fi
HOSTNAME=`hostname`
echo $HOSTNAME
DATE=`date +%F_%H_%M_%S`
TOPDIR="$1/top/$HOSTNAME/${DATE}_$$"
TDIR="$TOPDIR/Month/Day/Hour"
echo "$TOPDIR, $TDIR"
cat /proc/mounts | grep nfs
while [ true ]
do
date
df $1
for D in `seq $DIRS`
do
mkdir -p "$TDIR/Data_$D"
for F in `seq $FILES`
do
dd if=/dev/zero of="$TDIR/Data_$D/dataset_$F.dat" bs=$FILESIZE count=1
done
done
echo "Remove tree"
date
df $1
ls -l $TOPDIR
du -sh $TOPDIR
time rm -rf $TOPDIR
if [ -d $TOPDIR ]
then
echo "ERROR: rm -rf failed"
ls -lR $TOPDIR
fi
done
To start the test, make sure the two scripts are in the same directory
on the client and are executable. Make sure the client has mounted
the server using NFS 4.2 and start the test by passing the NFS mount
point to the start.sh script:
./start.sh /NFS_MOUNT_POINT
Each worker process will log to an individual log file in the local
directory. Each worker process will create about 2.7 GB of data, so
for 6 workers, you should have at least 20GB of free space on the NFS
server.
Thanks,
Eric
On Wed, Oct 26, 2016 at 12:21 AM, Malahal Naineni <mala...@gmail.com> wrote:
>
> Please post if you have an easy reproducer. We will try to recreate
> and root cause it.
>
> On Wed, Oct 26, 2016 at 6:15 AM, Eric Eastman
> <eric.east...@keepertech.com> wrote:
> > A little more info on this issue. I did a 24 hour run of my test
> > using the POSIX FSAL with an ext4 file system as the backstore, and
> > saw 9 asserts during this test run, all caused by the variable
> > "refcount" ending up at -1. The errors seem to be occurring while
> > running "rm -rf" on a directory with 1000 sub-directories, with each
> > having 11 files in it.
> >
> > This looks to me like a race condition and I am having issues finding
> > the root cause reading through the source code. There are notes from
> > commit e7307c5, dated Jan 5 2016, on "Resolve race between
> > get_state_owner and dec_state_owner_ref differently" so this looks
> > like an area that there has been issues before.
> >
> > If anyone has an idea on what the root problem is or where to look,
> > please let me know, as we cannot use Ganesha NFS if it is going to
> > assert during production.
> >
> > Thanks,
> > Eric
> >
> > On Thu, Oct 20, 2016 at 1:22 AM, Eric Eastman
> > <eric.east...@keepertech.com> wrote:
> >> While testing Ganesha NFS V2.4.0.3 using the CEPH FSAL to a ceph file
> >> system, I am seeing the ganesha.nfsd process die due to an assert call
> >> multiple times per hour. I have also seen it die at the same place in
> >> the code using the VFS FSAL with a ext4 file system, but it dies much
> >> less often.
> >>
> >> It is dying at line 917 in src/SAL/state_misc.c, which is called by
> >> src/SAL/state_misc.c at line 1010. The assert call is in
> >> dec_state_owner_ref() at the line:
> >>
> >> assert(refcount > 0);
> >>
> >> Looking at the core files and adding in some debugging code confirms
> >> that refcount is -1 when the assert call is made.
> >>
> >> It looks like the owner count is trying to go to -1 in
> >> uncache_nfs4_owner(), but as it occurs only on occasions, I think it
> >> is a race condition.
> >>
> >> Info on the build:
> >>
> >> Host OS is Ubuntu 14.04 with a 4.8.2 x86_64 kernel on a 8 processor system
> >>
> >> Cmake command:
> >> # cmake -DCMAKE_INSTALL_PREFIX=/opt/keeper -DALLOCATOR=jemalloc
> >> -DUSE_ADMIN_TOOLS=ON -DUSE_DBUS=ON ../src
> >>
> >> # ganesha.nfsd -v
> >> ganesha.nfsd compiled on Oct 17 2016 at 16:50:18
> >> Release = V2.4.0.3
> >> Release comment = GANESHA file server is 64 bits compliant and
> >> supports NFS v3,4.0,4.1 (pNFS) and 9P
> >> Git HEAD = 0f55a9a97a4bf232fb0e42542e4ca7491fbf84ce
> >> Git Describe = V2.4.0.3-0-g0f55a9a
> >>
> >> # ceph -v
> >> ceph version 10.2.3 (ecc23778eb545d8dd55e2e4735b53cc93f92e65b)
> >>
> >> # cat ganesha.conf
> >> LOG {
> >> components {
> >> ALL = INFO;
> >> }
> >> }
> >>
> >> EXPORT_DEFAULTS {
> >> SecType = none, sys;
> >> Protocols = 3, 4;
> >> Transports = TCP;
> >> }
> >>
> >> # define CephFS export
> >> EXPORT {
> >> Export_ID = 42;
> >> Path = /top;
> >> Pseudo = /top;
> >> Access_Type = RW;
> >> Squash = No_Root_Squash;
> >> FSAL {
> >> Name = CEPH;
> >> }
> >> }
> >>
> >> The VFS export for the ext4 tests was:
> >>
> >> # define CephFS export
> >> EXPORT {
> >> Export_ID = 43;
> >> Path = /var/top;
> >> Pseudo = /var/top;
> >> Access_Type = RW;
> >> Squash = No_Root_Squash;
> >> FSAL {
> >> Name = VFS;
> >> }
> >> }
> >>
> >> The test was 2 Ubuntu 14.04 NFS clients each having 6 processes,
> >> writing 11,000 256k files in separate directory trees with 11 files
> >> per lowest level node. On each Ubuntu client, 3 processes wrote to a
> >> NFS 3 mount and 3 wrote to a NFS 4 mount. The files are then read and
> >> verified, deleted, and the test restarts.
> >>
> >> Regards,
> >> Eric
> >
> > ------------------------------------------------------------------------------
> > The Command Line: Reinvented for Modern Developers
> > Did the resurgence of CLI tooling catch you by surprise?
> > Reconnect with the command line and become more productive.
> > Learn the new .NET and ASP.NET CLI. Get your free copy!
> > http://sdm.link/telerik
> > _______________________________________________
> > Nfs-ganesha-devel mailing list
> > Nfs-ganesha-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
