subject:"NFSv4 performance degradation with 12.0\-CURRENT client"

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-27 Thread Konstantin Belousov

On Sat, Nov 26, 2016 at 11:19:19PM +, Rick Macklem wrote:
> Konstantin Belousov wrote:
> [stuff snipped]
> >I thought that the issue was in tracking any opens and mmaps, but from this
> >reply it is not that clear.  Do you need callback when all opens and mmaps
> >have ended, or only opens and mmaps for write ?  If later, we already have
> >a suitable mechanism VOP_ADD_WRITECOUNT().
> 
> Not quite. The NFSv4 client needs to Close the NFSv4 Open after all I/O on
> the file has been done. This applies to both reads and writes.
> Since mmap'd files can generate I/O after the VOP_CLOSE(), the NFSv4 client
> can't do the NFSv4 Close in VOP_CLOSE().
> Since it can't do it then, it waits until VOP_INACTIVE() to do the NFSv4 
> Close.
> 
> This might be improved by:
> - A flag that indicates that an open file descriptor has been mmap()d, which
>   VOP_CLOSE() could check to decide if it can do the NFSv4 Close.
>   (ie. It could do the NFSv4 Close if the file descriptor hasn't been 
> mmap()d.)
> - If the system knows when mmap()d I/O is done (the process has terminated?),
>   it could do a VOP_MMAP_IO_DONE() and the NFSv4 client would do the NFSv4 
> Close
>   in it.
>   --> I don't know if this is feasible and I suspect if it could be done, 
> that it would
> usually happen just before VOP_INACTIVE(). { This case of nullfs 
> caching was an
> exception, I think? }
> 
> Does this clarify it? rick

Thank you, yes, it clarifies the things, and makes it clear that my idea
is not feasible.

It is not hard to count number of mappings for the vnode object, but I
do not want to do this by straight-forward addition of the counter to
the vnode or vm object, since that would significantly increase amount
of memory used by VFS.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-26 Thread Rick Macklem

Alan Somers wrote:
[stuff snipped]
>Mounting nullfs with the nocache option, ad kib suggested, fixed the
>problem.  Also, applying kib's patch and then mounting nullfs with
>default options also fixed the problem.  Here is the nfsstat output
>for "ls -al" when using kib's patch.  Notice the client has far fewer
>opens:
I did a quick test which confirmed that the opens get closed when the "nocache"
option is used on the nullfs mount as well.

Kostik, I think your patch is a good idea and you can consider it reviewed by me
if you'd like.

I also did a quick test wth unionfs and it did not accumulate opens, so it 
doesn't
seem to suffer from this problem. (It does have issues, as noted by the BUGS
section of the mount_unionfs man page.)

rick

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-26 Thread Rick Macklem

Konstantin Belousov wrote:
[stuff snipped]
>I thought that the issue was in tracking any opens and mmaps, but from this
>reply it is not that clear.  Do you need callback when all opens and mmaps
>have ended, or only opens and mmaps for write ?  If later, we already have
>a suitable mechanism VOP_ADD_WRITECOUNT().

Not quite. The NFSv4 client needs to Close the NFSv4 Open after all I/O on
the file has been done. This applies to both reads and writes.
Since mmap'd files can generate I/O after the VOP_CLOSE(), the NFSv4 client
can't do the NFSv4 Close in VOP_CLOSE().
Since it can't do it then, it waits until VOP_INACTIVE() to do the NFSv4 Close.

This might be improved by:
- A flag that indicates that an open file descriptor has been mmap()d, which
  VOP_CLOSE() could check to decide if it can do the NFSv4 Close.
  (ie. It could do the NFSv4 Close if the file descriptor hasn't been mmap()d.)
- If the system knows when mmap()d I/O is done (the process has terminated?),
  it could do a VOP_MMAP_IO_DONE() and the NFSv4 client would do the NFSv4 Close
  in it.
  --> I don't know if this is feasible and I suspect if it could be done, that 
it would
usually happen just before VOP_INACTIVE(). { This case of nullfs 
caching was an
exception, I think? }

Does this clarify it? rick

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-25 Thread Alan Somers

On Fri, Nov 25, 2016 at 6:57 AM, Konstantin Belousov
 wrote:
> On Fri, Nov 25, 2016 at 12:54:07PM +, Rick Macklem wrote:
>> Well, ideally theer would be a VOP_MMAPDONE() or something like that, which
>> would tell the NFSv4 client that I/O is done on the vnode so it can close it.
>> If there was some way for the NFSv4 VOP_CLOSE() to be able to tell if the 
>> file
>> has been mmap'd, that would help since it could close the ones that are not
>> mmap'd on the last descriptor close.
>> (A counter wouldn't be as useful, since NFSv4 would have to keep checking it 
>> to
>>  see if it can do the close yet, but it might still be doable.)
>
> I thought that the issue was in tracking any opens and mmaps, but from this
> reply it is not that clear.  Do you need callback when all opens and mmaps
> have ended, or only opens and mmaps for write ?  If later, we already have
> a suitable mechanism VOP_ADD_WRITECOUNT().

Mounting nullfs with the nocache option, ad kib suggested, fixed the
problem.  Also, applying kib's patch and then mounting nullfs with
default options also fixed the problem.  Here is the nfsstat output
for "ls -al" when using kib's patch.  Notice the client has far fewer
opens:
nfsstat -s -e -z

Server Info:
  Getattr   SetattrLookup  Readlink  Read WriteCreateRemove
  494 0 0 0 0 1 0 0
   Rename  Link   Symlink Mkdir Rmdir   Readdir  RdirPlusAccess
0 0 0 0 0 0 0 0
MknodFsstatFsinfo  PathConfCommit   LookupP   SetClId SetClIdCf
0 0 0 0 0 0 0 0
 Open  OpenAttr OpenDwnGr  OpenCfrm DelePurge   DeleRet GetFH  Lock
0 0 0 0 0 0 0 0
LockT LockU CloseVerify   NVerify PutFH  PutPubFH PutRootFH
0 0 0 0 0   494 0 0
Renew RestoreFHSaveFH   Secinfo RelLckOwn  V4Create
0 0 0 0 0 0
Server:
RetfailedFaults   Clients
0 0 0
OpenOwner Opens LockOwner LocksDelegs
0 0 0 0 0
Server Cache Stats:
   Inprog  Idem  Non-idemMisses CacheSize   TCPPeak
0 0 0   495 17280 17280
nfsstat -c -e -z
Client Info:
Rpc Counts:
  Getattr   SetattrLookup  Readlink  Read WriteCreateRemove
   14 0 0 0 0 0 0 0
   Rename  Link   Symlink Mkdir Rmdir   Readdir  RdirPlusAccess
0 0 0 0 0 0 0 0
MknodFsstatFsinfo  PathConfCommit   SetClId SetClIdCf  Lock
0 0 0 0 0 0 0 0
LockT LockU  Open   OpenCfr
0 0 0 0
OpenOwner Opens LockOwner LocksDelegs  LocalOwn LocalOpen LocalLOwn
  592   588 0 0 0 0 0 0
LocalLock
0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
0 0 0 0   494
Cache Info:
Attr HitsMisses Lkup HitsMisses BioR HitsMisses BioW HitsMisses
 143912   960 0 0 0 0 0
BioRLHitsMisses BioD HitsMisses DirE HitsMisses
1 0 4 0 1 0

Thanks for the help, guys.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-25 Thread Rick Macklem

Konstantin Belousov wrote:

>On Thu, Nov 24, 2016 at 10:45:51PM +, Rick Macklem wrote:
>> asom...@gmail.com wrote:
>> >OpenOwner Opens LockOwner LocksDelegs  LocalOwn LocalOpen 
>> >LocalLOwn
>> > 5638141453 0 0 0 0 0   
>> >   0
>> Ok, I think this shows us the problem. 141453 opens is a lot and the client 
>> would have
>> to chek these every time another open is done (there goes all that CPU;-).
>>
>> Now, why has this occurred?
>> Well, the NFSv4 client can't close NFSv4 Opens on a vnode until that vnode's
>> v_usecount goes to 0. This is because mmap'd files might do I/O after the 
>> file
>> descriptor is closed.
>> Now, hopefully Kostik will know something about nullfs and can help with 
>> this.
>> My guess is that nullfs ends up acquiring a refcnt on the NFS vnode so the
>> v_usecount doesn't go to 0 and, therefore, the client never closes the NFSv4 
>> Opens.
>> Kostik, do you know if this is the case and whether or not it can be changed?
>You are absolutely right. Nullfs vnode keeps a reference to the lower
>vnode which is below the nullfs one, i.e. to the nfs vnode in this case.
>If cache option is specified for the nullfs mount (default), the nullfs
>vnodes are cached normally to avoid the cost of creating and destroying
>nullfs vnode on each operation, and related cost of the exclusive locks
>on the lower vnode.
>
>An answer to my question in the previous mail to try with nocache
>option would give the confirmation. Really, I suspected that v_hash
>is calculated differently for NFSv3 and v4 mounts, but if opens are
>accumulated until use ref is dropped, that would explain things as well.
Hopefully Alan can test this and let us know if "nocache" on the nullfs mount
fixes the problem.

>Assuming your diagnosis is correct, are you in fact stating that the
>current VFS KPI is flawed ?  It sounds as if either some another callback
>or counter needs to exist to track number of mapping references to the
>vm object of the vnode, in addition to VOP_OPEN/VOP_CLOSE ?
>
>Currently a rough estimation of the number of mappings, which is sometimes
>slightly wrong, can be obtained by the expression
>vp->v_object->ref_count - vp->v_object->shadow_count
Well, ideally theer would be a VOP_MMAPDONE() or something like that, which
would tell the NFSv4 client that I/O is done on the vnode so it can close it.
If there was some way for the NFSv4 VOP_CLOSE() to be able to tell if the file
has been mmap'd, that would help since it could close the ones that are not
mmap'd on the last descriptor close.
(A counter wouldn't be as useful, since NFSv4 would have to keep checking it to
 see if it can do the close yet, but it might still be doable.)
>
>> >LocalLock
>> >0
>> >Rpc Info:
>> >TimedOut   Invalid X Replies   Retries  Requests
>> >0 0 0 0   662
>> >Cache Info:
>> >Attr HitsMisses Lkup HitsMisses BioR HitsMisses BioW Hits
>> >Misses
>> > 127558   837   121 0 0 0   
>> >   0
>> >BioRLHitsMisses BioD HitsMisses DirE HitsMisses
>> >1 0 6 0 1 0
>> >
>> [more stuff snipped]
>> >What role could nullfs be playing?
>> As noted above, my hunch is that is acquiring a refcnt on the NFS client 
>> vnode such
>> that the v_usecount doesn't go to zero (at least for a long time) and without
>> a VOP_INACTIVE() on the NFSv4 vnode, the NFSv4 Opens don't get closed and
>> accumulate.
>> (If that isn't correct, it is somehow interfering with the client Closing 
>> the NFSv4 Opens
>>  in some other way.)
>>
>The following patch should automatically unset cache option for nullfs
>mounts over NFSv4 filesystem.
>
>diff --git a/sys/fs/nfsclient/nfs_clvfsops.c b/sys/fs/nfsclient/nfs_clvfsops.c
>index 524a372..a7e9fe3 100644
>--- a/sys/fs/nfsclient/nfs_clvfsops.c
>+++ b/sys/fs/nfsclient/nfs_clvfsops.c
>@@ -1320,6 +1320,8 @@ out:
>MNT_ILOCK(mp);
>mp->mnt_kern_flag |= MNTK_LOOKUP_SHARED | MNTK_NO_IOPF |
>MNTK_USES_BCACHE;
>+   if ((VFSTONFS(mp)->nm_flag & NFSMNT_NFSV4) != 0)
>+   mp->mnt_kern_flag |= MNTK_NULL_NOCACHE;
>MNT_IUNLOCK(mp);
>}
>return (error);
>diff --git a/sys/fs/nullfs/null_vfsops.c b/sys/fs/nullfs/null_vfsops.c
>index 49bae28..de05e8b 100644
>--- a/sys/fs/nullfs/null_vfsops.c
>+++ b/sys/fs/nullfs/null_vfsops.c
>@@ -188,7 +188,8 @@ nullfs_mount(struct mount *mp)
>}
>
>   xmp->nullm_flags |= NULLM_CACHE;
>-   if (vfs_getopt(mp->mnt_optnew, "nocache", NULL, NULL) == 0)
>+   if (vfs_getopt(mp->mnt_optnew, "nocache", NULL, NULL) == 0 ||
>+   (xmp->nullm_vfs->mnt_kern_flag & MNTK_NULL_NOCACHE) != 0)
>xmp->nullm_flags &= ~NULLM_CACHE;
>
>MNT_ILOCK(mp);
>diff --git a/sys/sys/mount.h b/sys/sys/mount.h
>index 94cabb6..b6f9fec 100644

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-25 Thread O. Hartmann

Am Thu, 24 Nov 2016 22:35:42 +0200
Konstantin Belousov  schrieb:

> On Thu, Nov 24, 2016 at 11:42:41AM -0700, Alan Somers wrote:
> > On Thu, Nov 24, 2016 at 5:53 AM, Rick Macklem  wrote: 
> >  
> > >
> > > On Wed, Nov 23, 2016 at 10:17:25PM -0700, Alan Somers wrote:  
> > >> I have a FreeBSD 10.3-RELEASE-p12 server exporting its home
> > >> directories over both NFSv3 and NFSv4.  I have a TrueOS client (based
> > >> on 12.0-CURRENT on the drm-next-4.7 branch, built on 28-October)
> > >> mounting the home directories over NFSv4.  At first, everything is
> > >> fine and performance is good.  But if the client does a buildworld
> > >> using sources on NFS and locally stored objects, performance slowly
> > >> degrades.  The degradation is most noticeable with metadata-heavy
> > >> operations.  For example, "ls -l" in a directory with 153 files takes
> > >> less than 0.1 seconds right after booting.  But the longer the
> > >> buildworld goes on, the slower it gets.  Eventually that same "ls -l"
> > >> takes 19 seconds.  When the home directories are mounted over NFSv3
> > >> instead, I see no degradation.
> > >>
> > >> top shows negligible CPU consumption on the server, and very high
> > >> consumption on the client when using NFSv4 (nearly 100%).  The
> > >> NFS-using process is spending almost all of its time in system mode,
> > >> and dtrace shows that almost all of its time is spent in
> > >> ncl_getpages().
> > >>  
> > > A couple of things you could do when it slow (as well as what Kostik 
> > > suggested):
> > > - nfsstat -c -e on client and nfsstat -e -s on server, to see what RPCs 
> > > are being
> > > done and how quickly. (nfsstat -s -e will also show you how big the DRC 
> > > is,
> > > although a large DRC should show up as increased CPU consumption on the 
> > > server)
> > > - capture packets with tcpdump -s 0 -w test.pcap host 
> > >   - then you can email me test.pcap as an attachment. I can look at it in 
> > > wireshark
> > > and see if there seem to protocol and/or TCP issues. (You can look at 
> > > in
> > > wireshark yourself, the look for NFS4ERR_xxx, TCP segment retransmits...)
> > >
> > > If you are using either "intr" or "soft" on the mounts, try without those 
> > > mount
> > > options. (The Bugs section of mount_nfs recommends against using them. If 
> > > an RPC
> > > fails due to these options, something called a seqid# can be "out of 
> > > sync" between
> > > client/server and that causes serious problems.)  
> > > --> These seqid#s are not used by NFSv4.1, so you could try that by 
> > > adding  
> > >   "minorversion=1" to your mount options.
> > >
> > > Good luck with it, rick  
> > 
> > I've reproduced the issue on stock FreeBSD 12, and I've also learned
> > that nullfs is a required factor.  Doing the buildworld directly on
> > the NFS mount doesn't cause any slowdown, but doing a buildworld on
> > the nullfs copy of the NFS mount does.  The slowdown affects the base
> > NFS mount as well as the nullfs copy.  Here is the nfsstat output for
> > both server and client duing "ls -al" on the client:
> > 
> > nfsstat -e -s -z
> > 
> > Server Info:
> >   Getattr   SetattrLookup  Readlink  Read WriteCreate
> > Remove
> >   800 0   121 0 0 2 0   
> >   0
> >Rename  Link   Symlink Mkdir Rmdir   Readdir  RdirPlus
> > Access
> > 0 0 0 0 0 0 0   
> >   8
> > MknodFsstatFsinfo  PathConfCommit   LookupP   SetClId 
> > SetClIdCf
> > 0 0 0 0 1 3 0   
> >   0
> >  Open  OpenAttr OpenDwnGr  OpenCfrm DelePurge   DeleRet GetFH  
> > Lock
> > 0 0 0 0 0 0   123   
> >   0
> > LockT LockU CloseVerify   NVerify PutFH  PutPubFH 
> > PutRootFH
> > 0 0 0 0 0   674 0   
> >   0
> > Renew RestoreFHSaveFH   Secinfo RelLckOwn  V4Create
> > 0 0 0 0 0 0
> > Server:
> > RetfailedFaults   Clients
> > 0 0 0
> > OpenOwner Opens LockOwner LocksDelegs
> > 0 0 0 0 0
> > Server Cache Stats:
> >Inprog  Idem  Non-idemMisses CacheSize   TCPPeak
> > 0 0 0   674 16738 16738
> > 
> > nfsstat -e -c -z
> > Client Info:
> > Rpc Counts:
> >   Getattr   SetattrLookup  Readlink  Read WriteCreate
> > Remove
> >60 0   119 0 0 0 0   
> >   0
> >Rename  Link   Symlink Mkdir Rmdir   Readdir  RdirPlus
> > Access
> > 0 0 0 0 0 0 0   
> >   3
> > MknodFsstatFsinfo

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-25 Thread Konstantin Belousov

On Fri, Nov 25, 2016 at 12:54:07PM +, Rick Macklem wrote:
> Well, ideally theer would be a VOP_MMAPDONE() or something like that, which
> would tell the NFSv4 client that I/O is done on the vnode so it can close it.
> If there was some way for the NFSv4 VOP_CLOSE() to be able to tell if the file
> has been mmap'd, that would help since it could close the ones that are not
> mmap'd on the last descriptor close.
> (A counter wouldn't be as useful, since NFSv4 would have to keep checking it 
> to
>  see if it can do the close yet, but it might still be doable.)

I thought that the issue was in tracking any opens and mmaps, but from this
reply it is not that clear.  Do you need callback when all opens and mmaps
have ended, or only opens and mmaps for write ?  If later, we already have
a suitable mechanism VOP_ADD_WRITECOUNT().
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-25 Thread Konstantin Belousov

On Thu, Nov 24, 2016 at 10:45:51PM +, Rick Macklem wrote:
> asom...@gmail.com wrote:
> >OpenOwner Opens LockOwner LocksDelegs  LocalOwn LocalOpen 
> >LocalLOwn
> > 5638141453 0 0 0 0 0
> >  0
> Ok, I think this shows us the problem. 141453 opens is a lot and the client 
> would have
> to chek these every time another open is done (there goes all that CPU;-).
> 
> Now, why has this occurred?
> Well, the NFSv4 client can't close NFSv4 Opens on a vnode until that vnode's
> v_usecount goes to 0. This is because mmap'd files might do I/O after the file
> descriptor is closed.
> Now, hopefully Kostik will know something about nullfs and can help with this.
> My guess is that nullfs ends up acquiring a refcnt on the NFS vnode so the
> v_usecount doesn't go to 0 and, therefore, the client never closes the NFSv4 
> Opens.
> Kostik, do you know if this is the case and whether or not it can be changed?
You are absolutely right. Nullfs vnode keeps a reference to the lower
vnode which is below the nullfs one, i.e. to the nfs vnode in this case.
If cache option is specified for the nullfs mount (default), the nullfs
vnodes are cached normally to avoid the cost of creating and destroying
nullfs vnode on each operation, and related cost of the exclusive locks
on the lower vnode.

An answer to my question in the previous mail to try with nocache
option would give the confirmation. Really, I suspected that v_hash
is calculated differently for NFSv3 and v4 mounts, but if opens are
accumulated until use ref is dropped, that would explain things as well.

Assuming your diagnosis is correct, are you in fact stating that the
current VFS KPI is flawed ?  It sounds as if either some another callback
or counter needs to exist to track number of mapping references to the
vm object of the vnode, in addition to VOP_OPEN/VOP_CLOSE ?

Currently a rough estimation of the number of mappings, which is sometimes
slightly wrong, can be obtained by the expression
vp->v_object->ref_count - vp->v_object->shadow_count


> >LocalLock
> >0
> >Rpc Info:
> >TimedOut   Invalid X Replies   Retries  Requests
> >0 0 0 0   662
> >Cache Info:
> >Attr HitsMisses Lkup HitsMisses BioR HitsMisses BioW Hits
> >Misses
> > 127558   837   121 0 0 0
> >  0
> >BioRLHitsMisses BioD HitsMisses DirE HitsMisses
> >1 0 6 0 1 0
> >
> [more stuff snipped]
> >What role could nullfs be playing?
> As noted above, my hunch is that is acquiring a refcnt on the NFS client 
> vnode such
> that the v_usecount doesn't go to zero (at least for a long time) and without
> a VOP_INACTIVE() on the NFSv4 vnode, the NFSv4 Opens don't get closed and
> accumulate.
> (If that isn't correct, it is somehow interfering with the client Closing the 
> NFSv4 Opens
>  in some other way.)

The following patch should automatically unset cache option for nullfs
mounts over NFSv4 filesystem.

diff --git a/sys/fs/nfsclient/nfs_clvfsops.c b/sys/fs/nfsclient/nfs_clvfsops.c
index 524a372..a7e9fe3 100644
--- a/sys/fs/nfsclient/nfs_clvfsops.c
+++ b/sys/fs/nfsclient/nfs_clvfsops.c
@@ -1320,6 +1320,8 @@ out:
MNT_ILOCK(mp);
mp->mnt_kern_flag |= MNTK_LOOKUP_SHARED | MNTK_NO_IOPF |
MNTK_USES_BCACHE;
+   if ((VFSTONFS(mp)->nm_flag & NFSMNT_NFSV4) != 0)
+   mp->mnt_kern_flag |= MNTK_NULL_NOCACHE;
MNT_IUNLOCK(mp);
}
return (error);
diff --git a/sys/fs/nullfs/null_vfsops.c b/sys/fs/nullfs/null_vfsops.c
index 49bae28..de05e8b 100644
--- a/sys/fs/nullfs/null_vfsops.c
+++ b/sys/fs/nullfs/null_vfsops.c
@@ -188,7 +188,8 @@ nullfs_mount(struct mount *mp)
}
 
xmp->nullm_flags |= NULLM_CACHE;
-   if (vfs_getopt(mp->mnt_optnew, "nocache", NULL, NULL) == 0)
+   if (vfs_getopt(mp->mnt_optnew, "nocache", NULL, NULL) == 0 ||
+   (xmp->nullm_vfs->mnt_kern_flag & MNTK_NULL_NOCACHE) != 0)
xmp->nullm_flags &= ~NULLM_CACHE;
 
MNT_ILOCK(mp);
diff --git a/sys/sys/mount.h b/sys/sys/mount.h
index 94cabb6..b6f9fec 100644
--- a/sys/sys/mount.h
+++ b/sys/sys/mount.h
@@ -370,7 +370,8 @@ void  __mnt_vnode_markerfree_active(struct vnode 
**mvp, struct mount *);
 #defineMNTK_SUSPEND0x0800  /* request write suspension */
 #defineMNTK_SUSPEND2   0x0400  /* block secondary writes */
 #defineMNTK_SUSPENDED  0x1000  /* write operations are 
suspended */
-#defineMNTK_UNUSED10x2000
+#defineMNTK_NULL_NOCACHE   0x2000 /* auto disable cache for 
nullfs
+ mounts over this fs */
 #define MNTK_LOOKUP_SHARED 0x4000 /* FS supports shared lock lookups */
 #defineMNTK_NOKNOTE0x8000

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-24 Thread Rick Macklem

asom...@gmail.com wrote:
[stuff snipped]
>I've reproduced the issue on stock FreeBSD 12, and I've also learned
>that nullfs is a required factor.  Doing the buildworld directly on
>the NFS mount doesn't cause any slowdown, but doing a buildworld on
>the nullfs copy of the NFS mount does.  The slowdown affects the base
>NFS mount as well as the nullfs copy.  Here is the nfsstat output for
>both server and client duing "ls -al" on the client:
>
>nfsstat -e -s -z
If you do this again, avoid using "-z" and I think you'll see the Opens (below 
Server:)
going up and up...
>
>Server Info:
>  Getattr   SetattrLookup  Readlink  Read WriteCreateRemove
>  800 0   121 0 0 2 0 0
>   Rename  Link   Symlink Mkdir Rmdir   Readdir  RdirPlusAccess
>0 0 0 0 0 0 0 8
>MknodFsstatFsinfo  PathConfCommit   LookupP   SetClId SetClIdCf
>   0 0 0 0 1 3 0 0
> Open  OpenAttr OpenDwnGr  OpenCfrm DelePurge   DeleRet GetFH  Lock
>0 0 0 0 0 0   123 0
>LockT LockU CloseVerify   NVerify PutFH  PutPubFH PutRootFH
>0 0 0 0 0   674 0 0
>Renew RestoreFHSaveFH   Secinfo RelLckOwn  V4Create
>0 0 0 0 0 0
>Server:
>RetfailedFaults   Clients
>0 0 0
>OpenOwner Opens LockOwner LocksDelegs
>0 0 0 0 0
Oops, I think this is an nfsstats bug. I don't normally use "-z", so I didn't 
notice
it clears these counts and it probably should not, since they are "how many of
these that are currently allocated".
I'll check this. (Not relevant to this issue, but needs fixin.;-)
>Server Cache Stats:
>   Inprog  Idem  Non-idemMisses CacheSize   TCPPeak
>0 0 0   674 16738 16738
>
>nfsstat -e -c -z
>Client Info:
>Rpc Counts:
> Getattr   SetattrLookup  Readlink  Read WriteCreateRemove
>   60 0   119 0 0 0 0 0
>   Rename  Link   Symlink Mkdir Rmdir   Readdir  RdirPlusAccess
>0 0 0 0 0 0 0 3
>MknodFsstatFsinfo  PathConfCommit   SetClId SetClIdCf  Lock
>0 0 0 0 0 0 0 0
>LockT LockU  Open   OpenCfr
>0 0 0 0
>OpenOwner Opens LockOwner LocksDelegs  LocalOwn LocalOpen LocalLOwn
> 5638141453 0 0 0 0 0 0
Ok, I think this shows us the problem. 141453 opens is a lot and the client 
would have
to chek these every time another open is done (there goes all that CPU;-).

Now, why has this occurred?
Well, the NFSv4 client can't close NFSv4 Opens on a vnode until that vnode's
v_usecount goes to 0. This is because mmap'd files might do I/O after the file
descriptor is closed.
Now, hopefully Kostik will know something about nullfs and can help with this.
My guess is that nullfs ends up acquiring a refcnt on the NFS vnode so the
v_usecount doesn't go to 0 and, therefore, the client never closes the NFSv4 
Opens.
Kostik, do you know if this is the case and whether or not it can be changed?
>LocalLock
>0
>Rpc Info:
>TimedOut   Invalid X Replies   Retries  Requests
>0 0 0 0   662
>Cache Info:
>Attr HitsMisses Lkup HitsMisses BioR HitsMisses BioW HitsMisses
> 127558   837   121 0 0 0 0
>BioRLHitsMisses BioD HitsMisses DirE HitsMisses
>1 0 6 0 1 0
>
[more stuff snipped]
>What role could nullfs be playing?
As noted above, my hunch is that is acquiring a refcnt on the NFS client vnode 
such
that the v_usecount doesn't go to zero (at least for a long time) and without
a VOP_INACTIVE() on the NFSv4 vnode, the NFSv4 Opens don't get closed and
accumulate.
(If that isn't correct, it is somehow interfering with the client Closing the 
NFSv4 Opens
 in some other way.)

rick

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-24 Thread Konstantin Belousov

On Thu, Nov 24, 2016 at 11:42:41AM -0700, Alan Somers wrote:
> On Thu, Nov 24, 2016 at 5:53 AM, Rick Macklem  wrote:
> >
> > On Wed, Nov 23, 2016 at 10:17:25PM -0700, Alan Somers wrote:
> >> I have a FreeBSD 10.3-RELEASE-p12 server exporting its home
> >> directories over both NFSv3 and NFSv4.  I have a TrueOS client (based
> >> on 12.0-CURRENT on the drm-next-4.7 branch, built on 28-October)
> >> mounting the home directories over NFSv4.  At first, everything is
> >> fine and performance is good.  But if the client does a buildworld
> >> using sources on NFS and locally stored objects, performance slowly
> >> degrades.  The degradation is most noticeable with metadata-heavy
> >> operations.  For example, "ls -l" in a directory with 153 files takes
> >> less than 0.1 seconds right after booting.  But the longer the
> >> buildworld goes on, the slower it gets.  Eventually that same "ls -l"
> >> takes 19 seconds.  When the home directories are mounted over NFSv3
> >> instead, I see no degradation.
> >>
> >> top shows negligible CPU consumption on the server, and very high
> >> consumption on the client when using NFSv4 (nearly 100%).  The
> >> NFS-using process is spending almost all of its time in system mode,
> >> and dtrace shows that almost all of its time is spent in
> >> ncl_getpages().
> >>
> > A couple of things you could do when it slow (as well as what Kostik 
> > suggested):
> > - nfsstat -c -e on client and nfsstat -e -s on server, to see what RPCs are 
> > being done
> >   and how quickly. (nfsstat -s -e will also show you how big the DRC is, 
> > although a
> >   large DRC should show up as increased CPU consumption on the server)
> > - capture packets with tcpdump -s 0 -w test.pcap host 
> >   - then you can email me test.pcap as an attachment. I can look at it in 
> > wireshark
> > and see if there seem to protocol and/or TCP issues. (You can look at 
> > in wireshark
> > yourself, the look for NFS4ERR_xxx, TCP segment retransmits...)
> >
> > If you are using either "intr" or "soft" on the mounts, try without those 
> > mount options.
> > (The Bugs section of mount_nfs recommends against using them. If an RPC 
> > fails due to
> >  these options, something called a seqid# can be "out of sync" between 
> > client/server and
> >  that causes serious problems.)
> > --> These seqid#s are not used by NFSv4.1, so you could try that by adding
> >   "minorversion=1" to your mount options.
> >
> > Good luck with it, rick
> 
> I've reproduced the issue on stock FreeBSD 12, and I've also learned
> that nullfs is a required factor.  Doing the buildworld directly on
> the NFS mount doesn't cause any slowdown, but doing a buildworld on
> the nullfs copy of the NFS mount does.  The slowdown affects the base
> NFS mount as well as the nullfs copy.  Here is the nfsstat output for
> both server and client duing "ls -al" on the client:
> 
> nfsstat -e -s -z
> 
> Server Info:
>   Getattr   SetattrLookup  Readlink  Read WriteCreate
> Remove
>   800 0   121 0 0 2 0 > 0
>Rename  Link   Symlink Mkdir Rmdir   Readdir  RdirPlus
> Access
> 0 0 0 0 0 0 0 
> 8
> MknodFsstatFsinfo  PathConfCommit   LookupP   SetClId 
> SetClIdCf
> 0 0 0 0 1 3 0 > 0
>  Open  OpenAttr OpenDwnGr  OpenCfrm DelePurge   DeleRet GetFH  
> Lock
> 0 0 0 0 0 0   123 > 0
> LockT LockU CloseVerify   NVerify PutFH  PutPubFH 
> PutRootFH
> 0 0 0 0 0   674 0 > 0
> Renew RestoreFHSaveFH   Secinfo RelLckOwn  V4Create
> 0 0 0 0 0 0
> Server:
> RetfailedFaults   Clients
> 0 0 0
> OpenOwner Opens LockOwner LocksDelegs
> 0 0 0 0 0
> Server Cache Stats:
>Inprog  Idem  Non-idemMisses CacheSize   TCPPeak
> 0 0 0   674 16738 16738
> 
> nfsstat -e -c -z
> Client Info:
> Rpc Counts:
>   Getattr   SetattrLookup  Readlink  Read WriteCreate
> Remove
>60 0   119 0 0 0 0 > 0
>Rename  Link   Symlink Mkdir Rmdir   Readdir  RdirPlus
> Access
> 0 0 0 0 0 0 0 
> 3
> MknodFsstatFsinfo  PathConfCommit   SetClId SetClIdCf  
> Lock
> 0 0 0 0 0 0 0 > 0
> LockT LockU  Open   OpenCfr
> 0 0 0 0
> OpenOwner Opens LockOwner LocksDelegs  LocalOwn LocalOpen 
> LocalLOwn
>  5638

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-24 Thread Alan Somers

On Thu, Nov 24, 2016 at 5:53 AM, Rick Macklem  wrote:
>
> On Wed, Nov 23, 2016 at 10:17:25PM -0700, Alan Somers wrote:
>> I have a FreeBSD 10.3-RELEASE-p12 server exporting its home
>> directories over both NFSv3 and NFSv4.  I have a TrueOS client (based
>> on 12.0-CURRENT on the drm-next-4.7 branch, built on 28-October)
>> mounting the home directories over NFSv4.  At first, everything is
>> fine and performance is good.  But if the client does a buildworld
>> using sources on NFS and locally stored objects, performance slowly
>> degrades.  The degradation is most noticeable with metadata-heavy
>> operations.  For example, "ls -l" in a directory with 153 files takes
>> less than 0.1 seconds right after booting.  But the longer the
>> buildworld goes on, the slower it gets.  Eventually that same "ls -l"
>> takes 19 seconds.  When the home directories are mounted over NFSv3
>> instead, I see no degradation.
>>
>> top shows negligible CPU consumption on the server, and very high
>> consumption on the client when using NFSv4 (nearly 100%).  The
>> NFS-using process is spending almost all of its time in system mode,
>> and dtrace shows that almost all of its time is spent in
>> ncl_getpages().
>>
> A couple of things you could do when it slow (as well as what Kostik 
> suggested):
> - nfsstat -c -e on client and nfsstat -e -s on server, to see what RPCs are 
> being done
>   and how quickly. (nfsstat -s -e will also show you how big the DRC is, 
> although a
>   large DRC should show up as increased CPU consumption on the server)
> - capture packets with tcpdump -s 0 -w test.pcap host 
>   - then you can email me test.pcap as an attachment. I can look at it in 
> wireshark
> and see if there seem to protocol and/or TCP issues. (You can look at in 
> wireshark
> yourself, the look for NFS4ERR_xxx, TCP segment retransmits...)
>
> If you are using either "intr" or "soft" on the mounts, try without those 
> mount options.
> (The Bugs section of mount_nfs recommends against using them. If an RPC fails 
> due to
>  these options, something called a seqid# can be "out of sync" between 
> client/server and
>  that causes serious problems.)
> --> These seqid#s are not used by NFSv4.1, so you could try that by adding
>   "minorversion=1" to your mount options.
>
> Good luck with it, rick

I've reproduced the issue on stock FreeBSD 12, and I've also learned
that nullfs is a required factor.  Doing the buildworld directly on
the NFS mount doesn't cause any slowdown, but doing a buildworld on
the nullfs copy of the NFS mount does.  The slowdown affects the base
NFS mount as well as the nullfs copy.  Here is the nfsstat output for
both server and client duing "ls -al" on the client:

nfsstat -e -s -z

Server Info:
  Getattr   SetattrLookup  Readlink  Read WriteCreateRemove
  800 0   121 0 0 2 0 0
   Rename  Link   Symlink Mkdir Rmdir   Readdir  RdirPlusAccess
0 0 0 0 0 0 0 8
MknodFsstatFsinfo  PathConfCommit   LookupP   SetClId SetClIdCf
0 0 0 0 1 3 0 0
 Open  OpenAttr OpenDwnGr  OpenCfrm DelePurge   DeleRet GetFH  Lock
0 0 0 0 0 0   123 0
LockT LockU CloseVerify   NVerify PutFH  PutPubFH PutRootFH
0 0 0 0 0   674 0 0
Renew RestoreFHSaveFH   Secinfo RelLckOwn  V4Create
0 0 0 0 0 0
Server:
RetfailedFaults   Clients
0 0 0
OpenOwner Opens LockOwner LocksDelegs
0 0 0 0 0
Server Cache Stats:
   Inprog  Idem  Non-idemMisses CacheSize   TCPPeak
0 0 0   674 16738 16738

nfsstat -e -c -z
Client Info:
Rpc Counts:
  Getattr   SetattrLookup  Readlink  Read WriteCreateRemove
   60 0   119 0 0 0 0 0
   Rename  Link   Symlink Mkdir Rmdir   Readdir  RdirPlusAccess
0 0 0 0 0 0 0 3
MknodFsstatFsinfo  PathConfCommit   SetClId SetClIdCf  Lock
0 0 0 0 0 0 0 0
LockT LockU  Open   OpenCfr
0 0 0 0
OpenOwner Opens LockOwner LocksDelegs  LocalOwn LocalOpen LocalLOwn
 5638141453 0 0 0 0 0 0
LocalLock
0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
0 0 0 0   662
Cache Info:
Attr HitsMisses Lkup HitsMisses BioR HitsMisses BioW HitsMisses

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-24 Thread Rick Macklem


On Wed, Nov 23, 2016 at 10:17:25PM -0700, Alan Somers wrote:
> I have a FreeBSD 10.3-RELEASE-p12 server exporting its home
> directories over both NFSv3 and NFSv4.  I have a TrueOS client (based
> on 12.0-CURRENT on the drm-next-4.7 branch, built on 28-October)
> mounting the home directories over NFSv4.  At first, everything is
> fine and performance is good.  But if the client does a buildworld
> using sources on NFS and locally stored objects, performance slowly
> degrades.  The degradation is most noticeable with metadata-heavy
> operations.  For example, "ls -l" in a directory with 153 files takes
> less than 0.1 seconds right after booting.  But the longer the
> buildworld goes on, the slower it gets.  Eventually that same "ls -l"
> takes 19 seconds.  When the home directories are mounted over NFSv3
> instead, I see no degradation.
>
> top shows negligible CPU consumption on the server, and very high
> consumption on the client when using NFSv4 (nearly 100%).  The
> NFS-using process is spending almost all of its time in system mode,
> and dtrace shows that almost all of its time is spent in
> ncl_getpages().
>
A couple of things you could do when it slow (as well as what Kostik suggested):
- nfsstat -c -e on client and nfsstat -e -s on server, to see what RPCs are 
being done
  and how quickly. (nfsstat -s -e will also show you how big the DRC is, 
although a
  large DRC should show up as increased CPU consumption on the server)
- capture packets with tcpdump -s 0 -w test.pcap host 
  - then you can email me test.pcap as an attachment. I can look at it in 
wireshark
and see if there seem to protocol and/or TCP issues. (You can look at in 
wireshark
yourself, the look for NFS4ERR_xxx, TCP segment retransmits...)

If you are using either "intr" or "soft" on the mounts, try without those mount 
options.
(The Bugs section of mount_nfs recommends against using them. If an RPC fails 
due to
 these options, something called a seqid# can be "out of sync" between 
client/server and
 that causes serious problems.)
--> These seqid#s are not used by NFSv4.1, so you could try that by adding
  "minorversion=1" to your mount options.

Good luck with it, rick
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: NFSv4 performance degradation with 12.0-CURRENT client

2016-11-24 Thread Konstantin Belousov

On Wed, Nov 23, 2016 at 10:17:25PM -0700, Alan Somers wrote:
> I have a FreeBSD 10.3-RELEASE-p12 server exporting its home
> directories over both NFSv3 and NFSv4.  I have a TrueOS client (based
> on 12.0-CURRENT on the drm-next-4.7 branch, built on 28-October)
> mounting the home directories over NFSv4.  At first, everything is
> fine and performance is good.  But if the client does a buildworld
> using sources on NFS and locally stored objects, performance slowly
> degrades.  The degradation is most noticeable with metadata-heavy
> operations.  For example, "ls -l" in a directory with 153 files takes
> less than 0.1 seconds right after booting.  But the longer the
> buildworld goes on, the slower it gets.  Eventually that same "ls -l"
> takes 19 seconds.  When the home directories are mounted over NFSv3
> instead, I see no degradation.
> 
> top shows negligible CPU consumption on the server, and very high
> consumption on the client when using NFSv4 (nearly 100%).  The
> NFS-using process is spending almost all of its time in system mode,
> and dtrace shows that almost all of its time is spent in
> ncl_getpages().
> 
> I have delegations disabled on the server.  On the client, the home
> directories are nullfs mounted to two additional locations, and the
> buildworld was actually using one of those nullfs mounts, not the NFS
> mount directly.
> 
> Any ideas?

Try stock FreeBSD first.

If reproducable on the stock HEAD, can you point to the lines of
ncl_getpages() where the time is spent ?  Does reading of the problematic
files, as opposed to mmaping it, also cause the behaviour ?  E.g. try dd.

There is really no time-unbounded loops in the ncl_getpages() itself.
I could understand the situation if e.g. time is spent in getpbuf() or
ncl_readrpc(), but not in ncl_getpages() directly.

Also, as an experiment, you could see if HEAD after r308980 demonstrates
any difference.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

NFSv4 performance degradation with 12.0-CURRENT client

2016-11-23 Thread Alan Somers

I have a FreeBSD 10.3-RELEASE-p12 server exporting its home
directories over both NFSv3 and NFSv4.  I have a TrueOS client (based
on 12.0-CURRENT on the drm-next-4.7 branch, built on 28-October)
mounting the home directories over NFSv4.  At first, everything is
fine and performance is good.  But if the client does a buildworld
using sources on NFS and locally stored objects, performance slowly
degrades.  The degradation is most noticeable with metadata-heavy
operations.  For example, "ls -l" in a directory with 153 files takes
less than 0.1 seconds right after booting.  But the longer the
buildworld goes on, the slower it gets.  Eventually that same "ls -l"
takes 19 seconds.  When the home directories are mounted over NFSv3
instead, I see no degradation.

top shows negligible CPU consumption on the server, and very high
consumption on the client when using NFSv4 (nearly 100%).  The
NFS-using process is spending almost all of its time in system mode,
and dtrace shows that almost all of its time is spent in
ncl_getpages().

I have delegations disabled on the server.  On the client, the home
directories are nullfs mounted to two additional locations, and the
buildworld was actually using one of those nullfs mounts, not the NFS
mount directly.

Any ideas?

-Alan
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: NFSv4 performance degradation with 12.0-CURRENT client

Re: NFSv4 performance degradation with 12.0-CURRENT client

Re: NFSv4 performance degradation with 12.0-CURRENT client

Re: NFSv4 performance degradation with 12.0-CURRENT client

Re: NFSv4 performance degradation with 12.0-CURRENT client

Re: NFSv4 performance degradation with 12.0-CURRENT client

Re: NFSv4 performance degradation with 12.0-CURRENT client

Re: NFSv4 performance degradation with 12.0-CURRENT client

Re: NFSv4 performance degradation with 12.0-CURRENT client

Re: NFSv4 performance degradation with 12.0-CURRENT client

Re: NFSv4 performance degradation with 12.0-CURRENT client

Re: NFSv4 performance degradation with 12.0-CURRENT client

Re: NFSv4 performance degradation with 12.0-CURRENT client

NFSv4 performance degradation with 12.0-CURRENT client

14 matches

Site Navigation

Mail list logo

Footer information