Re: NFS: Kill so_upcall NET_LOCK() recursion

[Christian Weisgerber]

On 2017-02-06, Martin Pieuchot <m...@openbsd.org> wrote:

> guenther@ pointed out during a2k17 that nfsrv_rcv() already has a way to
> not re-enter the socket layer.  Instead of doing soreceive() directly in
> the receiving path, we notify and wakeup nfsd(8) to do it.
>
> I'd like to do that unconditionally to fix one of the blocking
> recursions.

This more than halves performance for a FreeBSD client _writing_
to an OpenBSD server.  Read performance is not (significantly)
affected.

===> /obsd
dd bs=64k
from NFS
8110515331 bytes transferred in 113.948541 secs (71177000 bytes/sec)
8110515331 bytes transferred in 114.227510 secs (71003170 bytes/sec)
8110515331 bytes transferred in 114.647424 secs (70743110 bytes/sec)

to NFS
8110515331 bytes transferred in 258.461309 secs (31379998 bytes/sec)
8110515331 bytes transferred in 220.057884 secs (36856282 bytes/sec)
8110515331 bytes transferred in 219.894926 secs (36883595 bytes/sec)

===> /bsd   "NFS: Kill so_upcall NET_LOCK() recursion"
dd bs=64k
from NFS
8110515331 bytes transferred in 115.323728 secs (70328244 bytes/sec)
8110515331 bytes transferred in 117.878431 secs (68804066 bytes/sec)
8110515331 bytes transferred in 117.852977 secs (68818926 bytes/sec)

to NFS
8110515331 bytes transferred in 559.963407 secs (14484010 bytes/sec)
8110515331 bytes transferred in 558.174282 secs (14530435 bytes/sec)
8110515331 bytes transferred in 552.301516 secs (14684941 bytes/sec)

-- 
Christian "naddy" Weisgerber  na...@mips.inka.de

NFS: Kill so_upcall NET_LOCK() recursion

[Martin Pieuchot]

guenther@ pointed out during a2k17 that nfsrv_rcv() already has a way to
not re-enter the socket layer.  Instead of doing soreceive() directly in
the receiving path, we notify and wakeup nfsd(8) to do it.

I'd like to do that unconditionally to fix one of the blocking
recursions.

I just finished a bake a build on NFS.  I'd appreciate more tests an
oks.

Index: kern/uipc_socket.c
===
RCS file: /cvs/src/sys/kern/uipc_socket.c,v
retrieving revision 1.176
diff -u -p -r1.176 uipc_socket.c
--- kern/uipc_socket.c  1 Feb 2017 20:59:47 -   1.176
+++ kern/uipc_socket.c  6 Feb 2017 14:33:48 -
@@ -1530,12 +1530,8 @@ sorwakeup(struct socket *so)
return;
 #endif
sowakeup(so, >so_rcv);
-   if (so->so_upcall) {
-   /* XXXSMP breaks atomicity */
-   rw_exit_write();
+   if (so->so_upcall)
(*(so->so_upcall))(so, so->so_upcallarg, M_DONTWAIT);
-   rw_enter_write();
-   }
 }
 
 void
Index: nfs/nfs_socket.c
===
RCS file: /cvs/src/sys/nfs/nfs_socket.c,v
retrieving revision 1.112
diff -u -p -r1.112 nfs_socket.c
--- nfs/nfs_socket.c19 Dec 2016 08:36:50 -  1.112
+++ nfs/nfs_socket.c6 Feb 2017 14:16:39 -
@@ -1570,26 +1570,15 @@ nfsrv_rcv(struct socket *so, caddr_t arg
 
if ((slp->ns_flag & SLP_VALID) == 0)
return;
-#ifdef notdef
-   /*
-* Define this to test for nfsds handling this under heavy load.
-*/
+
+   /* Defer soreceive() to an nfsd. */
if (waitflag == M_DONTWAIT) {
-   slp->ns_flag |= SLP_NEEDQ; goto dorecs;
+   slp->ns_flag |= SLP_NEEDQ;
+   goto dorecs;
}
-#endif
+
auio.uio_procp = NULL;
if (so->so_type == SOCK_STREAM) {
-   /*
-* If there are already records on the queue, defer soreceive()
-* to an nfsd so that there is feedback to the TCP layer that
-* the nfs servers are heavily loaded.
-*/
-   if (slp->ns_rec && waitflag == M_DONTWAIT) {
-   slp->ns_flag |= SLP_NEEDQ;
-   goto dorecs;
-   }
-
/*
 * Do soreceive().
 */
Index: conf/GENERIC
===
RCS file: /cvs/src/sys/conf/GENERIC,v
retrieving revision 1.239
diff -u -p -r1.239 GENERIC
--- conf/GENERIC25 Jan 2017 00:48:36 -  1.239
+++ conf/GENERIC6 Feb 2017 14:36:04 -
@@ -59,7 +59,7 @@ optionPPP_DEFLATE
 option PIPEX   # Ppp IP EXtension, for npppd
 option MROUTING# Multicast router
 option MPLS# Multi-Protocol Label Switching
-option BFD # Bi-directional Forwarding Detection
+#optionBFD # Bi-directional Forwarding Detection
 
 #mpath0at root # SCSI Multipathing
 #scsibus*  at mpath?