** Changed in: linux (Ubuntu Focal)
Importance: Undecided => Medium
** Changed in: linux (Ubuntu Focal)
Status: In Progress => Fix Committed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1957986
Title:
Recalled NFSv4 files delegations overwhelm server
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Focal:
Fix Committed
Status in linux source package in Hirsute:
Fix Released
Status in linux source package in Impish:
Fix Released
Status in linux source package in Jammy:
Fix Released
Bug description:
[Impact]
An NFSv4 client that does a lot of opens/closes can overwhelm and NFSv4
server, causing a significant drop in performance. In my testing, I've seen
performance drop from ~700MiB/s down to < 10MiB/s. The same workload using
NFSv3 does not have this problem.
[Test Case]
This can be demonstrated using the elbencho benchmark from
https://github.com/breuner/elbencho:
$ elbencho -t 40 -r -n 10 -N 5000 -s 128k -b 128k /mnt/nfs/ubuntu
You'll notice the nfsd threads (I stuck w/ the default of 4) start to
consume 100% CPU, and the performance of the elbencho benchmark will
begin to trickle.
[Fix]
The following fix solves the problem, but there are a number of patches
dependencies required before it will apply to focal:
commit 10717f45639f6c1bc27b56405252c3a027406d92 (refs/bisect/bad)
Author: Trond Myklebust <[email protected]>
Date: Mon Jan 27 09:58:19 2020 -0500
NFSv4: Limit the total number of cached delegations
Delegations can be expensive to return, and can cause scalability issues
for the server. Let's therefore try to limit the number of inactive
delegations we hold.
Once the number of delegations is above a certain threshold, start
to return them on close.
Signed-off-by: Trond Myklebust <[email protected]>
Signed-off-by: Anna Schumaker <[email protected]>
[What could go wrong]
The fixes are restricted to NFS code, so problems should be limited to NFS
users. They could include performance issues, crashes, etc. Because these
changes are mostly related to NFS delegations, I use the `nfstest_delegation`
test suite from nfstest[*] to try and identify any regressions:
./nfstest_delegation --client 192.168.42.1 --server 192.168.42.2 -e
/srv/nfstest --trcdelay 4
Both before and after applying the fixes, I see the same 146 tests
pass and 23 failures. The 23 failures are expected because I was using
a Linux-based NFSv4 server which does not support all of the
delegation modes that the test checks for.
[*] git://git.linux-nfs.org/projects/mora/nfstest.git
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1957986/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : [email protected]
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
