[Bug 1887607] Re: NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server
This bug was fixed in the package linux - 4.15.0-115.116
---
linux (4.15.0-115.116) bionic; urgency=medium
* bionic/linux: 4.15.0-115.116 -proposed tracker (LP: #1893055)
* [Potential Regression] dscr_inherit_exec_test from powerpc in
ubuntu_kernel_selftests failed on B/E/F (LP: #1888332)
- powerpc/64s: Don't init FSCR_DSCR in __init_FSCR()
linux (4.15.0-114.115) bionic; urgency=medium
* bionic/linux: 4.15.0-114.115 -proposed tracker (LP: #1891052)
* ipsec: policy priority management is broken (LP: #1890796)
- xfrm: policy: match with both mark and mask on user interfaces
linux (4.15.0-113.114) bionic; urgency=medium
* bionic/linux: 4.15.0-113.114 -proposed tracker (LP: #1890705)
* Packaging resync (LP: #1786013)
- update dkms package versions
* Reapply "usb: handle warm-reset port requests on hub resume" (LP: #1859873)
- usb: handle warm-reset port requests on hub resume
* Bionic update: upstream stable patchset 2020-07-29 (LP: #1889474)
- gpio: arizona: handle pm_runtime_get_sync failure case
- gpio: arizona: put pm_runtime in case of failure
- pinctrl: amd: fix npins for uart0 in kerncz_groups
- mac80211: allow rx of mesh eapol frames with default rx key
- scsi: scsi_transport_spi: Fix function pointer check
- xtensa: fix __sync_fetch_and_{and,or}_4 declarations
- xtensa: update *pos in cpuinfo_op.next
- drivers/net/wan/lapbether: Fixed the value of hard_header_len
- net: sky2: initialize return of gm_phy_read
- drm/nouveau/i2c/g94-: increase NV_PMGR_DP_AUXCTL_TRANSACTREQ timeout
- irqdomain/treewide: Keep firmware node unconditionally allocated
- SUNRPC reverting d03727b248d0 ("NFSv4 fix CLOSE not waiting for direct IO
compeletion")
- spi: spi-fsl-dspi: Exit the ISR with IRQ_NONE when it's not ours
- IB/umem: fix reference count leak in ib_umem_odp_get()
- uprobes: Change handle_swbp() to send SIGTRAP with si_code=SI_KERNEL, to
fix
GDB regression
- ALSA: info: Drop WARN_ON() from buffer NULL sanity check
- ASoC: rt5670: Correct RT5670_LDO_SEL_MASK
- btrfs: fix double free on ulist after backref resolution failure
- btrfs: fix mount failure caused by race with umount
- btrfs: fix page leaks after failure to lock page for delalloc
- bnxt_en: Fix race when modifying pause settings.
- hippi: Fix a size used in a 'pci_free_consistent()' in an error handling
path
- ax88172a: fix ax88172a_unbind() failures
- net: dp83640: fix SIOCSHWTSTAMP to update the struct with actual
configuration
- drm: sun4i: hdmi: Fix inverted HPD result
- net: smc91x: Fix possible memory leak in smc_drv_probe()
- bonding: check error value of register_netdevice() immediately
- mlxsw: destroy workqueue when trap_register in mlxsw_emad_init
- ipvs: fix the connection sync failed in some cases
- i2c: rcar: always clear ICSAR to avoid side effects
- bonding: check return value of register_netdevice() in bond_newlink()
- serial: exar: Fix GPIO configuration for Sealevel cards based on XR17V35X
- scripts/decode_stacktrace: strip basepath from all paths
- HID: i2c-hid: add Mediacom FlexBook edge13 to descriptor override
- HID: apple: Disable Fn-key key-re-mapping on clone keyboards
- dmaengine: tegra210-adma: Fix runtime PM imbalance on error
- Input: add `SW_MACHINE_COVER`
- spi: mediatek: use correct SPI_CFG2_REG MACRO
- regmap: dev_get_regmap_match(): fix string comparison
- hwmon: (aspeed-pwm-tacho) Avoid possible buffer overflow
- dmaengine: ioat setting ioat timeout as module parameter
- Input: synaptics - enable InterTouch for ThinkPad X1E 1st gen
- usb: gadget: udc: gr_udc: fix memleak on error handling path in
gr_ep_init()
- arm64: Use test_tsk_thread_flag() for checking TIF_SINGLESTEP
- x86: math-emu: Fix up 'cmp' insn for clang ias
- binder: Don't use mmput() from shrinker function.
- usb: xhci-mtk: fix the failure of bandwidth allocation
- usb: xhci: Fix ASM2142/ASM3142 DMA addressing
- Revert "cifs: Fix the target file was deleted when rename failed."
- staging: wlan-ng: properly check endpoint types
- staging: comedi: addi_apci_1032: check INSN_CONFIG_DIGITAL_TRIG shift
- staging: comedi: ni_6527: fix INSN_CONFIG_DIGITAL_TRIG support
- staging: comedi: addi_apci_1500: check INSN_CONFIG_DIGITAL_TRIG shift
- staging: comedi: addi_apci_1564: check INSN_CONFIG_DIGITAL_TRIG shift
- serial: 8250: fix null-ptr-deref in serial8250_start_tx()
- serial: 8250_mtk: Fix high-speed baud rates clamping
- fbdev: Detect integer underflow at "struct fbcon_ops"->clear_margins.
- vt: Reject zero-sized screen buffer size.
- Makefile: Fix GCC_TOOLCHAIN_DIR prefix for Clang cross compilation
- mm/memcg: fix refcount error while moving and swapping
- io-mapping: indicate mapping failure
- parisc: Add atomic64_set_release()
[Bug 1887607] Re: NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server
I installed 4.15.0-114-generic from -proposed to my test client machine, which is a Ubuntu 18.04 Desktop VM. I mounted two NFS shares, one with sec=sys, and the other with sec=krb5p. I then opened each share up in separate tabs in Nautilus. I then CUT a file from the sec=sys share, and PASTED it into the sec=krb5p share. The file pasted to the new share correctly, and was removed from the original share. I repeated this several times, and each file movement was successful with no hangs. I then moved all files from the sec=krb5p share, back to the sec=sys share, selected all files, then CUT and PASTED them to the sec=krb5p share. All files moved without the system hanging. I repeated this a number of times, and the system functions correctly. There is an occasional delay when pasting some files, that lasts for no more than a few seconds, which happens when a file copy is interrupted, but the NFS subsystem recovers, likely finding the correct slot and sequence number, and the file moves successfully. The delay is likely due to the NFS client needing a few round trips to the server to discover the correct slot and sequence number. The hang is fixed, and I am happy to mark this as verified. ** Tags removed: verification-needed-bionic ** Tags added: verification-done-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1887607 Title: NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1887607/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887607] Re: NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed- bionic'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1887607 Title: NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1887607/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887607] Re: NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server
** Changed in: linux (Ubuntu Bionic) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1887607 Title: NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1887607/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887607] Re: NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server
** Description changed: BugLink: https://bugs.launchpad.net/bugs/1887607 [Impact] There is a bug in NFS v4.1 that causes a large amount of RPC calls between a client and server when a previous RPC call is interrupted. This uses a large amount of bandwidth and can saturate the network. The symptoms are so: * On NFS clients: Attempts to access mounted NFS shares associated with the affected server block indefinitely. * On the network: A storm of repeated RPCs between NFS client and server uses a lot of bandwidth. Each RPC is acknowledged by the server with an NFS4ERR_SEQ_MISORDERED error. * Other NFS clients connected to the same NFS server: Performance drops dramatically. This occurs during a "false retry", when a client attempts to make a new RPC call using a slot+sequence number that references an older, cached call. This happens when a user process interrupts an RPC call that is in progress. I had previously fixed this for Disco in bug 1828978, and now a customer has run into the issue in Bionic. A reproducer is supplied in the testcase section, which was something missing from bug 1828978, since we never determined how the issue actually occurred back then. [Fix] This was fixed in 5.1 upstream with the below commit: commit 3453d5708b33efe76f40eca1c0ed60923094b971 Author: Trond Myklebust Date: Wed Jun 20 17:53:34 2018 -0400 Subject: NFSv4.1: Avoid false retries when RPC calls are interrupted The fix is to pre-emptively increment the sequence number if an RPC call is interrupted, and to address corner cases we interpret the NFS4ERR_SEQ_MISORDERED error as a sign we need to locate an appropriate sequence number between the value we sent, and the last successfully acked SEQUENCE call. The commit also requires two fixup commits, which landed in 5.5 and 5.8-rc6 respectively: commit 5c441544f045e679afd6c3c6d9f7aaf5fa5f37b0 Author: Trond Myklebust Date: Wed Nov 13 08:34:00 2019 +0100 Subject: NFSv4.x: Handle bad/dead sessions correctly in nfs41_sequence_process() commit 913fadc5b105c3619d9e8d0fe8899ff1593cc737 Author: Anna Schumaker Date: Wed Jul 8 10:33:40 2020 -0400 Subject: NFS: Fix interrupted slots by sending a solo SEQUENCE operation Commits 3453d5708b33efe76f40eca1c0ed60923094b971 and 913fadc5b105c3619d9e8d0fe8899ff1593cc737 require small backports to bionic, as struct rpc_cred changed to const struct cred in 5.0, and the backports swap them back to struct rpc_cred since that is how 4.15 works. [Testcase] You will need four machines. The first, is a kerberos KDC. Set up Kerberos correctly and create new service principals for the NFS server and for the client. I used: nfs/nfskerb.mydomain.com and nfs/client.mydomain.com. The second machine will be a NFS server with the krb5p share. Add the nfs server kerberos keys to the system's keytab, and set up a NFS server that exports a directory with sec=krb5p. Example export: /mnt/secretfolder *.mydomain.com(rw,sync,no_subtree_check,sec=krb5p) The third machine is a regular NFS server. Export a directory with normal sec=sys security. Example export: /mnt/sharedfolder *.mydomain.com(rw,sync) The fourth is a desktop machine. Add the client kerberos keys to the system's keytab. Mount both NFS shares, making sure to use the NFS v4.2 protocol. I used the commands: mount -t nfs4 nfskerb.mydomain.com:/mnt/secretfolder /mnt/secretfolder_client/ mount -t nfs4 nfs.mydomain.com:/mnt/sharedfolder /mnt/sharedfolder_client Check "mount -l" to ensure that NFS v4.2 is used: nfskerb.mydomain.com:/mnt/secretfolder on /mnt/secretfolder_client type nfs4 (rw,relatime,vers=4.2,<...>,sec=krb5p,<...>) nfs.mydomain.com:/mnt/sharedfolder on /mnt/sharedfolder_client type nfs4 (rw,relatime,vers=4.2,<...>,sec=sys,<...>) Generate some files full of random data. I found 20MB from /dev/random works great. Open each NFS share up in tabs in Nautilus. Copy the random data files to the sec=sys NFS share. When they are done, one at a time cut and then paste the file into the sec=krb5p NFS share. The bug will trigger either on the first, or subsequent tries, but less than 10 tries are needed usually. There is a test kernel available in the following PPA: https://launchpad.net/~mruffell/+archive/ubuntu/sf285439-test If you install the test kernel, files will cut and paste correctly, and NFS will work as expected. [Regression Potential] The changes are localised to NFS v4.1 and v4.2 only, and other versions of NFS are not affected. If a regression occurs, users can downgrade NFS versions to v4.0 or v3.x until a fix is made. The changes only impact when connections are interrupted, and under typical blue sky scenarios would not be invoked. - There have been no fixup commits or commits near the requested commit in - newer kernels, which points
[Bug 1887607] Re: NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server
** Description changed: BugLink: https://bugs.launchpad.net/bugs/1887607 [Impact] There is a bug in NFS v4.1 that causes a large amount of RPC calls between a client and server when a previous RPC call is interrupted. This uses a large amount of bandwidth and can saturate the network. The symptoms are so: * On NFS clients: Attempts to access mounted NFS shares associated with the affected server block indefinitely. * On the network: - A storm of repeated RPCs between NFS client and server uses a lot of bandwidth. Each RPC is acknoledged by the server with an NFS4ERR_SEQ_MISORDERED error. + A storm of repeated RPCs between NFS client and server uses a lot of bandwidth. Each RPC is acknowledged by the server with an NFS4ERR_SEQ_MISORDERED error. * Other NFS clients connected to the same NFS server: Performance drops dramatically. This occurs during a "false retry", when a client attempts to make a new RPC call using a slot+sequence number that references an older, cached call. This happens when a user process interrupts an RPC call that is in progress. I had previously fixed this for Disco in bug 1828978, and now a customer has run into the issue in Bionic. A reproducer is supplied in the testcase section, which was something missing from bug 1828978, since we - never determined how the issue actually occured back then. + never determined how the issue actually occurred back then. [Fix] This was fixed in 5.1 upstream with the below commit: commit 3453d5708b33efe76f40eca1c0ed60923094b971 Author: Trond Myklebust Date: Wed Jun 20 17:53:34 2018 -0400 Subject: NFSv4.1: Avoid false retries when RPC calls are interrupted The fix is to pre-emptively increment the sequence number if an RPC call is interrupted, and to address corner cases we interpret the - NFS4ERR_SEQ_MISORDERED error as a sign we need to locate an approperiate + NFS4ERR_SEQ_MISORDERED error as a sign we need to locate an appropriate sequence number between the value we sent, and the last successfully acked SEQUENCE call. The commit also requires two fixup commits, which landed in 5.5 and 5.8-rc6 respectively: commit 5c441544f045e679afd6c3c6d9f7aaf5fa5f37b0 Author: Trond Myklebust Date: Wed Nov 13 08:34:00 2019 +0100 Subject: NFSv4.x: Handle bad/dead sessions correctly in nfs41_sequence_process() commit 913fadc5b105c3619d9e8d0fe8899ff1593cc737 Author: Anna Schumaker Date: Wed Jul 8 10:33:40 2020 -0400 Subject: NFS: Fix interrupted slots by sending a solo SEQUENCE operation Commits 3453d5708b33efe76f40eca1c0ed60923094b971 and 913fadc5b105c3619d9e8d0fe8899ff1593cc737 require small backports to bionic, as struct rpc_cred changed to const struct cred in 5.0, and the backports swap them back to struct rpc_cred since that is how 4.15 works. [Testcase] You will need four machines. The first, is a kerberos KDC. Set up Kerberos correctly and create new service principals for the NFS server and for the client. I used: nfs/nfskerb.mydomain.com and nfs/client.mydomain.com. The second machine will be a NFS server with the krb5p share. Add the nfs server kerberos keys to the system's keytab, and set up a NFS server that exports a directory with sec=krb5p. Example export: /mnt/secretfolder *.mydomain.com(rw,sync,no_subtree_check,sec=krb5p) The third machine is a regular NFS server. Export a directory with normal sec=sys security. Example export: /mnt/sharedfolder *.mydomain.com(rw,sync) The fourth is a desktop machine. Add the client kerberos keys to the system's keytab. Mount both NFS shares, making sure to use the NFS v4.2 protocol. I used the commands: mount -t nfs4 nfskerb.mydomain.com:/mnt/secretfolder /mnt/secretfolder_client/ mount -t nfs4 nfs.mydomain.com:/mnt/sharedfolder /mnt/sharedfolder_client Check "mount -l" to ensure that NFS v4.2 is used: nfskerb.mydomain.com:/mnt/secretfolder on /mnt/secretfolder_client type nfs4 (rw,relatime,vers=4.2,<...>,sec=krb5p,<...>) nfs.mydomain.com:/mnt/sharedfolder on /mnt/sharedfolder_client type nfs4 (rw,relatime,vers=4.2,<...>,sec=sys,<...>) Generate some files full of random data. I found 20MB from /dev/random works great. Open each NFS share up in tabs in Nautilus. Copy the random data files to the sec=sys NFS share. When they are done, one at a time cut and then paste the file into the sec=krb5p NFS share. The bug will trigger either on the first, or subsequent tries, but less than 10 tries are needed usually. There is a test kernel available in the following PPA: https://launchpad.net/~mruffell/+archive/ubuntu/sf285439-test If you install the test kernel, files will cut and paste correctly, and NFS will work as expected. [Regression Potential] The changes are localised to NFS v4.1 and v4.2 only, and other versions of NFS are not affected. If a regression occurs, users
[Bug 1887607] Re: NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server
** Summary changed: - NFS4.2: Cutting and Pasting files from NFS sec=sys to NFS sec=krb5p causes NFS to hang + NFSv4.1: Interrupted connections cause high bandwidth RPC ping-pong between client and server ** Description changed: BugLink: https://bugs.launchpad.net/bugs/1887607 [Impact] - If you have a desktop system, with two NFS v4.2 mounts: - - One that uses the baseline IP based security, aka sec=sys, - - and the other that uses Kerberos sec=krb5p based security, + There is a bug in NFS v4.1 that causes a large amount of RPC calls + between a client and server when a previous RPC call is interrupted. + This uses a large amount of bandwidth and can saturate the network. - If you try and cut a file from the normal NFS mount, and paste it to a - directory on the kerberos krb5p mount (using Nautilus), the NFS - subsystem will lock up, Nautilus will hang, and the file won't be moved. + The symptoms are so: - The problem only reproduces if you cut and paste. Copying and pasting - does not trigger any problems. Using mv in terminal doesn't reproduce - either, you need to use Nautilus. + * On NFS clients: + Attempts to access mounted NFS shares associated with the affected server block indefinitely. - The issue was introduced into 4.15.0-60-generic, by the commit: + * On the network: + A storm of repeated RPCs between NFS client and server uses a lot of bandwidth. Each RPC is acknoledged by the server with an NFS4ERR_SEQ_MISORDERED error. - commit 594d1644cd59447f4fceb592448d5cd09eb09b5e - Author: Chris Perl - Date: Mon Dec 17 10:56:38 2018 -0500 - Subject: NFS: nfs_compare_mount_options always compare auth flavors. - Link: https://github.com/torvalds/linux/commit/594d1644cd59447f4fceb592448d5cd09eb09b5e + * Other NFS clients connected to the same NFS server: + Performance drops dramatically. - It was backported to 4.15.0-60-generic from upstream -stable, and landed - in 4.4.175, 4.14.99 and 4.19.21. The commit itself does not actually - cause the problem, it simply removes a check for NFS server security - settings, which simply reveals a broken codepath which the testcase - triggers. + This occurs during a "false retry", when a client attempts to make a new + RPC call using a slot+sequence number that references an older, cached + call. This happens when a user process interrupts an RPC call that is in + progress. - Xenial 4.4.0-185-generic is not affected, only Bionic 4.15.0-60-generic - onward. + I had previously fixed this for Disco in bug 1828978, and now a customer + has run into the issue in Bionic. A reproducer is supplied in the + testcase section, which was something missing from bug 1828978, since we + never determined how the issue actually occured back then. [Fix] - The fix landed in 5.1-rc1, in the following commit: + This was fixed in 5.1 upstream with the below commit: - commit 02ef04e432babf8fc703104212314e54112ecd2d - Author: Chuck Lever - Date: Mon Feb 11 11:25:25 2019 -0500 - Subject: NFS: Account for XDR pad of buf->pages - Link: https://github.com/torvalds/linux/commit/02ef04e432babf8fc703104212314e54112ecd2d + commit 3453d5708b33efe76f40eca1c0ed60923094b971 + Author: Trond Myklebust + Date: Wed Jun 20 17:53:34 2018 -0400 + Subject: NFSv4.1: Avoid false retries when RPC calls are interrupted - The above commit more or less relies on the below commit as a - dependency, and is included in the SRU: + The fix is to pre-emptively increment the sequence number if an RPC call + is interrupted, and to address corner cases we interpret the + NFS4ERR_SEQ_MISORDERED error as a sign we need to locate an approperiate + sequence number between the value we sent, and the last successfully + acked SEQUENCE call. - commit cf500bac8fd48b57f38ece890235923d4ed5ee91 - Author: Chuck Lever - Date: Mon Feb 11 11:25:20 2019 -0500 - Subject: SUNRPC: Introduce rpc_prepare_reply_pages() - Link: https://github.com/torvalds/linux/commit/cf500bac8fd48b57f38ece890235923d4ed5ee91 + The commit also requires two fixup commits, which landed in 5.5 and + 5.8-rc6 respectively: - Also, as Andrea Righi pointed out, we need the following fixes commit: + commit 5c441544f045e679afd6c3c6d9f7aaf5fa5f37b0 + Author: Trond Myklebust + Date: Wed Nov 13 08:34:00 2019 +0100 + Subject: NFSv4.x: Handle bad/dead sessions correctly in nfs41_sequence_process() - commit 29e7ca715f2a0b6c0a99b1aec1b0956d1f271955 - Author: Chuck Lever - Date: Tue Apr 9 10:44:16 2019 -0400 - Subject: NFS: Fix handling of reply page vector - Link: https://github.com/torvalds/linux/commit/29e7ca715f2a0b6c0a99b1aec1b0956d1f271955 + commit 913fadc5b105c3619d9e8d0fe8899ff1593cc737 + Author: Anna Schumaker + Date: Wed Jul 8 10:33:40 2020 -0400 + Subject: NFS: Fix interrupted slots by sending a solo SEQUENCE operation - It appears that some NFS calls return a NFS payload which is not a - multiple of 4 bytes, but any payload sent over the network needs to be - padded to an exact multiple
