apport information ** Attachment added: "ProcCpuinfoMinimal.txt" https://bugs.launchpad.net/bugs/2046572/+attachment/5729588/+files/ProcCpuinfoMinimal.txt
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2046572 Title: NFS latency spikes during openat syscalls associated with test_stateid RPC command Status in linux package in Ubuntu: New Bug description: About once a week the NFS mounts on our eda servers (the nfs clients) slow to a crawl for a large number of workloads. strace reveals that openat system calls for files in the mounts take between 0.1 and 0.3 seconds this can be seen directly by touching any file in the homedir (this normally takes 0.001-0.002 seconds). This causes obvious problems, an IPython instance in a conda environment installed in NFS takes around 2-3 minutes to start for example. I originally thought that this was a manifestation of #2009325, but we are on 5.15.0-89 and have seen this problem across several kernel versions. Additionally we don't see access rpc calls but we do see loads of test_stateid calls: - bursts of 300 or so periodically every couple seconds while idle - 380,452 test_stateid calls when an ipython instance is starting (zero access calls in the same period, when at idle we see 10,267 test_stateid calls over the same ~2:30 time period and again no access calls) We're not using kerberos, just ip based authentication, our mount options are: exoserver.example.com:/volume1/homes/cudaa on /home/cudaa type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=xxx.xxx.xxx.219,local_lock=none,addr=xxx.xxx.xxx.146) We see this on servers with both linux-image-5.15.0-89-generic and linux-headers-5.15.0-83-generic, and have seen it across a few versions (we have the hwe stack installed if that matters). I could upgrade them to 91 but the issue clears after a reboot and we don't have a reliable way to reproduce it so I'm going to keep one online for debugging. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu27.27 Architecture: amd64 AudioDevicesInUse: USER PID ACCESS COMMAND /dev/snd/controlC0: gdm 3334 F.... pulseaudio CasperMD5CheckResult: skip DistroRelease: Ubuntu 20.04 InstallationDate: Installed on 2022-04-05 (618 days ago) InstallationMedia: Ubuntu 20.04.4 LTS "Focal Fossa" - Release amd64 (20220223) IwConfig: lo no wireless extensions. enp69s0 no wireless extensions. enp68s0 no wireless extensions. MachineType: ASUS System Product Name NonfreeKernelModules: nvidia_modeset nvidia Package: linux (not installed) ProcEnviron: TERM=screen PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/zsh ProcFB: 0 EFI VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-89-generic root=UUID=ec5ca229-1bd1-461a-98fd-d0f4eba0dd26 ro fsck.mode=force fsck.repair=yes ProcVersionSignature: Ubuntu 5.15.0-89.99~20.04.1-generic 5.15.126 PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon. RelatedPackageVersions: linux-restricted-modules-5.15.0-89-generic N/A linux-backports-modules-5.15.0-89-generic N/A linux-firmware 1.187.39 RfKill: Tags: focal Uname: Linux 5.15.0-89-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True dmi.bios.date: 03/04/2022 dmi.bios.release: 16.3 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1603 dmi.board.asset.tag: Default string dmi.board.name: ROG ZENITH II EXTREME dmi.board.vendor: ASUSTeK COMPUTER INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Default string dmi.chassis.type: 3 dmi.chassis.vendor: Default string dmi.chassis.version: Default string dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1603:bd03/04/2022:br16.3:svnASUS:pnSystemProductName:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnROGZENITHIIEXTREME:rvrRev1.xx:cvnDefaultstring:ct3:cvrDefaultstring:skuSKU: dmi.product.family: To be filled by O.E.M. dmi.product.name: System Product Name dmi.product.sku: SKU dmi.product.version: System Version dmi.sys.vendor: ASUS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2046572/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
