Bug#1025537: nfsd: Kernel Oops while serving NFS
Package: src:linux Followup-For: Bug #1025537 Dear Maintainer, Just a heads up that, after some uptime and an utimely reboot, I am now on 6.4.0-3, which no longer seems to exhibit the issue from earlier 6.x series. While we didn't get to the root cause, I think this bug can be tentatively closed. -- Package-specific info: ** Version: Linux version 6.4.0-3-amd64 (debian-kernel@lists.debian.org) (gcc-13 (Debian 13.2.0-2) 13.2.0, GNU ld (GNU Binutils for Debian) 2.41) #1 SMP PREEMPT_DYNAMIC Debian 6.4.11-1 (2023-08-17) ** Command line: BOOT_IMAGE=/boot/vmlinuz-6.4.0-3-amd64 root=UUID=0c917590-acc7-464c-8961-64f79e1d1c69 ro init=/lib/systemd/systemd init=/lib/systemd/systemd ** Not tainted ** Kernel log: ** Model information sys_vendor: [ 513.266864] veth75b1ef3: entered allmulticast mode [ 513.278803] veth75b1ef3: entered promiscuous mode [ 513.318708] eth0: renamed from veth289468d [ 513.350910] IPv6: ADDRCONF(NETDEV_CHANGE): vethd452ab0: link becomes ready [ 513.354043] br-167ce2d5a745: port 1(vethd452ab0) entered blocking state [ 513.357003] br-167ce2d5a745: port 1(vethd452ab0) entered forwarding state [ 513.360079] IPv6: ADDRCONF(NETDEV_CHANGE): br-167ce2d5a745: link becomes ready [ 516.185425] br-8faf6cb95741: port 3(veth3348892) entered blocking state [ 516.189032] br-8faf6cb95741: port 3(veth3348892) entered disabled state [ 516.203888] veth3348892: entered allmulticast mode [ 516.210796] veth3348892: entered promiscuous mode [ 529.090856] eth0: renamed from vethd1244c6 [ 529.111282] IPv6: ADDRCONF(NETDEV_CHANGE): veth20d9bda: link becomes ready [ 529.114664] br-410ffb6f054c: port 1(veth20d9bda) entered blocking state [ 529.117918] br-410ffb6f054c: port 1(veth20d9bda) entered forwarding state [ 529.121900] IPv6: ADDRCONF(NETDEV_CHANGE): br-410ffb6f054c: link becomes ready [ 550.583717] br-8faf6cb95741: port 4(vethabeb253) entered blocking state [ 550.587030] br-8faf6cb95741: port 4(vethabeb253) entered disabled state [ 550.593753] vethabeb253: entered allmulticast mode [ 550.597325] vethabeb253: entered promiscuous mode [ 566.713758] eth0: renamed from vetha110c01 [ 566.736720] IPv6: ADDRCONF(NETDEV_CHANGE): veth6843607: link becomes ready [ 566.740228] br-167ce2d5a745: port 2(veth6843607) entered blocking state [ 566.743665] br-167ce2d5a745: port 2(veth6843607) entered forwarding state [ 566.982165] eth0: renamed from vethb129f19 [ 567.004696] IPv6: ADDRCONF(NETDEV_CHANGE): veth7291cce: link becomes ready [ 567.008299] br-8faf6cb95741: port 1(veth7291cce) entered blocking state [ 567.011785] br-8faf6cb95741: port 1(veth7291cce) entered forwarding state [ 567.020597] IPv6: ADDRCONF(NETDEV_CHANGE): br-8faf6cb95741: link becomes ready [ 569.328071] eth0: renamed from veth0690f82 [ 569.345031] IPv6: ADDRCONF(NETDEV_CHANGE): veth908ed4d: link becomes ready [ 569.348796] br-8fa0fed72bf0: port 2(veth908ed4d) entered blocking state [ 569.352495] br-8fa0fed72bf0: port 2(veth908ed4d) entered forwarding state [ 570.025027] eth0: renamed from veth8259256 [ 570.045606] IPv6: ADDRCONF(NETDEV_CHANGE): vetha2c6a28: link becomes ready [ 570.050336] br-8fa0fed72bf0: port 3(vetha2c6a28) entered blocking state [ 570.054287] br-8fa0fed72bf0: port 3(vetha2c6a28) entered forwarding state [ 570.469070] eth0: renamed from veth452c69a [ 570.489637] IPv6: ADDRCONF(NETDEV_CHANGE): vethd58db50: link becomes ready [ 570.493575] br-410ffb6f054c: port 2(vethd58db50) entered blocking state [ 570.497349] br-410ffb6f054c: port 2(vethd58db50) entered forwarding state [ 570.841451] eth0: renamed from vethd61b822 [ 570.865532] IPv6: ADDRCONF(NETDEV_CHANGE): veth3fcbeaa: link becomes ready [ 570.870006] br-8faf6cb95741: port 2(veth3fcbeaa) entered blocking state [ 570.874333] br-8faf6cb95741: port 2(veth3fcbeaa) entered forwarding state [ 572.555320] eth0: renamed from vethb7b0199 [ 572.581041] IPv6: ADDRCONF(NETDEV_CHANGE): veth75b1ef3: link becomes ready [ 572.584993] br-8fa0fed72bf0: port 4(veth75b1ef3) entered blocking state [ 572.588752] br-8fa0fed72bf0: port 4(veth75b1ef3) entered forwarding state [ 575.757608] eth0: renamed from veth55404ed [ 575.777370] IPv6: ADDRCONF(NETDEV_CHANGE): veth3348892: link becomes ready [ 575.781477] br-8faf6cb95741: port 3(veth3348892) entered blocking state [ 575.785425] br-8faf6cb95741: port 3(veth3348892) entered forwarding state [ 579.384248] eth0: renamed from vethdd62334 [ 579.406715] IPv6: ADDRCONF(NETDEV_CHANGE): vethabeb253: link becomes ready [ 579.411269] br-8faf6cb95741: port 4(vethabeb253) entered blocking state [ 579.415641] br-8faf6cb95741: port 4(vethabeb253) entered forwarding state [ 649.519156] kauditd_printk_skb: 8 callbacks suppressed [ 649.519163] audit: type=1400 audit(1693550398.502:20): apparmor="DENIED" operation="capable" class="cap" profile="/usr/sbin/cupsd" pid=6365 comm="cupsd" capability=12 capname="net_admin" [22570.680650] systemd-journald[328]:
Bug#1025537: nfsd: Kernel Oops while serving NFS
Source: linux Followup-For: Bug #1025537 The 6.1.0 Changelog had some notes about a gnarly bug in the NFS bug having been fixed, so I thought I'd give it a try. Unfortunately, the problem reported here is still present. It has made the machine quite unstable, but I was able to retrieve logs of previous Oopses. 2023-03-02T12:51:02.923421+11:00 supahwinch kernel: [0.00] Linux version 6.1.0-5-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.12-1 (2023-02-15) [SNIP]-8< 2023-03-02T13:11:21.468088+11:00 supahwinch kernel: [ 1262.015083] stack segment: [#1] PREEMPT SMP NOPTI 2023-03-02T13:11:21.473246+11:00 supahwinch kernel: [ 1262.019939] CPU: 0 PID: 3345 Comm: nfsd Not tainted 6.1.0-5-amd64 #1 Debian 6.1.12-1 2023-03-02T13:11:21.473258+11:00 supahwinch kernel: [ 1262.024779] Hardware name: HP ProLiant MicroServer, BIOS O41 10/01/2013 2023-03-02T13:11:21.473261+11:00 supahwinch kernel: [ 1262.029613] RIP: 0010:release_pages+0xcd/0x4d0 2023-03-02T13:11:21.473264+11:00 supahwinch kernel: [ 1262.034410] Code: 84 c0 74 1a 48 8b 04 24 48 8d 4c 24 30 49 89 46 08 48 89 44 24 30 4c 89 75 08 48 89 4d 10 49 83 c7 08 4c 39 fb 74 75 49 8b 2f <48> 8b 45 08 a8 01 0f 85 58 01 00 00 0f 1f 44 00 00 4d 85 ed 74 0e 2023-03-02T13:11:21.473268+11:00 supahwinch kernel: [ 1262.044534] RSP: 0018:bc550156fe30 EFLAGS: 00010206 2023-03-02T13:11:21.473271+11:00 supahwinch kernel: [ 1262.049614] RAX: 0007 RBX: 97c2ae3b4b78 RCX: bc550156fe60 2023-03-02T13:11:21.473274+11:00 supahwinch kernel: [ 1262.054703] RDX: bc550156fe60 RSI: bc550156fe60 RDI: de0185ba2148 2023-03-02T13:11:21.473277+11:00 supahwinch kernel: [ 1262.059850] RBP: 000fc000 R08: de0180d57d08 R09: 00019198 2023-03-02T13:11:21.473279+11:00 supahwinch kernel: [ 1262.064975] R10: 0003 R11: R12: 2023-03-02T13:11:21.473282+11:00 supahwinch kernel: [ 1262.070058] R13: R14: de0180d57d08 R15: 97c2ae3b4b28 2023-03-02T13:11:21.473284+11:00 supahwinch kernel: [ 1262.075101] FS: () GS:97c357c0() knlGS: 2023-03-02T13:11:21.473287+11:00 supahwinch kernel: [ 1262.080166] CS: 0010 DS: ES: CR0: 80050033 2023-03-02T13:11:21.473290+11:00 supahwinch kernel: [ 1262.085268] CR2: 7f6e0e512416 CR3: 000109c92000 CR4: 06f0 2023-03-02T13:11:21.473292+11:00 supahwinch kernel: [ 1262.090377] Call Trace: 2023-03-02T13:11:21.473295+11:00 supahwinch kernel: [ 1262.095402] 2023-03-02T13:11:21.473298+11:00 supahwinch kernel: [ 1262.100346] ? nfsd_shutdown_threads+0x90/0x90 [nfsd] 2023-03-02T13:11:21.473300+11:00 supahwinch kernel: [ 1262.105417] __pagevec_release+0x1b/0x30 2023-03-02T13:11:21.473303+11:00 supahwinch kernel: [ 1262.110399] svc_xprt_release+0x1a3/0x1e0 [sunrpc] 2023-03-02T13:11:21.473306+11:00 supahwinch kernel: [ 1262.115547] svc_send+0x59/0x160 [sunrpc] 2023-03-02T13:11:21.473308+11:00 supahwinch kernel: [ 1262.120602] nfsd+0xd5/0x190 [nfsd] 2023-03-02T13:11:21.473311+11:00 supahwinch kernel: [ 1262.125538] kthread+0xe9/0x110 2023-03-02T13:11:21.473313+11:00 supahwinch kernel: [ 1262.130372] ? kthread_complete_and_exit+0x20/0x20 2023-03-02T13:11:21.473315+11:00 supahwinch kernel: [ 1262.135195] ret_from_fork+0x22/0x30 2023-03-02T13:11:21.473318+11:00 supahwinch kernel: [ 1262.139953] 2023-03-02T13:11:21.473321+11:00 supahwinch kernel: [ 1262.144620] Modules linked in: veth xt_nat nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc overlay cmac algif_hash ecb algif_skcipher af_alg bnep ip6t_REJECT nf_reject_ipv6 xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink binfmt_misc amdgpu isofs gpu_sched cdrom drm_buddy xc2028 zl10353 rc_fusionhdtv_mce ir_kbd_i2c cx23885 altera_ci tda18271 altera_stapl m88ds3103 i2c_mux btusb btrtl cx2341x btbcm tveeprom btintel videobuf2_dvb dvb_core btmtk bluetooth videobuf2_dma_sg videobuf2_memops videobuf2_v4l2 amd64_edac jitterentropy_rng videobuf2_common edac_mce_amd radeon sha512_generic kvm_amd videodev video ctr ccp wmi mc drbg cp210x rng_core snd_pcm ansi_cprng usbserial drm_display_helper ecdh_generic snd_timer joydev snd rfkill kvm cec soundcore evdev ecc pcspkr rc_core drm_ttm_helper xfs ttm irqbypass drm_kms_helper sp5100_tco watchdog i2c_algo_bit acpi_cpufreq sg button 2023-03-02T13:11:21.473327+11:00 supahwinch kernel: [ 1262.144868] k10temp w83795 jc42 tun loop nfsd msr auth_rpcgss nfs_acl lockd parport_pc grace ppdev sunrpc lp drm parport efi_pstore fuse configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic zstd_compress raid10 raid456
Bug#1025537: nfsd: Kernel Oops while serving NFS
Package: linux-image-5.19.0-10315-g310d9d5a5009 Followup-For: Bug #1025537 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Olivier Mehani To: Debian Bug Tracking System <1025...@bugs.debian.org> Subject: Re: nfsd: Kernel Oops while serving NFS Message-ID: <167361292314.1643173.3991370234085262557.report...@supahwinch.narf.ssji.net> X-Mailer: reportbug 11.6.0 Date: Fri, 13 Jan 2023 23:28:43 +1100 Package: linux-image-5.19.0-10316-gf0f6b614f83d Followup-For: Bug #1025537 OK, I think I have a strong suspect [0]: f0f6b614f83dbae99d283b7b12ab5dd2e04df979 (tags/v6.0-rc1~55^2~2) This is a pretty generic commit, that doesn't seem to touch the nfs stack, but changes something that looks deep enough in the iovec code that it could have an impact. I have solidly re-tested the preceding commit, 310d9d5a5009a93377200b98daa2d84aa2bd8160 (tags/v6.0-rc1~55^2~3), and the Oops doesn't happen there. There is a series of commits that were suggested as part of the `git bisect`, that I wasn't able to build and had to skip. The error was FAILED: load BTF from vmlinux: Invalid argument but no amount of cajoling `pahole` has helped. The merge parent for the series is at 8447d0e75099eb54eea9306c2d43ecfc956d09ed (tags/v6.0-rc1~56^2), which is the preceding merge to the one containing the first bad commit above. I think this is noteworthy as a lot of the commits on this branch are labeled as `remoteproc` which I, as a lay person, think might be related to RPCs. Is there any other tests that I can perform to help diagnose / fix this further? [0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f0f6b614f83dbae99d283b7b12ab5dd2e04df97 (but dockerd Oops, 10623384), nah, BAD -- System Information: Debian Release: bookworm/sid APT prefers testing APT policy: (990, 'testing') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 5.19.0-10315-g310d9d5a5009 (SMP w/2 CPU threads; PREEMPT) Locale: LANG=en_AU, LC_CTYPE=en_AU (charmap=UTF-8) (ignored: LC_ALL set to en_AU.UTF8), LANGUAGE=en_AU:en_GB Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled -- no debconf information
Bug#1025537: nfsd: Kernel Oops while serving NFS
Package: linux-image-5.19.0-13797-ge091ba5cf827 Version: 5.19.0-13797-ge091ba5cf827-17 Followup-For: Bug #1025537 Nope... I just re-triggered the Oops on the preceding commit, e091ba5cf827. I'll restart the bisection from there... [ 2976.696730] Oops: [#1] PREEMPT SMP NOPTI [ 2976.700471] CPU: 0 PID: 1969 Comm: nfsd Not tainted 5.19.0-13797-ge091ba5cf827 #17 [ 2976.704341] Hardware name: HP ProLiant MicroServer, BIOS O41 10/01/2013 [ 2976.708272] RIP: 0010:kernel_sendpage+0x1d/0x120 [ 2976.712213] Code: 01 00 e9 46 ff ff ff 66 0f 1f 44 00 00 0f 1f 44 00 00 53 48 89 fb 48 83 ec 18 48 8b 47 20 4c 8b 88 a0 00 00 00 4d 85 c9 74 35 <48> 8b 46 08 a8 01 75 63 66 90 48 89 f0 48 8b 00 f6 c4 02 74 5c 80 [ 2976.720602] RSP: 0018:a7f641733dc0 EFLAGS: 00010282 [ 2976.724805] RAX: b1933e60 RBX: 9897918e7400 RCX: 1000 [ 2976.729097] RDX: RSI: RDI: 9897918e7400 [ 2976.733433] RBP: 9896dab74280 R08: R09: b12492e0 [ 2976.737779] R10: 0003 R11: R12: 9897918e7400 [ 2976.742168] R13: 1000 R14: 9897aa997010 R15: b8fd [ 2976.746524] FS: () GS:9897d7c0() knlGS: [ 2976.750910] CS: 0010 DS: ES: CR0: 80050033 [ 2976.755323] CR2: 0008 CR3: 0001ff94a000 CR4: 06f0 [ 2976.759755] Call Trace: [ 2976.764102] [ 2976.768390] ? sock_sendmsg+0x58/0x70 [ 2976.772648] svc_tcp_sendmsg+0x121/0x180 [sunrpc] [ 2976.776967] ? nfsd_shutdown_threads+0x90/0x90 [nfsd] [ 2976.781283] svc_tcp_sendto+0x90/0x190 [sunrpc] [ 2976.785577] svc_send+0x4d/0x160 [sunrpc] [ 2976.789807] nfsd+0xd5/0x190 [nfsd] [ 2976.794006] kthread+0xe9/0x110 [ 2976.798048] ? kthread_complete_and_exit+0x20/0x20 [ 2976.802159] ret_from_fork+0x22/0x30 [ 2976.806223] [ 2976.810202] Modules linked in: veth nf_conntrack_netlink xfrm_user xfrm_algo br_netfilter bridge stp llc overlay tls cmac algif_hash ecb algif_skcipher af_alg bnep ip6t_REJECT nf_reject_ipv6 nft_chain_nat xt_nat xt_MASQUERADE nf_nat xt_addrtype ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink binfmt_misc amdgpu gpu_sched drm_buddy xc2028 zl10353 rc_fusionhdtv_mce ir_kbd_i2c cx23885 btusb tveeprom btrtl altera_ci btbcm cx2341x btintel btmtk tda18271 snd_pcm bluetooth snd_timer isofs rfkill amd64_edac snd jitterentropy_rng cdrom soundcore sha512_ssse3 edac_mce_amd sha512_generic altera_stapl kvm_amd radeon videobuf2_dvb ctr drbg videobuf2_dma_sg videobuf2_memops cp210x ansi_cprng m88ds3103 ccp i2c_mux usbserial i2c_algo_bit dvb_core videobuf2_v4l2 drm_ttm_helper rng_core evdev videobuf2_common joydev ttm kvm ecdh_generic drm_display_helper irqbypass ecc videodev xfs drm_kms_helper mc cec pcspkr k10temp rc_core [ 2976.810298] sp5100_tco button watchdog sg acpi_cpufreq w83795 jc42 tun loop msr nfsd parport_pc ppdev auth_rpcgss lp nfs_acl parport lockd grace drm efi_pstore sunrpc fuse configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor uas usb_storage raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod dm_mod hid_generic usbhid hid sd_mod t10_pi crc64_rocksoft_generic crc64_rocksoft crc_t10dif crct10dif_generic crc64 crct10dif_common ahci libahci libata ohci_pci scsi_mod ohci_hcd ehci_pci ehci_hcd scsi_common tg3 i2c_piix4 usbcore ptp pps_core libphy usb_common [ 2976.876303] CR2: 0008 [ 2976.881421] ---[ end trace ]--- [ 2976.886579] RIP: 0010:kernel_sendpage+0x1d/0x120 [ 2976.891691] Code: 01 00 e9 46 ff ff ff 66 0f 1f 44 00 00 0f 1f 44 00 00 53 48 89 fb 48 83 ec 18 48 8b 47 20 4c 8b 88 a0 00 00 00 4d 85 c9 74 35 <48> 8b 46 08 a8 01 75 63 66 90 48 89 f0 48 8b 00 f6 c4 02 74 5c 80 [ 2976.902360] RSP: 0018:a7f641733dc0 EFLAGS: 00010282 [ 2976.907711] RAX: b1933e60 RBX: 9897918e7400 RCX: 1000 [ 2976.913121] RDX: RSI: RDI: 9897918e7400 [ 2976.918506] RBP: 9896dab74280 R08: R09: b12492e0 [ 2976.923843] R10: 0003 R11: R12: 9897918e7400 [ 2976.929141] R13: 1000 R14: 9897aa997010 R15: b8fd [ 2976.934339] FS: () GS:9897d7c0() knlGS: [ 2976.939481] CS: 0010 DS: ES: CR0: 80050033 [ 2976.944570] CR2: 0008 CR3: 0001ff94a000 CR4: 06f0 -- System Information: Debian Release: bookworm/sid APT prefers testing APT policy: (990, 'testing') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 5.19.0-13797-ge091ba5cf827 (SMP w/2 CPU threads; PREEMPT) Kernel taint flags: TAINT_DIE Locale: LANG=en_AU, LC_CTYPE=en_AU (charmap=UTF-8)
Bug#1025537: nfsd: Kernel Oops while serving NFS
Package: linux-image-5.19.0-13930-g7ebfc85e2cd7 Version: 5.19.0-13930-g7ebfc85e2cd7-23 Followup-For: Bug #1025537 Ah, here we go! It just Oopsed again, on what is presumably the first commit introducing the issue. I'm gonna try to buidthe previous commit and see if it's happy for a while. [ 2047.224083] Oops: [#1] PREEMPT SMP NOPTI [ 2047.228190] CPU: 1 PID: 1987 Comm: nfsd Not tainted 5.19.0-13930-g7ebfc85e2cd7 #23 [ 2047.232427] Hardware name: HP ProLiant MicroServer, BIOS O41 10/01/2013 [ 2047.236740] RIP: 0010:kernel_sendpage+0x1d/0x120 [ 2047.241021] Code: 01 00 e9 46 ff ff ff 66 0f 1f 44 00 00 0f 1f 44 00 00 53 48 89 fb 48 83 ec 18 48 8b 47 20 4c 8b 88 a0 00 00 00 4d 85 c9 74 35 <48> 8b 46 08 a8 01 75 63 66 90 48 89 f0 48 8b 00 f6 c4 02 74 5c 80 [ 2047.250067] RSP: 0018:b12541e2fdc0 EFLAGS: 00010282 [ 2047.254722] RAX: 86733e60 RBX: 88f681d5b740 RCX: 1000 [ 2047.259428] RDX: RSI: RDI: 88f681d5b740 [ 2047.264219] RBP: 88f7aede4280 R08: R09: 860491b0 [ 2047.268977] R10: R11: 88f785ed0900 R12: 88f681d5b740 [ 2047.273716] R13: 1000 R14: 88f8645ed010 R15: 181e [ 2047.278481] FS: () GS:88f897c8() knlGS: [ 2047.283269] CS: 0010 DS: ES: CR0: 80050033 [ 2047.288017] CR2: 0008 CR3: 000160e42000 CR4: 06e0 [ 2047.292772] Call Trace: [ 2047.297454] [ 2047.302061] ? sock_sendmsg+0x58/0x70 [ 2047.306715] svc_tcp_sendmsg+0x121/0x180 [sunrpc] [ 2047.311515] ? nfsd_shutdown_threads+0x90/0x90 [nfsd] [ 2047.316291] svc_tcp_sendto+0x90/0x190 [sunrpc] [ 2047.321083] svc_send+0x4d/0x160 [sunrpc] [ 2047.325811] nfsd+0xd5/0x190 [nfsd] [ 2047.330486] kthread+0xe9/0x110 [ 2047.334999] ? kthread_complete_and_exit+0x20/0x20 [ 2047.339497] ret_from_fork+0x22/0x30 [ 2047.343930] [ 2047.348280] Modules linked in: veth nf_conntrack_netlink xfrm_user xfrm_algo br_netfilter bridge stp llc overlay cmac algif_hash ecb algif_skcipher af_alg bnep tls ip6t_REJECT nf_reject_ipv6 nft_chain_nat xt_nat xt_MASQUERADE nf_nat xt_addrtype ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink binfmt_misc amdgpu gpu_sched drm_buddy xc2028 zl10353 rc_fusionhdtv_mce ir_kbd_i2c cx23885 tveeprom altera_ci cx2341x btusb tda18271 btrtl snd_pcm btbcm snd_timer btintel snd btmtk soundcore bluetooth altera_stapl rfkill videobuf2_dvb jitterentropy_rng sha512_ssse3 isofs videobuf2_dma_sg radeon cdrom sha512_generic videobuf2_memops ctr m88ds3103 amd64_edac edac_mce_amd drbg i2c_algo_bit i2c_mux cp210x ansi_cprng drm_ttm_helper dvb_core joydev kvm_amd videobuf2_v4l2 ttm videobuf2_common drm_display_helper usbserial ccp evdev rng_core drm_kms_helper videodev ecdh_generic cec mc kvm ecc rc_core irqbypass xfs pcspkr sp5100_tco [ 2047.348518] k10temp sg watchdog button acpi_cpufreq w83795 jc42 tun loop msr nfsd parport_pc ppdev auth_rpcgss lp nfs_acl lockd parport grace sunrpc efi_pstore drm fuse configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic zstd_compress uas usb_storage raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod dm_mod hid_generic usbhid hid sd_mod t10_pi crc64_rocksoft_generic crc64_rocksoft crc_t10dif crct10dif_generic crc64 crct10dif_common ahci libahci ohci_pci libata tg3 ohci_hcd ptp ehci_pci ehci_hcd i2c_piix4 usbcore scsi_mod pps_core libphy scsi_common usb_common [ 2047.421645] CR2: 0008 [ 2047.427434] ---[ end trace ]--- [ 2047.433257] RIP: 0010:kernel_sendpage+0x1d/0x120 [ 2047.439046] Code: 01 00 e9 46 ff ff ff 66 0f 1f 44 00 00 0f 1f 44 00 00 53 48 89 fb 48 83 ec 18 48 8b 47 20 4c 8b 88 a0 00 00 00 4d 85 c9 74 35 <48> 8b 46 08 a8 01 75 63 66 90 48 89 f0 48 8b 00 f6 c4 02 74 5c 80 [ 2047.451057] RSP: 0018:b12541e2fdc0 EFLAGS: 00010282 [ 2047.457127] RAX: 86733e60 RBX: 88f681d5b740 RCX: 1000 [ 2047.463252] RDX: RSI: RDI: 88f681d5b740 [ 2047.469361] RBP: 88f7aede4280 R08: R09: 860491b0 [ 2047.475419] R10: R11: 88f785ed0900 R12: 88f681d5b740 [ 2047.481451] R13: 1000 R14: 88f8645ed010 R15: 181e [ 2047.487369] FS: () GS:88f897c8() knlGS: [ 2047.493199] CS: 0010 DS: ES: CR0: 80050033 [ 2047.498970] CR2: 0008 CR3: 000160e42000 CR4: 06e0
Bug#1025537: nfsd: Kernel Oops while serving NFS
Package: linux-image-5.19.0-13930-g7ebfc85e2cd7 Version: 5.19.0-13930-g7ebfc85e2cd7-23 Followup-For: Bug #1025537 Ok, so the bisect got me down to commit 7ebfc85e2cd7b08f518b526173e9a33b56b3913b (v6.0-rc1~28). This is unfortunate, as it is quite a large one [0]. A caveat to this report is that I'm not 100% sure my way of triggering the issue always worked. This commit OOPSed the first time round, but when it was found to be the first faulty commit at the end of the bisect pass (logs below), I could no longer trigger the bug on a newly rebuilt kernel off the same commit. I now realise that I didn't clean between builds, but I assume `make bindeb-pkg` would start from a fresh clean state. This commit doesn't seem to directly touch NFS code, but does touch the networking state, so it is still a plausible suspect. As per my previous message, the latest kernel also exhibit the issue. Worse so, it seems to even hang the kernel, which didn't seem to happen on earlier versions. [0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7ebfc85e2cd7b08f518b526173e9a33b56b3913b -- System Information: Debian Release: bookworm/sid APT prefers testing APT policy: (990, 'testing') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 5.19.0-13930-g7ebfc85e2cd7 (SMP w/2 CPU threads; PREEMPT) Locale: LANG=en_AU, LC_CTYPE=en_AU (charmap=UTF-8) (ignored: LC_ALL set to en_AU.UTF8), LANGUAGE=en_AU:en_GB Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled -- no debconf information -- Bisect log: git bisect start # good: [3d7cb6b04c3f3115719235cc6866b10326de34cd] Linux 5.19 git bisect good 3d7cb6b04c3f3115719235cc6866b10326de34cd # good: [fcf22aefe87101424563a8dd8adec8a75b8c7576] Linux 5.19.11 git bisect good fcf22aefe87101424563a8dd8adec8a75b8c7576 # bad: [e60276b8c11ab4a8be23807bc67b048cfb937dfa] Linux 6.0.8 git bisect bad e60276b8c11ab4a8be23807bc67b048cfb937dfa # good: [7c5c3a6177fa9646884114fc7f2e970b0bc50dc9] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm git bisect good 7c5c3a6177fa9646884114fc7f2e970b0bc50dc9 # bad: [5e2e7383b57fa03ec2b00c82bb7f49a4a707c1f7] Merge tag 'pinctrl-v6.0-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect bad 5e2e7383b57fa03ec2b00c82bb7f49a4a707c1f7 # bad: [83e4b196838d90799a8879e5054a3beecf9ed256] selftests: forwarding: add shebang for sch_red.sh git bisect bad 83e4b196838d90799a8879e5054a3beecf9ed256 # bad: [d974730c8884cd784810b4f2fe903ac882a5fec9] Merge branch 'net-lantiq_xrx200-fix-errors-under-memory-pressure' git bisect bad d974730c8884cd784810b4f2fe903ac882a5fec9 # skip: [7a53e17accce9d310d2e522dfc701d8da7ccfa65] Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost git bisect skip 7a53e17accce9d310d2e522dfc701d8da7ccfa65 # bad: [96f86ff08332d88defd35c330fc6dae219b9e264] Merge tag 'perf-tools-fixes-for-v6.0-2022-08-13' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux git bisect bad 96f86ff08332d88defd35c330fc6dae219b9e264 # bad: [e140f731f9807035e967c401198171f316744696] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi git bisect bad e140f731f9807035e967c401198171f316744696 # bad: [6c833c0581f1c15db2e0344da19360cba75a3351] Merge tag 'devicetree-fixes-for-6.0-1' of git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux git bisect bad 6c833c0581f1c15db2e0344da19360cba75a3351 # skip: [8b30b09317ec6fda5f36a428e9e331253b5c4739] dt-bindings: rtc: nuvoton: add NCT3018Y Real Time Clock git bisect skip 8b30b09317ec6fda5f36a428e9e331253b5c4739 # good: [668c3c237f5ddc2889879b08f26d2374231f3287] Merge tag 'sound-6.0-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound git bisect good 668c3c237f5ddc2889879b08f26d2374231f3287 # good: [a9e9c93966afdaae74a6a7533552391646b93f2c] Documentation/mm: add details about kmap_local_page() and preemption git bisect good a9e9c93966afdaae74a6a7533552391646b93f2c # good: [c235698355fa94df7073b51befda7d4be00a0e23] Merge tag 'cxl-for-6.0' of git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl git bisect good c235698355fa94df7073b51befda7d4be00a0e23 # bad: [7ebfc85e2cd7b08f518b526173e9a33b56b3913b] Merge tag 'net-6.0-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net git bisect bad 7ebfc85e2cd7b08f518b526173e9a33b56b3913b # good: [786da5da5671c2d4cf812fe1ccc980bdde30c69e] Merge tag 'ceph-for-5.20-rc1' of https://github.com/ceph/ceph-client git bisect good 786da5da5671c2d4cf812fe1ccc980bdde30c69e # good: [996237d9ba4d092469fbfca18995656c32834ac7] Merge branch 'do-not-use-rt_tos-for-ipv6-flowlabel' git bisect good 996237d9ba4d092469fbfca18995656c32834ac7 # good: [fbe8870f72e8e71bb57b883d29c600aaaca6cd20] Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf git bisect good fbe8870f72e8e71bb57b883d29c600aaaca6cd20 # good: [e091ba5cf82714c8691d978781696cd1fc2dec70] Merge tag
Bug#1025537: nfsd: Kernel Oops while serving NFS
Package: src:linux Followup-For: Bug #1025537 Confirming this still happens on the latest version of the kernel. -- Package-specific info: ** Version: Linux version 6.0.0-6-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-9.1) 12.2.0, GNU ld (GNU Binutils for Debian) 2.39) #1 SMP PREEMPT_DYNAMIC Debian 6.0.12-1 (2022-12-09) ** Command line: BOOT_IMAGE=/boot/vmlinuz-6.0.0-6-amd64 root=UUID=0c917590-acc7-464c-8961-64f79e1d1c69 ro init=/lib/systemd/systemd init=/lib/systemd/systemd ** Tainted: D (128) * kernel died recently, i.e. there was an OOPS or BUG ** Kernel log: ** Model information [ 2046.604163] __pagevec_release+0x1b/0x30 [ 2046.610505] svc_xprt_release+0x1a3/0x1e0 [sunrpc] [ 2046.617070] svc_send+0x59/0x160 [sunrpc] [ 2046.623473] nfsd+0xd5/0x190 [nfsd] [ 2046.629656] kthread+0xe9/0x110 [ 2046.635602] ? kthread_complete_and_exit+0x20/0x20 [ 2046.641456] ret_from_fork+0x22/0x30 [ 2046.647159] [ 2046.652696] Modules linked in: veth nf_conntrack_netlink xfrm_user xfrm_algo br_netfilter bridge stp llc overlay tls cmac algif_hash ecb algif_skcipher af_alg bnep ip6t_REJECT nf_reject_ipv6 nft_chain_nat xt_nat xt_MASQUERADE nf_nat xt_addrtype ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink binfmt_misc amdgpu gpu_sched drm_buddy xc2028 zl10353 rc_fusionhdtv_mce ir_kbd_i2c cx23885 altera_ci tda18271 btusb altera_stapl btrtl btbcm m88ds3103 btintel i2c_mux btmtk cx2341x bluetooth tveeprom videobuf2_dvb dvb_core jitterentropy_rng isofs videobuf2_dma_sg videobuf2_memops videobuf2_v4l2 sha512_ssse3 videobuf2_common sha512_generic amd64_edac videodev cdrom edac_mce_amd radeon ctr kvm_amd mc drbg snd_pcm ccp snd_timer ansi_cprng snd drm_display_helper rng_core cec ecdh_generic cp210x soundcore rc_core usbserial kvm drm_ttm_helper joydev evdev rfkill xfs ttm irqbypass ecc drm_kms_helper i2c_algo_bit pcspkr k10temp [ 2046.652939] sp5100_tco watchdog sg button acpi_cpufreq w83795 jc42 tun loop msr nfsd parport_pc ppdev lp auth_rpcgss nfs_acl parport lockd grace sunrpc drm efi_pstore fuse configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic zstd_compress uas usb_storage raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod dm_mod hid_generic usbhid hid sd_mod t10_pi crc64_rocksoft_generic crc64_rocksoft crc_t10dif crct10dif_generic crc64 crct10dif_common ohci_pci ahci libahci tg3 libata ohci_hcd ehci_pci ehci_hcd usbcore i2c_piix4 libphy scsi_mod usb_common ptp pps_core scsi_common [ 2046.733846] ---[ end trace ]--- [ 2046.739371] RIP: 0010:release_pages+0xcd/0x500 [ 2046.744879] Code: 84 c0 74 1a 48 8b 04 24 48 8d 54 24 30 49 89 46 08 48 89 44 24 30 4c 89 75 08 48 89 55 10 49 83 c7 08 4c 39 fb 74 75 49 8b 2f <48> 8b 45 08 a8 01 0f 85 58 01 00 00 0f 1f 44 00 00 4d 85 ed 74 0e [ 2046.756004] RSP: 0018:af7a811cfe30 EFLAGS: 00010206 [ 2046.761509] RAX: 0007 RBX: 8faf16948b78 RCX: eaa8c45c8608 [ 2046.767032] RDX: af7a811cfe60 RSI: af7a811cfe60 RDI: eaa8c45c8608 [ 2046.772518] RBP: 0017c000 R08: eaa8c4ba4988 R09: 00018bf0 [ 2046.777973] R10: 0003 R11: R12: [ 2046.783387] R13: R14: eaa8c4ba4988 R15: 8faf16948b28 [ 2046.788737] FS: () GS:8fb017c8() knlGS: [ 2046.794102] CS: 0010 DS: ES: CR0: 80050033 [ 2046.799361] CR2: 7f505ca0db28 CR3: 000108692000 CR4: 06e0 [ 2047.009929] stack segment: [#7] PREEMPT SMP NOPTI [ 2047.015823] CPU: 1 PID: 2244 Comm: nfsd Tainted: G D 6.0.0-6-amd64 #1 Debian 6.0.12-1 [ 2047.021769] Hardware name: HP ProLiant MicroServer, BIOS O41 10/01/2013 [ 2047.027727] RIP: 0010:release_pages+0xcd/0x500 [ 2047.033679] Code: 84 c0 74 1a 48 8b 04 24 48 8d 54 24 30 49 89 46 08 48 89 44 24 30 4c 89 75 08 48 89 55 10 49 83 c7 08 4c 39 fb 74 75 49 8b 2f <48> 8b 45 08 a8 01 0f 85 58 01 00 00 0f 1f 44 00 00 4d 85 ed 74 0e [ 2047.046097] RSP: 0018:af7a81133e30 EFLAGS: 00010206 [ 2047.052343] RAX: 0007 RBX: 8faf1685cb78 RCX: eaa8c45a3e08 [ 2047.058605] RDX: af7a81133e60 RSI: af7a81133e60 RDI: eaa8c45a3e08 [ 2047.064809] RBP: 0017c000 R08: eaa8c45cba48 R09: 0077 [ 2047.071008] R10: 2d40 R11: R12: [ 2047.077241] R13: R14: eaa8c45cba48 R15: 8faf1685cb28 [ 2047.083490] FS: () GS:8fb017c8() knlGS: [ 2047.089783] CS: 0010 DS: ES: CR0: 80050033 [ 2047.096089] CR2: 7f505ca0db28 CR3: 00016dafa000 CR4: 06e0 [ 2047.102466] Call Trace: [ 2047.108808] [ 2047.115136] ? nfsd_shutdown_threads+0x90/0x90
Bug#1025537: nfsd: Kernel Oops while serving NFS
Hi Olivier, On Sun, Dec 11, 2022 at 12:01:31AM +1100, Olivier Mehani wrote: > Hi Salvatore, > > On Fri 09 Dec 2022 at 22:22:49 +0100, Salvatore Bonaccorso wrote: > > > On testing/bookworm, since booting on a 6-versioned linux-image, I have > > > noticed frequent hang ups of the nfs server, rendering it mostly > > > unusable. This is accompanied with Kernel Oops in the dmesg. > > > This sounds similar to previous bugs #1014793 and #1020548, both RESOLVED > > > by > > > later patches. > > How easy can you trigger and reproduce the issue? If you can easily > > reach that situation, can you try to bisect the issue? Easiest would > > be to first pin point between Debian revisions, and later further > > bisect in upstream stable series. > > Do you have the possibility to do that? > > It seems to be fairly reliable. > > I'll give that a go, starting with > * linux-image-5.19.0-2-amd64_5.19.11-1_amd64.deb > * linux-image-6.0.0-1-amd64_6.0.2-1_amd64.deb Great, thanks. > I'm not certain how to bisect further. Which source and which kconfig should > I use to build intermediate commits? Does the following reference helps you further? https://wiki.debian.org/DebianKernel/GitBisect Thanks for taking time for pinpointing your issue. Regards, Salvatore
Bug#1025537: nfsd: Kernel Oops while serving NFS
Hi Salvatore, On Fri 09 Dec 2022 at 22:22:49 +0100, Salvatore Bonaccorso wrote: On testing/bookworm, since booting on a 6-versioned linux-image, I have noticed frequent hang ups of the nfs server, rendering it mostly unusable. This is accompanied with Kernel Oops in the dmesg. This sounds similar to previous bugs #1014793 and #1020548, both RESOLVED by later patches. How easy can you trigger and reproduce the issue? If you can easily reach that situation, can you try to bisect the issue? Easiest would be to first pin point between Debian revisions, and later further bisect in upstream stable series. Do you have the possibility to do that? It seems to be fairly reliable. I'll give that a go, starting with * linux-image-5.19.0-2-amd64_5.19.11-1_amd64.deb * linux-image-6.0.0-1-amd64_6.0.2-1_amd64.deb I'm not certain how to bisect further. Which source and which kconfig should I use to build intermediate commits? -- Olivier Mehani PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE F5F9 F012 A6E2 98C6 6655 Confidentiality cannot be guaranteed on emails sent or received unencrypted.
Bug#1025537: nfsd: Kernel Oops while serving NFS
Hi Olivier, On Tue, Dec 06, 2022 at 10:54:31PM +1100, Olivier Mehani wrote: > Package: src:linux > Version: 6.0.10-1 > Severity: important > File: nfsd > > Dear Maintainer, > > *** Reporter, please consider answering these questions, where appropriate *** > >* What led up to the situation? > > On testing/bookworm, since booting on a 6-versioned linux-image, I have > noticed frequent hang ups of the nfs server, rendering it mostly > unusable. This is accompanied with Kernel Oops in the dmesg. > > This sounds similar to previous bugs #1014793 and #1020548, both RESOLVED by > later patches. How easy can you trigger and reproduce the issue? If you can easily reach that situation, can you try to bisect the issue? Easiest would be to first pin point between Debian revisions, and later further bisect in upstream stable series. Do you have the possibility to do that? Regards, Salvatore
Bug#1025537: nfsd: Kernel Oops while serving NFS
On Tue 06 Dec 2022 at 15:16:06 +0100, Diederik de Haas wrote: What's the version of NFS you're using? The clients are recent Linux hosts, and one Kodi 19.x. I don't specify a version in either their fstab or the server's exports. From my reading of the manpages, it seems that the client will start from 4.2 and go down. For the server, it _may_ vary due to the /etc/exports. I include mine below, as I'm not certain how to understand the `fsid` logic, though I think in my case it's a simple tree rooted at /. /home 192.168.103.0/24(rw,no_subtree_check,crossmnt,all_squash,anongid=999) :::::/64(rw,no_subtree_check,crossmnt,all_squash,anongid=999) 192.168.42.0/24(rw,no_subtree_check,crossmnt,all_squash,anongid=999) /data 192.168.103.0/24(rw,no_subtree_check,crossmnt,all_squash,anongid=999) :::::/64(rw,no_subtree_check,crossmnt,all_squash,anongid=999) 192.168.42.0/24(rw,no_subtree_check,crossmnt,all_squash,anongid=999) /srv/debian-live 192.168.103.0/24(ro,async,subtree_check,no_root_squash) :::::/64(ro,async,subtree_check,no_root_squash) 192.168.42.0/24(ro,async,subtree_check,no_root_squash) /run/archiso/bootmnt 192.168.103.0/24(ro,async,subtree_check,no_root_squash) :::::/64(ro,async,subtree_check,no_root_squash) 192.168.42.0/24(ro,async,subtree_check,no_root_squash) Note that additional filesystems are mounted in subdirectories of /data. -- Olivier Mehani PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE F5F9 F012 A6E2 98C6 6655 Confidentiality cannot be guaranteed on emails sent or received unencrypted.
Bug#1025537: nfsd: Kernel Oops while serving NFS
On Tue 06 Dec 2022 at 23:11:14 +0100, Diederik de Haas wrote: The clients are recent Linux hosts, and one Kodi 19.x. I don't specify a version in either their fstab or the server's exports. Sorry, I meant the package version of the NFS server Ah! No worries. [9:12:04] @supahwinch ~$ dpkg -l *nfs* Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==---= un knfs (no description available) un libnfs1 (no description available) ii libnfs13:amd64 4.0.0-1 amd64NFS client library (shared library) un libnfs4 (no description available) un libnfsidmap-dev (no description available) un libnfsidmap-regex(no description available) ii libnfsidmap1:amd64 1:2.6.2-2amd64NFS idmapping library un libnfsidmap2 (no description available) un nfs-client (no description available) ii nfs-common 1:2.6.2-2amd64NFS support files common to client and server ii nfs-kernel-server 1:2.6.2-2amd64support for NFS kernel server un nfs-server (no description available) And the relevant packages on one (ArchLinux) client: [9:16:20] ~$ pacman -Qs nfs | grep local ... local/gvfs-nfs 1.50.2-1 (gnome) local/libnfs 5.0.2-1 local/nfs-utils 2.6.2-1 local/nfsidmap 2.6.2-1 local/qemu-block-nfs 7.1.0-10 local/unionfs-fuse 3.2-1 -- Olivier Mehani PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE F5F9 F012 A6E2 98C6 6655 Confidentiality cannot be guaranteed on emails sent or received unencrypted.
Bug#1025537: nfsd: Kernel Oops while serving NFS
Ok, looks like v3, if I read this correctly. [9:09:42] @supahwinch ~$ /usr/sbin/rpcinfo -p 130 ↵ program vers proto port service 104 tcp111 portmapper 103 tcp111 portmapper 102 tcp111 portmapper 104 udp111 portmapper 103 udp111 portmapper 102 udp111 portmapper 151 udp 4002 mountd 151 tcp 4002 mountd 152 udp 4002 mountd 152 tcp 4002 mountd 153 udp 4002 mountd 153 tcp 4002 mountd 1000241 udp 4000 status 1000241 tcp 4000 status 133 tcp 2049 nfs 134 tcp 2049 nfs 1002273 tcp 2049 nfs_acl 1000211 udp 53497 nlockmgr 1000213 udp 53497 nlockmgr 1000214 udp 53497 nlockmgr 1000211 tcp 38889 nlockmgr 1000213 tcp 38889 nlockmgr 1000214 tcp 38889 nlockmgr [9:09:44] @supahwinch ~$ sudo nfsstat –s 0s Server rpc stats: calls badcalls badfmt badauthbadclnt 447324 0 0 0 0 Server nfs v3: null getattr setattr lookup access 4 0% 15575 3% 0 0% 399007 89% 8 0% readlink read writecreate mkdir 9478 2% 630% 0 0% 0 0% 0 0% symlink mknodremove rmdir rename 0 0% 0 0% 0 0% 0 0% 0 0% link readdir readdirplus fsstat fsinfo 0 0% 0 0% 23185 5% 0 0% 4 0% pathconf commit 0 0% 0 0% -- Olivier Mehani PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE F5F9 F012 A6E2 98C6 6655 Confidentiality cannot be guaranteed on emails sent or received unencrypted.
Bug#1025537: nfsd: Kernel Oops while serving NFS
On Tuesday, 6 December 2022 23:10:32 CET Olivier Mehani wrote: > >What's the version of NFS you're using? > > The clients are recent Linux hosts, and one Kodi 19.x. I don't specify a > version in either their fstab or the server's exports. Sorry, I meant the package version of the NFS server signature.asc Description: This is a digitally signed message part.
Bug#1025537: nfsd: Kernel Oops while serving NFS
On Tuesday, 6 December 2022 12:54:31 CET Olivier Mehani wrote: > NFS would still be accessible as before (: What's the version of NFS you're using? signature.asc Description: This is a digitally signed message part.
Bug#1025537: nfsd: Kernel Oops while serving NFS
Package: src:linux Version: 6.0.10-1 Severity: important File: nfsd Dear Maintainer, *** Reporter, please consider answering these questions, where appropriate *** * What led up to the situation? On testing/bookworm, since booting on a 6-versioned linux-image, I have noticed frequent hang ups of the nfs server, rendering it mostly unusable. This is accompanied with Kernel Oops in the dmesg. This sounds similar to previous bugs #1014793 and #1020548, both RESOLVED by later patches. * What exactly did you do (or not do) that was effective (or ineffective)? Upgrade from a 5.19 series kernel to a 6-series * What was the outcome of this action? The issue started happening. Nov 22 19:08:04 hostname kernel: [0.00] Linux version 5.19.0-2-amd64 (debian-kernel@lists.debian.org) (gcc-11 (Debian 11.3.0-6) 11.3.0, GNU ld (GNU Binutils for Debian) 2.38.90.20220713) #1 SMP PREEMPT_DYNAMIC Debian 5.19.11-1 (2022-09-24) 2022-11-24T11:09:43.398458+11:00 hostname kernel: [0.00] Linux version 6.0.0-4-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-9) 12.2.0, GNU ld (GNU Binutils for Debian) 2.39) #1 SMP PREEMPT_DYNAMIC Debian 6.0.8-1 (2022-11-11) 2022-11-24T22:09:32.716084+11:00 hostname kernel: [0.00] Linux version 6.0.0-4-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-9) 12.2.0, GNU ld (GNU Binutils for Debian) 2.39) #1 SMP PREEMPT_DYNAMIC Debian 6.0.8-1 (2022-11-11) 2022-11-24T22:14:29.941494+11:00 hostname kernel: [ 346.670278] Oops: [#4] PREEMPT SMP NOPTI 2022-11-24T22:14:30.124617+11:00 hostname kernel: [ 346.853344] Oops: [#6] PREEMPT SMP NOPTI 2022-11-24T22:14:30.216160+11:00 hostname kernel: [ 346.943972] Oops: [#7] PREEMPT SMP NOPTI 2022-11-24T22:14:30.306235+11:00 hostname kernel: [ 347.034549] Oops: [#8] PREEMPT SMP NOPTI 2022-11-25T22:23:26.062768+11:00 hostname kernel: [87286.216732] Oops: 0002 [#9] PREEMPT SMP NOPTI 2022-11-25T22:33:29.075088+11:00 hostname kernel: [0.00] Linux version 6.0.0-4-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-9) 12.2.0, GNU ld (GNU Binutils for Debian) 2.39) #1 SMP PREEMPT_DYNAMIC Debian 6.0.8-1 (2022-11-11) 2022-11-25T22:41:20.923037+11:00 hostname kernel: [ 520.000568] Oops: [#2] PREEMPT SMP NOPTI 2022-11-26T10:52:02.907042+11:00 hostname kernel: [0.00] Linux version 6.0.0-4-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-9) 12.2.0, GNU ld (GNU Binutils for Debian) 2.39) #1 SMP PREEMPT_DYNAMIC Debian 6.0.8-1 (2022-11-11) 2022-11-26T23:33:41.683961+11:00 hostname kernel: [0.00] Linux version 6.0.0-4-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-9) 12.2.0, GNU ld (GNU Binutils for Debian) 2.39) #1 SMP PREEMPT_DYNAMIC Debian 6.0.8-1 (2022-11-11) * What outcome did you expect instead? NFS would still be accessible as before (: *** End of the template - remove these template lines *** -- Package-specific info: ** Version: Linux version 6.0.0-5-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-9) 12.2.0, GNU ld (GNU Binutils for Debian) 2.39) #1 SMP PREEMPT_DYNAMIC Debian 6.0.10-1 (2022-11-26) ** Command line: BOOT_IMAGE=/boot/vmlinuz-6.0.0-5-amd64 root=UUID=0c917590-acc7-464c-8961-64f79e1d1c69 ro init=/lib/systemd/systemd init=/lib/systemd/systemd ** Tainted: D (128) * kernel died recently, i.e. there was an OOPS or BUG ** Kernel log: [ 433.757651] Hardware name: HP ProLiant MicroServer, BIOS O41 10/01/2013 [ 433.757702] RIP: 0010:release_pages+0xcd/0x500 [ 433.757761] Code: 84 c0 74 1a 48 8b 04 24 48 8d 54 24 30 49 89 46 08 48 89 44 24 30 4c 89 75 08 48 89 55 10 49 83 c7 08 4c 39 fb 74 75 49 8b 2f <48> 8b 45 08 a8 01 0f 85 58 01 00 00 0f 1f 44 00 00 4d 85 ed 74 0e [ 433.757831] RSP: 0018:aea000e83e30 EFLAGS: 00010202 [ 433.757884] RAX: 0007 RBX: 8e89a49fcb48 RCX: e5664199de48 [ 433.757935] RDX: aea000e83e60 RSI: aea000e83e60 RDI: e5664199de48 [ 433.757985] RBP: 000fc000 R08: e56641998e08 R09: 006a [ 433.758035] R10: 3890 R11: R12: [ 433.758085] R13: R14: e56641998e08 R15: 8e89a49fcb28 [ 433.758135] FS: () GS:8e8a97c8() knlGS: [ 433.758191] CS: 0010 DS: ES: CR0: 80050033 [ 433.758241] CR2: 7fb70339d000 CR3: 07e58000 CR4: 06e0 [ 433.758292] Call Trace: [ 433.758340] [ 433.758392] ? nfsd_shutdown_threads+0x90/0x90 [nfsd] [ 433.758541] __pagevec_release+0x1b/0x30 [ 433.758594] svc_xprt_release+0x1a3/0x1e0 [sunrpc] [ 433.758810] svc_send+0x59/0x160 [sunrpc] [ 433.759001] nfsd+0xd5/0x190 [nfsd] [ 433.759133] kthread+0xe9/0x110 [ 433.759183] ? kthread_complete_and_exit+0x20/0x20 [ 433.759235] ret_from_fork+0x22/0x30 [ 433.759292] [ 433.759335] Modules linked in: veth nf_conntrack_netlink xfrm_user
