[Kernel-packages] [Bug 2018960] Re: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0
Both -149 and -150 are exhibiting this behavior on a server I use as a VPN/NAT gateway. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed in Ubuntu. https://bugs.launchpad.net/bugs/2018960 Title: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 Status in linux-signed package in Ubuntu: Confirmed Status in linux-signed-kvm package in Ubuntu: Confirmed Bug description: After upgrading and rebooting this Ubuntu 20.04 LTS server (Ubuntu Focal), I noticed that it was suddenly getting a bunch of kernel log (dmesg) reports like: WARNING: CPU: 4 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 while investigating I determined that it is currently running the focal-proposed kernel (linux-image-5.4.0-149-generic), which it turns out was enabled for this server (clearly it seemed like a good idea at the time). I'm not expecting focal-proposed to be fixed as if it were a release package, but since I couldn't find any reports on Launchpad I figured I should let y'all know this focal-proposed package could do with some additional work before it's actually released :-) There have been at least 80 such reports in the last 5 hours since the server was rebooted, differing only by the CPU core and the process reported, although it seems the last one was a couple of hours ago, so I guess it's traffic dependent/timing dependent. ewen@naosr620:~$ uptime 16:27:32 up 5:19, 1 user, load average: 0.08, 0.14, 0.06 ewen@naosr620:~$ dmesg -t | grep WARNING | sed 's/CPU: [0-9]*/CPU: N/; s/PID: [0-9]*/PID: N/;' | uniq -c 88 WARNING: CPU: N PID: N at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 ewen@naosr620:~$ Ubuntu Release: ewen@naosr620:~$ lsb_release -rd Description: Ubuntu 20.04.6 LTS Release: 20.04 ewen@naosr620:~$ Kernel/package version affected: ewen@naosr620:~$ uname -a Linux naosr620 5.4.0-149-generic #166-Ubuntu SMP Tue Apr 18 16:51:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ewen@naosr620:~$ dpkg -l | grep linux-image | grep 149 ii linux-image-5.4.0-149-generic 5.4.0-149.166 amd64Signed kernel image generic ii linux-image-generic5.4.0.149.147 amd64Generic Linux kernel image ewen@naosr620:~$ apt-cache policy linux-image-5.4.0-149-generic linux-image-5.4.0-149-generic: Installed: 5.4.0-149.166 Candidate: 5.4.0-149.166 Version table: *** 5.4.0-149.166 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status ewen@naosr620:~$ apt-cache policy linux-image-generic linux-image-generic: Installed: 5.4.0.149.147 Candidate: 5.4.0.149.147 Version table: *** 5.4.0.149.147 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status 5.4.0.148.146 500 500 https://mirror.fsmg.org.nz/ubuntu focal-updates/main amd64 Packages 500 https://mirror.fsmg.org.nz/ubuntu focal-security/main amd64 Packages 5.4.0.26.32 500 500 https://mirror.fsmg.org.nz/ubuntu focal/main amd64 Packages ewen@naosr620:~$ ewen@naosr620:~$ apt-cache show linux-image-5.4.0-149-generic | grep Source: Source: linux-signed ewen@naosr620:~$ Full example dmesg, including stack trace (they all seem to be WARNINGs, and other than filling dmesg / system logs the system "appears to be running okay", so I'm not going to rush another reboot now -- near end of business day): ewen@naosr620:~$ date Tue 09 May 2023 16:34:56 NZST ewen@naosr620:~$ dmesg -T | tail -100 | grep -B 150 "end trace" | grep -A 999 "cut here" [Tue May 9 14:21:18 2023] [ cut here ] [Tue May 9 14:21:18 2023] WARNING: CPU: 10 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 [Tue May 9 14:21:18 2023] Modules linked in: mpt3sas raid_class scsi_transport_sas mptctl mptbase vhost_net vhost tap ip6t_REJECT nf_reject_ipv6 ip6table_mangle ip6table_nat ip6table_raw nf_log_ipv6 xt_recent ipt_REJECT nf_reject_ipv4 xt_hashlimit xt_addrtype xt_multiport xt_comment xt_conntrack xt_mark iptable_mangle xt_MASQUERADE iptable_nat xt_CT xt_tcpudp iptable_raw nfnetlink_log xt_NFLOG nf_log_ipv4 nf_log_common xt_LOG nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_irc ebtable_filter nf_nat_h323 ebtables nf_nat_ftp nf_nat_amanda ts_kmp ip6table_filter nf_conntrack_amanda nf_nat ip6_tables nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink nfnetlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bpfilter dell_rbu
[Kernel-packages] [Bug 1796895] Re: Kernel 4.15.0-36 network performance regression
Ok - for anyone that isn't used to installing kernels from proposed. It helps to install linux-image-generic and not linux-image-X-generic in order to pull in the modules package. Tested and verified - network speeds are back to normal in -37. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1796895 Title: Kernel 4.15.0-36 network performance regression Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Committed Bug description: Hardware: HP DL380 Gen 7 server with Gigabit interface. Ubuntu release: 18.04.1 Since upgrading from 4.15.0-33 to 4.15.0-36, I have noticed a dramatic slowdown in TCP over IPv6. Set up: two servers, Server A in US on Ubuntu 18.04.1, Server B in Europe. Test: Sending 400MB over TCP from Server B to Server A. With 4.15.0-33 on Server A, the transfer is completed in 24 seconds, with average speed of 139 mbps. With 4.15.0-36 on Server A, the same amount of data is transferred in 369 seconds, average speed less than 9 mbps. Looking at the result from tcpdump, I see that the there is a big difference in the TCP Window size. With 3.15-0.33 the TCP window size is around 3. But with 3.15-0.36, the window size is only 734. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1796895/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1796895] Re: Kernel 4.15.0-36 network performance regression
For whatever reason, after installing linux-image-4.15.0-37-generic the system boots up fine but doesn't appear to initialize the motherboard's built in realtek network adapter so there is no network connectivity to test. Is there a missing dependency in the package that needs to be installed to ensure that network connectivity works? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1796895 Title: Kernel 4.15.0-36 network performance regression Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Committed Bug description: Hardware: HP DL380 Gen 7 server with Gigabit interface. Ubuntu release: 18.04.1 Since upgrading from 4.15.0-33 to 4.15.0-36, I have noticed a dramatic slowdown in TCP over IPv6. Set up: two servers, Server A in US on Ubuntu 18.04.1, Server B in Europe. Test: Sending 400MB over TCP from Server B to Server A. With 4.15.0-33 on Server A, the transfer is completed in 24 seconds, with average speed of 139 mbps. With 4.15.0-36 on Server A, the same amount of data is transferred in 369 seconds, average speed less than 9 mbps. Looking at the result from tcpdump, I see that the there is a big difference in the TCP Window size. With 3.15-0.33 the TCP window size is around 3. But with 3.15-0.36, the window size is only 734. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1796895/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1796895] Re: Kernel 4.15.0-36 network performance regression
I installed the proposed kernel on a remote host and rebooted but of course the host didn't come back online for who only knows what reason. I'll post an update in a few hours once I have a chance to get my hands on it. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1796895 Title: Kernel 4.15.0-36 network performance regression Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Committed Bug description: Hardware: HP DL380 Gen 7 server with Gigabit interface. Ubuntu release: 18.04.1 Since upgrading from 4.15.0-33 to 4.15.0-36, I have noticed a dramatic slowdown in TCP over IPv6. Set up: two servers, Server A in US on Ubuntu 18.04.1, Server B in Europe. Test: Sending 400MB over TCP from Server B to Server A. With 4.15.0-33 on Server A, the transfer is completed in 24 seconds, with average speed of 139 mbps. With 4.15.0-36 on Server A, the same amount of data is transferred in 369 seconds, average speed less than 9 mbps. Looking at the result from tcpdump, I see that the there is a big difference in the TCP Window size. With 3.15-0.33 the TCP window size is around 3. But with 3.15-0.36, the window size is only 734. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1796895/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1796895] Re: Kernel 4.15.0-36 network performance regression
I'm seeing similar behavior on multiple physical hosts running 4.15.0-36-generic. The TCP window size only appears to grow very slightly (from initial of 224 to around 1400) during the entire length of the transfer. Downgrading to 4.15.0-34-generic fixes the issue for me - TCP window size continues to grow until it is around 24k. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1796895 Title: Kernel 4.15.0-36 network performance regression Status in linux package in Ubuntu: Confirmed Status in linux source package in Bionic: Confirmed Bug description: Hardware: HP DL380 Gen 7 server with Gigabit interface. Ubuntu release: 18.04.1 Since upgrading from 4.15.0-33 to 4.15.0-36, I have noticed a dramatic slowdown in TCP over IPv6. Set up: two servers, Server A in US on Ubuntu 18.04.1, Server B in Europe. Test: Sending 400MB over TCP from Server B to Server A. With 4.15.0-33 on Server A, the transfer is completed in 24 seconds, with average speed of 139 mbps. With 4.15.0-36 on Server A, the same amount of data is transferred in 369 seconds, average speed less than 9 mbps. Looking at the result from tcpdump, I see that the there is a big difference in the TCP Window size. With 3.15-0.33 the TCP window size is around 3. But with 3.15-0.36, the window size is only 734. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1796895/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1686837] Re: ubuntu fails to boot with btrfs root (unable to mount root)
I have indeed tried unplugging/swapping connected USB peripherals and that never appeared to help. The only thing that worked reliably for me was pinning the old .67 kernel. Something changed after that and the one thing that did stand out is a huge increase in the initrd image size of later kernels. Anyway, out of all the systems I run 16.04 on only two suffer from this issue and they are completely different beasts (one is a home brew server and the other is a dell optiplex workstation - they are also the only ones that run root on btrfs). I've updated the home brew system to 18.04 and it appears to no longer suffer from the issue but then again I don't reboot it very often either so who really knows. The Dell still requires me to hit 'e' and hang out in the grub editor for a period of time before continuing to boot reliably. It is very strange because manually adding sleep statements to the generated grub.cfg does not work. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1686837 Title: ubuntu fails to boot with btrfs root (unable to mount root) Status in linux package in Ubuntu: Expired Bug description: Hello, Failing to mount / as btrfs has been haunting btrfs users on ubuntu for some time. Coming and going with kernel updates. After I upgraded to 17.04, it seems that the problem is back. It seems that a non-clean umount reboot always leads to a non-bootable system. If I boot into rescue (selecting in grub, not a livecd), kernel can find the root partition. I can also boot normally if root=... kernel parameter is set with the device name (/dev/sda6) instead of UUID=. So, there might be a bug in kernel that does not allow linux to identify the boot partition using UUID in some circumstances (probably after a dirt reboot). Maybe other distros do some magic at initrd that cleans the btrfs problem because some, like opensuse, do uses btrfs as default root fs. Anyway, even ubuntu own rescue can boot normally. This is a tricky bug to debug as I get no logs written and no emergency shell. Just an ugly kernel error and a backstack. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1686837/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1686837] Re: ubuntu fails to boot with btrfs root (unable to mount root)
@luizluca Were you ever able to discover a consistent work-around? I have *exactly* the same issue with 16.04 since something like the .67 kernel and I've exhausted everything I can think of to try and figure out why it is behaving this way. The only thing that works (as you yourself stumbled upon) is hitting 'e' in grub and waiting a bit before allowing it to continue. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1686837 Title: ubuntu fails to boot with btrfs root (unable to mount root) Status in linux package in Ubuntu: Expired Bug description: Hello, Failing to mount / as btrfs has been haunting btrfs users on ubuntu for some time. Coming and going with kernel updates. After I upgraded to 17.04, it seems that the problem is back. It seems that a non-clean umount reboot always leads to a non-bootable system. If I boot into rescue (selecting in grub, not a livecd), kernel can find the root partition. I can also boot normally if root=... kernel parameter is set with the device name (/dev/sda6) instead of UUID=. So, there might be a bug in kernel that does not allow linux to identify the boot partition using UUID in some circumstances (probably after a dirt reboot). Maybe other distros do some magic at initrd that cleans the btrfs problem because some, like opensuse, do uses btrfs as default root fs. Anyway, even ubuntu own rescue can boot normally. This is a tricky bug to debug as I get no logs written and no emergency shell. Just an ugly kernel error and a backstack. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1686837/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
@nate Thank you! You just saved me a lot of hassle as I was about to unpin the 4.4.0-57 kernel and update a bunch of machines on the assumption the fix was in that version. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 Status in linux package in Ubuntu: Fix Released Status in linux source package in Xenial: Fix Committed Bug description: I recently replaced some Xenial servers, and started experiencing "Out of memory" problems with the default kernel. We bake Amazon AMIs based on an official Ubuntu-provided image (ami- e6b58e85, in ap-southeast-2, from https://cloud- images.ubuntu.com/locator/ec2/). Previous versions of our AMI included "4.4.0-57-generic", but the latest version picked up "4.4.0-59-generic" as part of a "dist-upgrade". Instances booted using the new AMI have been using more memory, and experiencing OOM issues - sometimes during boot, and sometimes a while afterwards. An example from the system log is: [ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final' at Wed, 11 Jan 2017 22:07:53 +. Up 29.28 seconds. [ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan 2017 22:09:35 +. Datasource DataSourceEc2. Up 130.09 seconds [29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or sacrifice child [29871.140816] Killed process 2920 (ruby) total-vm:675048kB, anon-rss:51184kB, file-rss:2164kB [29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or sacrifice child [29871.453282] Killed process 3258 (splunkd) total-vm:66272kB, anon-rss:6676kB, file-rss:0kB [29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or sacrifice child [29871.681872] Killed process 2647 (fluentd) total-vm:117944kB, anon-rss:23956kB, file-rss:1356kB I have a hunch that this may be related to the fix for https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400, introduced in linux (4.4.0-58.79). ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-59-generic 4.4.0-59.80 ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35 Uname: Linux 4.4.0-59-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 12 06:29 seq crw-rw 1 root audio 116, 33 Jan 12 06:29 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 12 06:38:45 2017 Ec2AMI: ami-0f93966c Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-southeast-2a Ec2InstanceType: t2.nano Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 cirrusdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0 RelatedPackageVersions: linux-restricted-modules-4.4.0-59-generic N/A linux-backports-modules-4.4.0-59-generic N/A linux-firmware1.157.6 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/09/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/09/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
