[Kernel-packages] [Bug 1531768] Re: arm64 kernel and multiple CPUs is unusably slow
lxd-armhf1 (8 CPUs) is again in a state where "lxc list" and even "top" hang forever. lxd-armhf2 was unfortunately shutdown in the previous days, so I just booted it again. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: arm64 kernel and multiple CPUs is unusably slow Status in linux package in Ubuntu: Confirmed Bug description: I created an 8 CPU arm64 instance on Canonical's Scalingstack (which I want to use for armhf autopkgtesting in LXD). I started with wily as that has lxd available (it's not yet available in trusty nor the PPA for arm64). However, pretty much any LXD task that I do (I haven't tried much else) on this machine takes unbearably long. A simple "lxc profile set default raw.lxc lxc.seccomp=" or "lxc list" takes several minutes. I see tons of [ 1020.971955] rcu_sched kthread starved for 6000 jiffies! g1095 c1094 f0x0 [ 1121.166926] INFO: task fsnotify_mark:69 blocked for more than 120 seconds. in dmesg (the attached apport info has the complete dmesg). ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-22-generic 4.2.0-22.27 ProcVersionSignature: User Name 4.2.0-22.27-generic 4.2.6 Uname: Linux 4.2.0-22-generic aarch64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 7 09:18 seq crw-rw 1 root audio 116, 33 Jan 7 09:18 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: arm64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jan 7 09:24:01 2016 IwConfig: eth0 no wireless extensions. lono wireless extensions. lxcbr0no wireless extensions. Lspci: 00:00.0 Host bridge [0600]: Red Hat, Inc. Device [1b36:0008] Subsystem: Red Hat, Inc Device [1af4:1100] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=LABEL=cloudimg-rootfs earlyprintk RelatedPackageVersions: linux-restricted-modules-4.2.0-22-generic N/A linux-backports-modules-4.2.0-22-generic N/A linux-firmware1.149.3 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev' UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1531768/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1531768] Re: arm64 kernel and multiple CPUs is unusably slow
I split out the xenial bridge regression into bug 1534545, so that this can keep focussing on the "processes become slow and hang after a while" main aspect. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: arm64 kernel and multiple CPUs is unusably slow Status in linux package in Ubuntu: Confirmed Bug description: I created an 8 CPU arm64 instance on Canonical's Scalingstack (which I want to use for armhf autopkgtesting in LXD). I started with wily as that has lxd available (it's not yet available in trusty nor the PPA for arm64). However, pretty much any LXD task that I do (I haven't tried much else) on this machine takes unbearably long. A simple "lxc profile set default raw.lxc lxc.seccomp=" or "lxc list" takes several minutes. I see tons of [ 1020.971955] rcu_sched kthread starved for 6000 jiffies! g1095 c1094 f0x0 [ 1121.166926] INFO: task fsnotify_mark:69 blocked for more than 120 seconds. in dmesg (the attached apport info has the complete dmesg). ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-22-generic 4.2.0-22.27 ProcVersionSignature: User Name 4.2.0-22.27-generic 4.2.6 Uname: Linux 4.2.0-22-generic aarch64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 7 09:18 seq crw-rw 1 root audio 116, 33 Jan 7 09:18 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: arm64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jan 7 09:24:01 2016 IwConfig: eth0 no wireless extensions. lono wireless extensions. lxcbr0no wireless extensions. Lspci: 00:00.0 Host bridge [0600]: Red Hat, Inc. Device [1b36:0008] Subsystem: Red Hat, Inc Device [1af4:1100] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=LABEL=cloudimg-rootfs earlyprintk RelatedPackageVersions: linux-restricted-modules-4.2.0-22-generic N/A linux-backports-modules-4.2.0-22-generic N/A linux-firmware1.149.3 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev' UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1531768/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1531768] Re: arm64 kernel and multiple CPUs is unusably slow
The lxc hangs component looks to be an lxd related issue. Specificially the go libraries in use consume a large ammount of entropy and hang waiting for it to become available. Installing haveged seems to resolve these hangs. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: arm64 kernel and multiple CPUs is unusably slow Status in linux package in Ubuntu: Confirmed Bug description: I created an 8 CPU arm64 instance on Canonical's Scalingstack (which I want to use for armhf autopkgtesting in LXD). I started with wily as that has lxd available (it's not yet available in trusty nor the PPA for arm64). However, pretty much any LXD task that I do (I haven't tried much else) on this machine takes unbearably long. A simple "lxc profile set default raw.lxc lxc.seccomp=" or "lxc list" takes several minutes. I see tons of [ 1020.971955] rcu_sched kthread starved for 6000 jiffies! g1095 c1094 f0x0 [ 1121.166926] INFO: task fsnotify_mark:69 blocked for more than 120 seconds. in dmesg (the attached apport info has the complete dmesg). ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-22-generic 4.2.0-22.27 ProcVersionSignature: User Name 4.2.0-22.27-generic 4.2.6 Uname: Linux 4.2.0-22-generic aarch64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 7 09:18 seq crw-rw 1 root audio 116, 33 Jan 7 09:18 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: arm64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jan 7 09:24:01 2016 IwConfig: eth0 no wireless extensions. lono wireless extensions. lxcbr0no wireless extensions. Lspci: 00:00.0 Host bridge [0600]: Red Hat, Inc. Device [1b36:0008] Subsystem: Red Hat, Inc Device [1af4:1100] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=LABEL=cloudimg-rootfs earlyprintk RelatedPackageVersions: linux-restricted-modules-4.2.0-22-generic N/A linux-backports-modules-4.2.0-22-generic N/A linux-firmware1.149.3 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev' UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1531768/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1531768] Re: arm64 kernel and multiple CPUs is unusably slow
The rcu messages though annoying do seem to be benign as they do not increase in time. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: arm64 kernel and multiple CPUs is unusably slow Status in linux package in Ubuntu: Confirmed Bug description: I created an 8 CPU arm64 instance on Canonical's Scalingstack (which I want to use for armhf autopkgtesting in LXD). I started with wily as that has lxd available (it's not yet available in trusty nor the PPA for arm64). However, pretty much any LXD task that I do (I haven't tried much else) on this machine takes unbearably long. A simple "lxc profile set default raw.lxc lxc.seccomp=" or "lxc list" takes several minutes. I see tons of [ 1020.971955] rcu_sched kthread starved for 6000 jiffies! g1095 c1094 f0x0 [ 1121.166926] INFO: task fsnotify_mark:69 blocked for more than 120 seconds. in dmesg (the attached apport info has the complete dmesg). ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-22-generic 4.2.0-22.27 ProcVersionSignature: User Name 4.2.0-22.27-generic 4.2.6 Uname: Linux 4.2.0-22-generic aarch64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 7 09:18 seq crw-rw 1 root audio 116, 33 Jan 7 09:18 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: arm64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jan 7 09:24:01 2016 IwConfig: eth0 no wireless extensions. lono wireless extensions. lxcbr0no wireless extensions. Lspci: 00:00.0 Host bridge [0600]: Red Hat, Inc. Device [1b36:0008] Subsystem: Red Hat, Inc Device [1af4:1100] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=LABEL=cloudimg-rootfs earlyprintk RelatedPackageVersions: linux-restricted-modules-4.2.0-22-generic N/A linux-backports-modules-4.2.0-22-generic N/A linux-firmware1.149.3 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev' UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1531768/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1531768] Re: arm64 kernel and multiple CPUs is unusably slow
Another data point: I tried to install 3.19 (the kernel that we have on the buildds) on the xlarge instance, and lxc list now hangs there as well. I haven't yet seem lxc list hang on a large (4 CPUs) instance, but the whole thing (running tests in containers) is still very slow. TBC on Monday.. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: arm64 kernel and multiple CPUs is unusably slow Status in linux package in Ubuntu: Confirmed Bug description: I created an 8 CPU arm64 instance on Canonical's Scalingstack (which I want to use for armhf autopkgtesting in LXD). I started with wily as that has lxd available (it's not yet available in trusty nor the PPA for arm64). However, pretty much any LXD task that I do (I haven't tried much else) on this machine takes unbearably long. A simple "lxc profile set default raw.lxc lxc.seccomp=" or "lxc list" takes several minutes. I see tons of [ 1020.971955] rcu_sched kthread starved for 6000 jiffies! g1095 c1094 f0x0 [ 1121.166926] INFO: task fsnotify_mark:69 blocked for more than 120 seconds. in dmesg (the attached apport info has the complete dmesg). ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-22-generic 4.2.0-22.27 ProcVersionSignature: User Name 4.2.0-22.27-generic 4.2.6 Uname: Linux 4.2.0-22-generic aarch64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 7 09:18 seq crw-rw 1 root audio 116, 33 Jan 7 09:18 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: arm64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jan 7 09:24:01 2016 IwConfig: eth0 no wireless extensions. lono wireless extensions. lxcbr0no wireless extensions. Lspci: 00:00.0 Host bridge [0600]: Red Hat, Inc. Device [1b36:0008] Subsystem: Red Hat, Inc Device [1af4:1100] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=LABEL=cloudimg-rootfs earlyprintk RelatedPackageVersions: linux-restricted-modules-4.2.0-22-generic N/A linux-backports-modules-4.2.0-22-generic N/A linux-firmware1.149.3 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev' UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1531768/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1531768] Re: arm64 kernel and multiple CPUs is unusably slow
I did install haveged which indeed seems to help quite a bit. But now after having used an xlarge (8 CPU) instance for a while, I again get hanging processes, like ubuntu2317 0.0 0.0 0 0 pts/0D+ 16:14 0:00 [tail] I used that tail on /var/log/lxd/lxd.log to see what's going on. lxd itself stopped responding much earlier (but not in kernel "D" state, it's sleeping). Sorry, this is still very unspecific.. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: arm64 kernel and multiple CPUs is unusably slow Status in linux package in Ubuntu: Confirmed Bug description: I created an 8 CPU arm64 instance on Canonical's Scalingstack (which I want to use for armhf autopkgtesting in LXD). I started with wily as that has lxd available (it's not yet available in trusty nor the PPA for arm64). However, pretty much any LXD task that I do (I haven't tried much else) on this machine takes unbearably long. A simple "lxc profile set default raw.lxc lxc.seccomp=" or "lxc list" takes several minutes. I see tons of [ 1020.971955] rcu_sched kthread starved for 6000 jiffies! g1095 c1094 f0x0 [ 1121.166926] INFO: task fsnotify_mark:69 blocked for more than 120 seconds. in dmesg (the attached apport info has the complete dmesg). ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-22-generic 4.2.0-22.27 ProcVersionSignature: User Name 4.2.0-22.27-generic 4.2.6 Uname: Linux 4.2.0-22-generic aarch64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 7 09:18 seq crw-rw 1 root audio 116, 33 Jan 7 09:18 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: arm64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jan 7 09:24:01 2016 IwConfig: eth0 no wireless extensions. lono wireless extensions. lxcbr0no wireless extensions. Lspci: 00:00.0 Host bridge [0600]: Red Hat, Inc. Device [1b36:0008] Subsystem: Red Hat, Inc Device [1af4:1100] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=LABEL=cloudimg-rootfs earlyprintk RelatedPackageVersions: linux-restricted-modules-4.2.0-22-generic N/A linux-backports-modules-4.2.0-22-generic N/A linux-firmware1.149.3 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev' UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1531768/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1531768] Re: arm64 kernel and multiple CPUs is unusably slow
FTR, the "networking broken in containers" was an MTU mismatch, worked around now. Thanks to Andy for figuring this out! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: arm64 kernel and multiple CPUs is unusably slow Status in linux package in Ubuntu: Confirmed Bug description: I created an 8 CPU arm64 instance on Canonical's Scalingstack (which I want to use for armhf autopkgtesting in LXD). I started with wily as that has lxd available (it's not yet available in trusty nor the PPA for arm64). However, pretty much any LXD task that I do (I haven't tried much else) on this machine takes unbearably long. A simple "lxc profile set default raw.lxc lxc.seccomp=" or "lxc list" takes several minutes. I see tons of [ 1020.971955] rcu_sched kthread starved for 6000 jiffies! g1095 c1094 f0x0 [ 1121.166926] INFO: task fsnotify_mark:69 blocked for more than 120 seconds. in dmesg (the attached apport info has the complete dmesg). ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-22-generic 4.2.0-22.27 ProcVersionSignature: User Name 4.2.0-22.27-generic 4.2.6 Uname: Linux 4.2.0-22-generic aarch64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 7 09:18 seq crw-rw 1 root audio 116, 33 Jan 7 09:18 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: arm64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jan 7 09:24:01 2016 IwConfig: eth0 no wireless extensions. lono wireless extensions. lxcbr0no wireless extensions. Lspci: 00:00.0 Host bridge [0600]: Red Hat, Inc. Device [1b36:0008] Subsystem: Red Hat, Inc Device [1af4:1100] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=LABEL=cloudimg-rootfs earlyprintk RelatedPackageVersions: linux-restricted-modules-4.2.0-22-generic N/A linux-backports-modules-4.2.0-22-generic N/A linux-firmware1.149.3 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev' UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1531768/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1531768] Re: arm64 kernel and multiple CPUs is unusably slow
I take that back. It does survive for much longer, but after some 15 minutes of running I again run into tons of [ 2424.611668] INFO: task systemd-udevd:1320 blocked for more than 120 seconds. [ 2424.613514] Tainted: GW 4.2.0-22-generic #27-Ubuntu [ 2424.615183] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2424.617166] systemd-udevd D ffc86ee4 0 1320 1 0x000c [ 2424.617176] Call trace: [ 2424.617842] [] __switch_to+0x94/0xa8 [ 2424.617851] [] __schedule+0x2b0/0x7b8 [ 2424.617854] [] schedule+0x3c/0x98 [ 2424.617859] [] schedule_timeout+0x1ec/0x280 [ 2424.617862] [] wait_for_common+0xcc/0x1a0 [ 2424.617866] [] wait_for_completion+0x28/0x38 [ 2424.617870] [] __synchronize_srcu+0x9c/0x180 [ 2424.617873] [] synchronize_srcu+0x38/0x48 [ 2424.617877] [] fsnotify_destroy_group+0x2c/0x60 [ 2424.617880] [] inotify_release+0x34/0x78 [ 2424.617885] [] __fput+0xa4/0x248 [ 2424.617887] [] fput+0x20/0x30 [ 2424.617892] [] task_work_run+0xbc/0xf8 [ 2424.617896] [] do_exit+0x2f0/0xa48 [ 2424.617898] [] do_group_exit+0x44/0xe8 [ 2424.617902] [] get_signal+0x3d8/0x578 [ 2424.617906] [] do_signal+0x90/0x530 [ 2424.617909] [] do_notify_resume+0x70/0x78 for all kinds of processes. ** Summary changed: - arm64 kernel and >= 8 CPUS (>= host CPU count?) is unusably slow with lxd operations + arm64 kernel and multiple CPUs is unusably slow -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: arm64 kernel and multiple CPUs is unusably slow Status in linux package in Ubuntu: Confirmed Bug description: I created an 8 CPU arm64 instance on Canonical's Scalingstack (which I want to use for armhf autopkgtesting in LXD). I started with wily as that has lxd available (it's not yet available in trusty nor the PPA for arm64). However, pretty much any LXD task that I do (I haven't tried much else) on this machine takes unbearably long. A simple "lxc profile set default raw.lxc lxc.seccomp=" or "lxc list" takes several minutes. I see tons of [ 1020.971955] rcu_sched kthread starved for 6000 jiffies! g1095 c1094 f0x0 [ 1121.166926] INFO: task fsnotify_mark:69 blocked for more than 120 seconds. in dmesg (the attached apport info has the complete dmesg). ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-22-generic 4.2.0-22.27 ProcVersionSignature: User Name 4.2.0-22.27-generic 4.2.6 Uname: Linux 4.2.0-22-generic aarch64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 7 09:18 seq crw-rw 1 root audio 116, 33 Jan 7 09:18 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: arm64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jan 7 09:24:01 2016 IwConfig: eth0 no wireless extensions. lono wireless extensions. lxcbr0no wireless extensions. Lspci: 00:00.0 Host bridge [0600]: Red Hat, Inc. Device [1b36:0008] Subsystem: Red Hat, Inc Device [1af4:1100] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=LABEL=cloudimg-rootfs earlyprintk RelatedPackageVersions: linux-restricted-modules-4.2.0-22-generic N/A linux-backports-modules-4.2.0-22-generic N/A linux-firmware1.149.3 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev' UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1531768/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1531768] Re: arm64 kernel and multiple CPUs is unusably slow
** Tags added: kernel-da-key -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: arm64 kernel and multiple CPUs is unusably slow Status in linux package in Ubuntu: Confirmed Bug description: I created an 8 CPU arm64 instance on Canonical's Scalingstack (which I want to use for armhf autopkgtesting in LXD). I started with wily as that has lxd available (it's not yet available in trusty nor the PPA for arm64). However, pretty much any LXD task that I do (I haven't tried much else) on this machine takes unbearably long. A simple "lxc profile set default raw.lxc lxc.seccomp=" or "lxc list" takes several minutes. I see tons of [ 1020.971955] rcu_sched kthread starved for 6000 jiffies! g1095 c1094 f0x0 [ 1121.166926] INFO: task fsnotify_mark:69 blocked for more than 120 seconds. in dmesg (the attached apport info has the complete dmesg). ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-22-generic 4.2.0-22.27 ProcVersionSignature: User Name 4.2.0-22.27-generic 4.2.6 Uname: Linux 4.2.0-22-generic aarch64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 7 09:18 seq crw-rw 1 root audio 116, 33 Jan 7 09:18 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: arm64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jan 7 09:24:01 2016 IwConfig: eth0 no wireless extensions. lono wireless extensions. lxcbr0no wireless extensions. Lspci: 00:00.0 Host bridge [0600]: Red Hat, Inc. Device [1b36:0008] Subsystem: Red Hat, Inc Device [1af4:1100] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=LABEL=cloudimg-rootfs earlyprintk RelatedPackageVersions: linux-restricted-modules-4.2.0-22-generic N/A linux-backports-modules-4.2.0-22-generic N/A linux-firmware1.149.3 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev' UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1531768/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1531768] Re: arm64 kernel and multiple CPUs is unusably slow
> Can you collect apport information from the host system as well? Sorry, I can't. I can create Scalingstack instances, but I have no access to the host systems. The IS team certainly can, though. > Do you get the same effects with a single vCPU? So far that test system is holding up and I haven't seen processes getting locked up. I do see that networking within containers is totally busted (transmitting 20 bytes took some10 minutes), but with the multi-CPU instance I didn't even get that far. This also happens with 3.19 (cannot test 4.3 due to its regression of creating bridges). Either way, this seems to be a separate bug. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1531768 Title: arm64 kernel and multiple CPUs is unusably slow Status in linux package in Ubuntu: Confirmed Bug description: I created an 8 CPU arm64 instance on Canonical's Scalingstack (which I want to use for armhf autopkgtesting in LXD). I started with wily as that has lxd available (it's not yet available in trusty nor the PPA for arm64). However, pretty much any LXD task that I do (I haven't tried much else) on this machine takes unbearably long. A simple "lxc profile set default raw.lxc lxc.seccomp=" or "lxc list" takes several minutes. I see tons of [ 1020.971955] rcu_sched kthread starved for 6000 jiffies! g1095 c1094 f0x0 [ 1121.166926] INFO: task fsnotify_mark:69 blocked for more than 120 seconds. in dmesg (the attached apport info has the complete dmesg). ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-22-generic 4.2.0-22.27 ProcVersionSignature: User Name 4.2.0-22.27-generic 4.2.6 Uname: Linux 4.2.0-22-generic aarch64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 7 09:18 seq crw-rw 1 root audio 116, 33 Jan 7 09:18 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: arm64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jan 7 09:24:01 2016 IwConfig: eth0 no wireless extensions. lono wireless extensions. lxcbr0no wireless extensions. Lspci: 00:00.0 Host bridge [0600]: Red Hat, Inc. Device [1b36:0008] Subsystem: Red Hat, Inc Device [1af4:1100] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-22-generic root=LABEL=cloudimg-rootfs earlyprintk RelatedPackageVersions: linux-restricted-modules-4.2.0-22-generic N/A linux-backports-modules-4.2.0-22-generic N/A linux-firmware1.149.3 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev' UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1531768/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
