[Kernel-packages] [Bug 1665113] Re: [Ubuntu 17.04] Kernel panics when large number of hugepages is passed as an boot argument to kernel.
** Tags added: cscc -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1665113 Title: [Ubuntu 17.04] Kernel panics when large number of hugepages is passed as an boot argument to kernel. Status in linux package in Ubuntu: Fix Released Status in linux source package in Zesty: Fix Released Bug description: Issue: --- Kernel unable to handle paging request and panic occurs when more number of hugepages is passed as a boot argument to the kernel . Environment: -- Power NV : Habanaro Bare metal OS : Ubuntu 17.04 Kernel Version : 4.9.0-11-generic Steps To reproduce: --- 1 - When the ubuntu Kernel boots try to add the boot argument 'hugepages = 1200'. The Kernel Panics and displays call traces like as below. [5.030274] Unable to handle kernel paging request for data at address 0x [5.030323] Faulting instruction address: 0xc0302848 [5.030366] Oops: Kernel access of bad area, sig: 11 [#1] [5.030399] SMP NR_CPUS=2048 [5.030416] NUMA [5.039443] PowerNV [5.039461] Modules linked in: [5.050091] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 4.9.0-11-generic #12-Ubuntu [5.053266] Workqueue: events pcpu_balance_workfn [5.080647] task: c03c8fe9b800 task.stack: c03ffb118000 [5.090876] NIP: c0302848 LR: c02709d4 CTR: c016cef0 [5.094175] REGS: c03ffb11b410 TRAP: 0300 Not tainted (4.9.0-11-generic) [5.103040] MSR: 92009033 [ 5.114466] CR: 22424222 XER: [5.124932] CFAR: c0008a60 DAR: DSISR: 4000 SOFTE: 1 GPR00: c02709d4 c03ffb11b690 c141a400 c03fff50e300 GPR04: 024001c2 c03ffb11b780 00219df5 GPR08: 003ffb09 c1454fd8 GPR12: 4400 c7b6 024001c2 024001c2 GPR16: 024001c2 0002 GPR20: 000c 024200c0 GPR24: c16eef48 c03fff50fd00 024001c2 GPR28: c03fff50fd00 c03fff50e300 c03ffb11b820 NIP [c0302848] mem_cgroup_soft_limit_reclaim+0xf8/0x4f0 [5.213613] LR [c02709d4] do_try_to_free_pages+0x1b4/0x450 [5.230521] Call Trace: [5.230643] [c03ffb11b760] [c02709d4] do_try_to_free_pages+0x1b4/0x450 [5.254184] [c03ffb11b800] [c0270d68] try_to_free_pages+0xf8/0x270 [5.281896] [c03ffb11b890] [c0259b88] __alloc_pages_nodemask+0x7a8/0xff0 [5.321407] [c03ffb11bab0] [c0282cd0] pcpu_populate_chunk+0x110/0x520 [5.336262] [c03ffb11bb50] [c02841b8] pcpu_balance_workfn+0x758/0x960 [5.351526] [c03ffb11bc50] [c00ecdd0] process_one_work+0x2b0/0x5a0 [5.362561] [c03ffb11bce0] [c00ed168] worker_thread+0xa8/0x660 [5.374007] [c03ffb11bd80] [c00f5320] kthread+0x110/0x130 [5.385160] [c03ffb11be30] [c000c0e8] ret_from_kernel_thread+0x5c/0x74 [5.389456] Instruction dump: [5.410036] eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 4e800020 3d230001 e9499a42 3d220004 [5.423598] 3929abd8 794a1f24 7d295214 eac90100 2fa9 419eff74 3b20 [5.436503] ---[ end trace 23b650e96be5c549 ]--- [5.439700] This is purely a negative scenario where the system does not have enough memory as the hugepages is given a very large argument. Free output in a system: free -h totalusedfree shared buff/cache available Mem: 251G2.1G248G5.2M502M 248G Swap: 2.0G159M1.8G The same scenario when tried after the linux is up like as, echo 1200 > /proc/sys/vm/nr_hugepages HugePages_Total: 15069 HugePages_Free:15069 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 16384 kB root@ltc-haba2:~# free -h totalusedfree shared buff/cache available Mem: 251G237G 13G5.6M311M 13G Swap: 2.0G159M1.8G In this case the kernel is able to allocate around 237 Gb for hugetlb. But while the system is booting it gives us panic so please let know if this scenario is expected to be handled. I identified the root cause of the panic. When the system is running with low memory during mem cgroup initialisation, because most of the page have been grabbed to be huge pages, we hit a chicken and egg issue because when trying to allocate memory for the node's cgroup
[Kernel-packages] [Bug 1665113] Re: [Ubuntu 17.04] Kernel panics when large number of hugepages is passed as an boot argument to kernel.
This bug was fixed in the package linux - 4.10.0-13.15 --- linux (4.10.0-13.15) zesty; urgency=low [ Tim Gardner ] * Release Tracking Bug - LP: #1671614 * ehci-platform needed in usb-modules udeb (LP: #1671589) - d-i: add ehci-platform to usb-modules * irqchip/gic-v3-its: Enable cacheable attribute Read-allocate hints (LP: #1671598) - irqchip/gic-v3-its: Enable cacheable attribute Read-allocate hints * iommu: Fix static checker warning in iommu_insert_device_resv_regions (LP: #1671599) - iommu: Fix static checker warning in iommu_insert_device_resv_regions * QDF2400: Fix panic introduced by erratum 1003 (LP: #1671602) - arm64: Avoid clobbering mm in erratum workaround on QDF2400 * QDF2400 PCI ports require ACS quirk (LP: #1671601) - PCI: Add ACS quirk for Qualcomm QDF2400 and QDF2432 * tty: pl011: Work around QDF2400 E44 stuck BUSY bit (LP: #1671600) - tty: pl011: Work around QDF2400 E44 stuck BUSY bit * CVE-2017-2636 - tty: n_hdlc: get rid of racy n_hdlc.tbuf * Sync virtualbox to 5.1.16-dfsg-1 in zesty (LP: #1671470) - ubuntu: vbox -- Update to 5.1.16-dfsg-1 -- Tim Gardner Thu, 09 Mar 2017 06:16:24 -0700 ** Changed in: linux (Ubuntu Zesty) Status: Fix Committed => Fix Released ** CVE added: http://www.cve.mitre.org/cgi- bin/cvename.cgi?name=2017-2636 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1665113 Title: [Ubuntu 17.04] Kernel panics when large number of hugepages is passed as an boot argument to kernel. Status in linux package in Ubuntu: Fix Released Status in linux source package in Zesty: Fix Released Bug description: Issue: --- Kernel unable to handle paging request and panic occurs when more number of hugepages is passed as a boot argument to the kernel . Environment: -- Power NV : Habanaro Bare metal OS : Ubuntu 17.04 Kernel Version : 4.9.0-11-generic Steps To reproduce: --- 1 - When the ubuntu Kernel boots try to add the boot argument 'hugepages = 1200'. The Kernel Panics and displays call traces like as below. [5.030274] Unable to handle kernel paging request for data at address 0x [5.030323] Faulting instruction address: 0xc0302848 [5.030366] Oops: Kernel access of bad area, sig: 11 [#1] [5.030399] SMP NR_CPUS=2048 [5.030416] NUMA [5.039443] PowerNV [5.039461] Modules linked in: [5.050091] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 4.9.0-11-generic #12-Ubuntu [5.053266] Workqueue: events pcpu_balance_workfn [5.080647] task: c03c8fe9b800 task.stack: c03ffb118000 [5.090876] NIP: c0302848 LR: c02709d4 CTR: c016cef0 [5.094175] REGS: c03ffb11b410 TRAP: 0300 Not tainted (4.9.0-11-generic) [5.103040] MSR: 92009033 [ 5.114466] CR: 22424222 XER: [5.124932] CFAR: c0008a60 DAR: DSISR: 4000 SOFTE: 1 GPR00: c02709d4 c03ffb11b690 c141a400 c03fff50e300 GPR04: 024001c2 c03ffb11b780 00219df5 GPR08: 003ffb09 c1454fd8 GPR12: 4400 c7b6 024001c2 024001c2 GPR16: 024001c2 0002 GPR20: 000c 024200c0 GPR24: c16eef48 c03fff50fd00 024001c2 GPR28: c03fff50fd00 c03fff50e300 c03ffb11b820 NIP [c0302848] mem_cgroup_soft_limit_reclaim+0xf8/0x4f0 [5.213613] LR [c02709d4] do_try_to_free_pages+0x1b4/0x450 [5.230521] Call Trace: [5.230643] [c03ffb11b760] [c02709d4] do_try_to_free_pages+0x1b4/0x450 [5.254184] [c03ffb11b800] [c0270d68] try_to_free_pages+0xf8/0x270 [5.281896] [c03ffb11b890] [c0259b88] __alloc_pages_nodemask+0x7a8/0xff0 [5.321407] [c03ffb11bab0] [c0282cd0] pcpu_populate_chunk+0x110/0x520 [5.336262] [c03ffb11bb50] [c02841b8] pcpu_balance_workfn+0x758/0x960 [5.351526] [c03ffb11bc50] [c00ecdd0] process_one_work+0x2b0/0x5a0 [5.362561] [c03ffb11bce0] [c00ed168] worker_thread+0xa8/0x660 [5.374007] [c03ffb11bd80] [c00f5320] kthread+0x110/0x130 [5.385160] [c03ffb11be30] [c000c0e8] ret_from_kernel_thread+0x5c/0x74 [5.389456] Instruction dump: [5.410036] eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 4e800020 3d230001 e9499a42 3d220004 [5.423598] 3929abd8 794a1f24 7d295214 eac90100 2fa9 419eff74 3b20 [5.436503] ---[ end trace 23b650e96be5c549 ]--- [5.
[Kernel-packages] [Bug 1665113] Re: [Ubuntu 17.04] Kernel panics when large number of hugepages is passed as an boot argument to kernel.
Applied 'mm/cgroup: avoid panic when init with low memory' to the Zesty 4.10 kernel ** Also affects: linux (Ubuntu Zesty) Importance: High Assignee: Canonical Kernel Team (canonical-kernel-team) Status: Triaged ** Changed in: linux (Ubuntu Zesty) Status: Triaged => Fix Committed ** Changed in: linux (Ubuntu Zesty) Assignee: Canonical Kernel Team (canonical-kernel-team) => Tim Gardner (timg-tpi) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1665113 Title: [Ubuntu 17.04] Kernel panics when large number of hugepages is passed as an boot argument to kernel. Status in linux package in Ubuntu: Fix Committed Status in linux source package in Zesty: Fix Committed Bug description: Issue: --- Kernel unable to handle paging request and panic occurs when more number of hugepages is passed as a boot argument to the kernel . Environment: -- Power NV : Habanaro Bare metal OS : Ubuntu 17.04 Kernel Version : 4.9.0-11-generic Steps To reproduce: --- 1 - When the ubuntu Kernel boots try to add the boot argument 'hugepages = 1200'. The Kernel Panics and displays call traces like as below. [5.030274] Unable to handle kernel paging request for data at address 0x [5.030323] Faulting instruction address: 0xc0302848 [5.030366] Oops: Kernel access of bad area, sig: 11 [#1] [5.030399] SMP NR_CPUS=2048 [5.030416] NUMA [5.039443] PowerNV [5.039461] Modules linked in: [5.050091] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 4.9.0-11-generic #12-Ubuntu [5.053266] Workqueue: events pcpu_balance_workfn [5.080647] task: c03c8fe9b800 task.stack: c03ffb118000 [5.090876] NIP: c0302848 LR: c02709d4 CTR: c016cef0 [5.094175] REGS: c03ffb11b410 TRAP: 0300 Not tainted (4.9.0-11-generic) [5.103040] MSR: 92009033 [ 5.114466] CR: 22424222 XER: [5.124932] CFAR: c0008a60 DAR: DSISR: 4000 SOFTE: 1 GPR00: c02709d4 c03ffb11b690 c141a400 c03fff50e300 GPR04: 024001c2 c03ffb11b780 00219df5 GPR08: 003ffb09 c1454fd8 GPR12: 4400 c7b6 024001c2 024001c2 GPR16: 024001c2 0002 GPR20: 000c 024200c0 GPR24: c16eef48 c03fff50fd00 024001c2 GPR28: c03fff50fd00 c03fff50e300 c03ffb11b820 NIP [c0302848] mem_cgroup_soft_limit_reclaim+0xf8/0x4f0 [5.213613] LR [c02709d4] do_try_to_free_pages+0x1b4/0x450 [5.230521] Call Trace: [5.230643] [c03ffb11b760] [c02709d4] do_try_to_free_pages+0x1b4/0x450 [5.254184] [c03ffb11b800] [c0270d68] try_to_free_pages+0xf8/0x270 [5.281896] [c03ffb11b890] [c0259b88] __alloc_pages_nodemask+0x7a8/0xff0 [5.321407] [c03ffb11bab0] [c0282cd0] pcpu_populate_chunk+0x110/0x520 [5.336262] [c03ffb11bb50] [c02841b8] pcpu_balance_workfn+0x758/0x960 [5.351526] [c03ffb11bc50] [c00ecdd0] process_one_work+0x2b0/0x5a0 [5.362561] [c03ffb11bce0] [c00ed168] worker_thread+0xa8/0x660 [5.374007] [c03ffb11bd80] [c00f5320] kthread+0x110/0x130 [5.385160] [c03ffb11be30] [c000c0e8] ret_from_kernel_thread+0x5c/0x74 [5.389456] Instruction dump: [5.410036] eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 4e800020 3d230001 e9499a42 3d220004 [5.423598] 3929abd8 794a1f24 7d295214 eac90100 2fa9 419eff74 3b20 [5.436503] ---[ end trace 23b650e96be5c549 ]--- [5.439700] This is purely a negative scenario where the system does not have enough memory as the hugepages is given a very large argument. Free output in a system: free -h totalusedfree shared buff/cache available Mem: 251G2.1G248G5.2M502M 248G Swap: 2.0G159M1.8G The same scenario when tried after the linux is up like as, echo 1200 > /proc/sys/vm/nr_hugepages HugePages_Total: 15069 HugePages_Free:15069 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 16384 kB root@ltc-haba2:~# free -h totalusedfree shared buff/cache available Mem: 251G237G 13G5.6M311M 13G Swap: 2.0G159M1.8G In this case the kernel is able to allocate around 237 Gb
[Kernel-packages] [Bug 1665113] Re: [Ubuntu 17.04] Kernel panics when large number of hugepages is passed as an boot argument to kernel.
** Changed in: linux (Ubuntu) Importance: Undecided => High ** Changed in: linux (Ubuntu) Status: New => Triaged ** Changed in: linux (Ubuntu) Assignee: Taco Screen team (taco-screen-team) => Canonical Kernel Team (canonical-kernel-team) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1665113 Title: [Ubuntu 17.04] Kernel panics when large number of hugepages is passed as an boot argument to kernel. Status in linux package in Ubuntu: Triaged Bug description: Issue: --- Kernel unable to handle paging request and panic occurs when more number of hugepages is passed as a boot argument to the kernel . Environment: -- Power NV : Habanaro Bare metal OS : Ubuntu 17.04 Kernel Version : 4.9.0-11-generic Steps To reproduce: --- 1 - When the ubuntu Kernel boots try to add the boot argument 'hugepages = 1200'. The Kernel Panics and displays call traces like as below. [5.030274] Unable to handle kernel paging request for data at address 0x [5.030323] Faulting instruction address: 0xc0302848 [5.030366] Oops: Kernel access of bad area, sig: 11 [#1] [5.030399] SMP NR_CPUS=2048 [5.030416] NUMA [5.039443] PowerNV [5.039461] Modules linked in: [5.050091] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 4.9.0-11-generic #12-Ubuntu [5.053266] Workqueue: events pcpu_balance_workfn [5.080647] task: c03c8fe9b800 task.stack: c03ffb118000 [5.090876] NIP: c0302848 LR: c02709d4 CTR: c016cef0 [5.094175] REGS: c03ffb11b410 TRAP: 0300 Not tainted (4.9.0-11-generic) [5.103040] MSR: 92009033 [ 5.114466] CR: 22424222 XER: [5.124932] CFAR: c0008a60 DAR: DSISR: 4000 SOFTE: 1 GPR00: c02709d4 c03ffb11b690 c141a400 c03fff50e300 GPR04: 024001c2 c03ffb11b780 00219df5 GPR08: 003ffb09 c1454fd8 GPR12: 4400 c7b6 024001c2 024001c2 GPR16: 024001c2 0002 GPR20: 000c 024200c0 GPR24: c16eef48 c03fff50fd00 024001c2 GPR28: c03fff50fd00 c03fff50e300 c03ffb11b820 NIP [c0302848] mem_cgroup_soft_limit_reclaim+0xf8/0x4f0 [5.213613] LR [c02709d4] do_try_to_free_pages+0x1b4/0x450 [5.230521] Call Trace: [5.230643] [c03ffb11b760] [c02709d4] do_try_to_free_pages+0x1b4/0x450 [5.254184] [c03ffb11b800] [c0270d68] try_to_free_pages+0xf8/0x270 [5.281896] [c03ffb11b890] [c0259b88] __alloc_pages_nodemask+0x7a8/0xff0 [5.321407] [c03ffb11bab0] [c0282cd0] pcpu_populate_chunk+0x110/0x520 [5.336262] [c03ffb11bb50] [c02841b8] pcpu_balance_workfn+0x758/0x960 [5.351526] [c03ffb11bc50] [c00ecdd0] process_one_work+0x2b0/0x5a0 [5.362561] [c03ffb11bce0] [c00ed168] worker_thread+0xa8/0x660 [5.374007] [c03ffb11bd80] [c00f5320] kthread+0x110/0x130 [5.385160] [c03ffb11be30] [c000c0e8] ret_from_kernel_thread+0x5c/0x74 [5.389456] Instruction dump: [5.410036] eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 4e800020 3d230001 e9499a42 3d220004 [5.423598] 3929abd8 794a1f24 7d295214 eac90100 2fa9 419eff74 3b20 [5.436503] ---[ end trace 23b650e96be5c549 ]--- [5.439700] This is purely a negative scenario where the system does not have enough memory as the hugepages is given a very large argument. Free output in a system: free -h totalusedfree shared buff/cache available Mem: 251G2.1G248G5.2M502M 248G Swap: 2.0G159M1.8G The same scenario when tried after the linux is up like as, echo 1200 > /proc/sys/vm/nr_hugepages HugePages_Total: 15069 HugePages_Free:15069 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 16384 kB root@ltc-haba2:~# free -h totalusedfree shared buff/cache available Mem: 251G237G 13G5.6M311M 13G Swap: 2.0G159M1.8G In this case the kernel is able to allocate around 237 Gb for hugetlb. But while the system is booting it gives us panic so please let know if this scenario is expected to be handled. I identified the root cause of the panic. When the system is running with low memory during mem cg
