[Bug 306830] Re: oom-killer kicks in while ~15G of ram are free, freezes server
Mike, please open a separate bug for you issue with the relevant details about the Hardware. I am marking this bug Invalid now since the OP has switched to x86_64 where the bug is not visible anymore. ** Changed in: linux (Ubuntu) Importance: High = Undecided ** Changed in: linux (Ubuntu) Status: Triaged = Invalid -- oom-killer kicks in while ~15G of ram are free, freezes server https://bugs.launchpad.net/bugs/306830 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 306830] Re: oom-killer kicks in while ~15G of ram are free, freezes server
A similar oom-killer freeze happens to x86_64 laptop with no particular memory load. It happens only infrequently, although it seems to appear immediately after an apparmor warning. % uname -a Linux ithaca 2.6.27-11-generic #1 SMP Thu Jan 29 19:28:32 UTC 2009 x86_64 GNU/Linux % cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=8.10 DISTRIB_CODENAME=intrepid DISTRIB_DESCRIPTION=Ubuntu 8.10 kern.log: Apr 1 19:45:28 ithaca kernel: [ 4748.957092] type=1503 audit(1238633060.027:8): operation=capable name=sys_admin pid=8559 profile=/usr/sbin/cupsd Apr 1 19:45:28 ithaca kernel: [ 4748.957104] type=1503 audit(1238633060.027:9): operation=capable name=sys_resource pid=8559 profile=/usr/sbin/cupsd Apr 1 19:45:28 ithaca kernel: [ 4748.957109] type=1503 audit(1238633060.027:10): operation=capable name=sys_rawio pid=8559 profile=/usr/sbin/cupsd Apr 1 19:45:28 ithaca kernel: [ 4748.957202] type=1503 audit(1238633060.027:11): operation=capable name=sys_resource pid=8559 profile=/usr/sbin/named Apr 1 19:45:28 ithaca kernel: [ 4748.957210] type=1503 audit(1238633060.027:12): operation=capable name=sys_resource pid=8559 profile=/usr/sbin/named Apr 1 19:45:28 ithaca kernel: [ 4748.957216] type=1503 audit(1238633060.027:13): operation=capable name=sys_resource pid=8559 profile=/usr/sbin/named Apr 1 19:45:28 ithaca kernel: [ 4748.957375] VBoxSVC invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 Apr 1 19:45:28 ithaca kernel: [ 4748.957379] Pid: 8559, comm: VBoxSVC Not tainted 2.6.27-11-generic #1 Apr 1 19:45:28 ithaca kernel: [ 4748.957381] Apr 1 19:45:28 ithaca kernel: [ 4748.957381] Call Trace: Apr 1 19:45:28 ithaca kernel: [ 4748.957392] [802af80a] oom_kill_process+0x9a/0x230 Apr 1 19:45:28 ithaca kernel: [ 4748.957395] [802afcef] ? select_bad_process+0xef/0x130 Apr 1 19:45:31 ithaca kernel: [ 4748.957398] [802afec5] out_of_memory+0x195/0x270 Apr 1 19:45:31 ithaca kernel: [ 4748.957402] [802b2d89] __alloc_pages_internal+0x4d9/0x520 Apr 1 19:45:31 ithaca kernel: [ 4748.957407] [802d5c0d] alloc_pages_current+0xad/0x110 Apr 1 19:45:31 ithaca kernel: [ 4748.957410] [802ac5a7] __page_cache_alloc+0x67/0x80 Apr 1 19:45:31 ithaca kernel: [ 4748.957414] [802b64ec] __do_page_cache_readahead+0xec/0x220 Apr 1 19:45:31 ithaca kernel: [ 4748.957417] [802ac6a0] ? sync_page+0x0/0x70 Apr 1 19:45:31 ithaca kernel: [ 4748.957421] [802b6683] do_page_cache_readahead+0x63/0x90 Apr 1 19:45:31 ithaca kernel: [ 4748.957424] [802adcda] filemap_fault+0x34a/0x430 Apr 1 19:45:31 ithaca kernel: [ 4748.957428] [802c2204] __do_fault+0x64/0x440 Apr 1 19:45:31 ithaca kernel: [ 4748.957433] [8026ad51] ? lock_hrtimer_base+0x31/0x60 Apr 1 19:45:31 ithaca kernel: [ 4748.957436] [8026ae14] ? hrtimer_try_to_cancel+0x44/0x90 Apr 1 19:45:31 ithaca kernel: [ 4748.957440] [802c319e] handle_mm_fault+0x1ee/0x470 Apr 1 19:45:31 ithaca kernel: [ 4748.957444] [8050640f] do_page_fault+0x34f/0x750 Apr 1 19:45:31 ithaca kernel: [ 4748.957448] [802744c9] ? get_futex_key+0x19/0x160 Apr 1 19:45:31 ithaca kernel: [ 4748.957453] [8050367e] ? _spin_lock+0xe/0x20 Apr 1 19:45:31 ithaca kernel: [ 4748.957455] [80275840] ? futex_wake+0x100/0x130 Apr 1 19:45:31 ithaca kernel: [ 4748.957458] [80276c95] ? do_futex+0x65/0x140 Apr 1 19:45:31 ithaca kernel: [ 4748.957461] [80276e2c] ? s_futexsy+0xbc/0x150 Apr 1 19:45:31 ithaca kernel: [ 4748.957465] [80503aba] error_exit+0x0/0x70 Apr 1 19:45:31 ithaca kernel: [ 4748.957467] Apr 1 19:45:31 ithaca kernel: [ 4748.957468] Mem-Info: Apr 1 19:45:31 ithaca kernel: [ 4748.957470] Node 0 DMA per-cpu: Apr 1 19:45:31 ithaca kernel: [ 4748.957473] CPU0: hi:0, btch: 1 usd: 0 Apr 1 19:45:31 ithaca kernel: [ 4748.957475] CPU1: hi:0, btch: 1 usd: 0 Apr 1 19:45:31 ithaca kernel: [ 4748.957477] Node 0 DMA32 per-cpu: Apr 1 19:45:31 ithaca kernel: [ 4748.957479] CPU0: hi: 186, btch: 31 usd: 160 Apr 1 19:45:31 ithaca kernel: [ 4748.957481] CPU1: hi: 186, btch: 31 usd: 155 Apr 1 19:45:31 ithaca kernel: [ 4748.957485] Active:694485 inactive:2563 dirty:1 writeback:0 unstable:0 Apr 1 19:45:31 ithaca kernel: [ 4748.957486] free:4088 slab:16287 mapped:91 pagetables:6823 bounce:0 Apr 1 19:45:31 ithaca kernel: [ 4748.957488] Node 0 DMA free:9416kB min:16kB low:20kB high:24kB active:0kB inactive:0kB present:8376kB pages_scanned:0 all_unreclaimable? yes Apr 1 19:45:31 ithaca kernel: [ 4748.957492] lowmem_reserve[]: 0 2999 2999 2999 Apr 1 19:45:31 ithaca kernel: [ 4748.957496] Node 0 DMA32 free:6936kB min:6996kB low:8744kB high:10492kB active:2777940kB inactive:10252kB present:3071184kB pages_scanned:491362 all_unreclaimable? no Apr 1 19:45:31 ithaca kernel: [ 4748.957501] lowmem_reserve[]: 0 0 0 0 Apr 1 19:45:31 ithaca kernel: [
[Bug 306830] Re: oom-killer kicks in while ~15G of ram are free, freezes server
We moved the servers to 2.6.24-22-server x86_64 two weeks ago and had no problem since, while running the same workloads and processes. -- oom-killer kicks in while ~15G of ram are free, freezes server https://bugs.launchpad.net/bugs/306830 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 306830] Re: oom-killer kicks in while ~15G of ram are free, freezes server
Upgraded machine to intrepid, to test with a 2.6.27-9-server kernel. happened again. dmesg attached. ** Attachment added: dmesg2.txt http://launchpadlibrarian.net/20632261/dmesg2.txt -- oom-killer kicks in while ~15G of ram are free, freezes server https://bugs.launchpad.net/bugs/306830 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 306830] Re: oom-killer kicks in while ~15G of ram are free, freezes server
** Changed in: linux (Ubuntu) Importance: Undecided = High Status: New = Triaged -- oom-killer kicks in while ~15G of ram are free, freezes server https://bugs.launchpad.net/bugs/306830 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 306830] Re: oom-killer kicks in while ~15G of ram are free, freezes server
** Attachment added: memlog http://launchpadlibrarian.net/20492192/memlog -- oom-killer kicks in while ~15G of ram are free, freezes server https://bugs.launchpad.net/bugs/306830 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 306830] Re: oom-killer kicks in while ~15G of ram are free, freezes server
It happened again on a different Machine (also ibm xseries 3650, kernel 2.6.24-22-server. I attach dmesg and a file that contains copies of /proc/meminfo each 60s. ** Attachment added: dmesg http://launchpadlibrarian.net/20492189/dmesg -- oom-killer kicks in while ~15G of ram are free, freezes server https://bugs.launchpad.net/bugs/306830 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 306830] Re: oom-killer kicks in while ~15G of ram are free, freezes server
It happened again, this time the machine didn't stall, and a bit more info could be collected: syslog: Dec 10 23:29:06 omg kernel: [51151.508470] brix invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=-17 Dec 10 23:29:07 omg kernel: [51151.508476] Pid: 995, comm: brix Not tainted 2.6.24-22-server #1 Dec 10 23:29:07 omg kernel: [51151.508489] [oom_kill_process+0x10a/0x120] oom_kill_process+0x10a/0x120 Dec 10 23:29:07 omg kernel: [51151.508499] [out_of_memory+0x167/0x1a0] out_of_memory+0x167/0x1a0 Dec 10 23:29:07 omg kernel: [51151.508504] [nfs:__alloc_pages+0x34c/0x380] __alloc_pages+0x34c/0x380 Dec 10 23:29:07 omg kernel: [51151.508512] [tcp_sendmsg+0x361/0xbc0] tcp_sendmsg+0x361/0xbc0 Dec 10 23:29:07 omg kernel: [51151.508523] [tcp_sendmsg+0x9c3/0xbc0] tcp_sendmsg+0x9c3/0xbc0 Dec 10 23:29:07 omg kernel: [51151.508526] [kmap_atomic_prot+0xfc/0x130] kmap_atomic_prot+0xfc/0x130 Dec 10 23:29:07 omg kernel: [51151.508532] [nfs:kmap_atomic+0x1c/0x30] kmap_atomic+0x1c/0x30 Dec 10 23:29:07 omg kernel: [51151.508538] [sock_aio_write+0x109/0x120] sock_aio_write+0x109/0x120 Dec 10 23:29:07 omg kernel: [51151.508544] [nfs:do_sync_write+0xd5/0x120] do_sync_write+0xd5/0x120 Dec 10 23:29:07 omg kernel: [51151.508550] [c0145f10] autoremove_wake_function+0x0/0x40 meminfo: MemTotal: 49924560 kB MemFree: 16970328 kB Buffers: 960 kB Cached: 31965780 kB SwapCached: 0 kB Active: 13818128 kB Inactive: 18619620 kB HighTotal:49413916 kB HighFree: 16961756 kB LowTotal: 510644 kB LowFree: 8572 kB SwapTotal: 9092748 kB SwapFree: 9092748 kB Dirty:5096 kB Writeback: 0 kB AnonPages: 470908 kB Mapped: 8668 kB Slab: 489532 kB SReclaimable: 440788 kB SUnreclaim: 48744 kB PageTables: 4784 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit: 34055028 kB Committed_AS: 1273408 kB VmallocTotal: 118776 kB VmallocUsed: 7844 kB VmallocChunk: 110616 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB brix processes: root 984 0.0 0.0 54628 33412 ?S14:22 0:00 (brix) wtf-searchd overlord root 987 6.3 0.0 63648 43912 ?S14:22 34:36 (brix) wtf-searchd - 65 rpm root 988 6.4 0.0 69236 49400 ?S14:22 35:15 (brix) wtf-searchd - 59 rpm root 991 6.5 0.0 58404 38696 ?S14:22 35:42 (brix) wtf-searchd - 68 rpm root 993 6.7 0.0 58788 39064 ?S14:22 37:03 (brix) wtf-searchd - 68 rpm root 994 6.6 0.0 58828 39020 ?S14:22 36:33 (brix) wtf-searchd - 76 rpm root 995 6.4 0.0 57592 37848 ?S14:22 35:37 (brix) wtf-searchd - 67 rpm root 996 6.1 0.0 59548 39812 ?R14:22 33:33 (brix) wtf-searchd - 62 rpm root 998 6.6 0.0 57560 37832 ?S14:22 36:46 (brix) wtf-searchd - 74 rpm root 999 6.3 0.0 57988 38224 ?S14:22 34:44 (brix) wtf-searchd - 65 rpm root 1000 6.7 0.0 57860 38132 ?S14:22 37:00 (brix) wtf-searchd - 74 rpm ** Attachment added: dmesg.log http://launchpadlibrarian.net/20359737/dmesg.log -- oom-killer kicks in while ~15G of ram are free, freezes server https://bugs.launchpad.net/bugs/306830 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 306830] Re: oom-killer kicks in while ~15G of ram are free, freezes server
** Attachment added: uname-a.log http://launchpadlibrarian.net/20341447/uname-a.log -- oom-killer kicks in while ~15G of ram are free, freezes server https://bugs.launchpad.net/bugs/306830 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 306830] Re: oom-killer kicks in while ~15G of ram are free, freezes server
** Attachment added: version.log http://launchpadlibrarian.net/20341458/version.log -- oom-killer kicks in while ~15G of ram are free, freezes server https://bugs.launchpad.net/bugs/306830 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 306830] Re: oom-killer kicks in while ~15G of ram are free, freezes server
** Attachment added: dmesg.log http://launchpadlibrarian.net/20341462/dmesg.log -- oom-killer kicks in while ~15G of ram are free, freezes server https://bugs.launchpad.net/bugs/306830 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 306830] Re: oom-killer kicks in while ~15G of ram are free, freezes server
** Attachment added: lspci-vnvn.log http://launchpadlibrarian.net/20341465/lspci-vnvn.log -- oom-killer kicks in while ~15G of ram are free, freezes server https://bugs.launchpad.net/bugs/306830 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
