Hi,
Seems the linux kernel has let loose the oom-killer on this system.
Over a period of about 6 hours the kernel has slowly chocked the system
to death using the oom-killer, I don't know what the cause is/was?
I found this (I like the analogy in red in the first link)
http://lwn.net/Articles/111408/ http://lwn.net/Articles/104179/
So I understand why the kernel started the oom-killer but why did it not stop
after killing off a few large processes?
Instead just kept killing off everything until nothing was left.
The syslog stopped writing to the disk at about 20:10 and the server was not
rebooted until the next day at 8:01am
The X screen was unresponsive no mouse movement or keyboard response.
Thanks
Simon.
Some details below the full log is quite large I have renamed it .txt as it
opens in a browser to look at easily.
full message log (780Kb) http://www.knight.gen.nz/message.txt
The server is running gentoo 24/7 since it was built up at Roberts mini install
fest April 06
Linux genserv 2.6.15-gentoo-r1 #2 SMP PREEMPT Tue Apr 11 23:47:35 NZST 2006
i686 Intel(R) Pentium(R) 4 CPU 3.00GHz GNU/Linux
everything seemed to be working o.k. until 14:11
<http://www.knight.gen.nz/message.txt>
Jul 29 14:11:54 oom-killer: gfp_mask=0x201d2, order=0
Jul 29 14:19:28 genserv Out of Memory: Killed process 28497 (pickup).
Jul 29 14:19:28 genserv Out of Memory: Killed process 9339 (asterisk).
Jul 29 14:19:28 genserv Out of Memory: Killed process 28715 (apache2).
Jul 29 14:19:28 genserv Out of Memory: Killed process 28749 (apache2).
Jul 29 14:19:28 genserv Out of Memory: Killed process 9830 (hald-runner).
Jul 29 15:11:05 genserv Out of Memory: Killed process 10092 (ivman).
Jul 29 15:11:05 genserv Out of Memory: Killed process 10614 (qmgr).
Jul 29 15:11:05 genserv Out of Memory: Killed process 10614 (qmgr).
Jul 29 15:11:05 genserv Out of Memory: Killed process 9552 (dbus-daemon).
Jul 29 15:11:19 genserv printk: 558 messages suppressed.
Jul 29 16:08:05 genserv Out of Memory: Killed process 29523 (qmgr).
Jul 29 16:08:05 genserv postfix/master[10599]: warning: process
/usr/lib/postfix/qmgr pid 29523 killed by signal 9
Jul 29 16:36:06 genserv 142 pages pagetables
Jul 29 16:36:06 genserv Out of Memory: Killed process 29522 (pickup).
Jul 29 16:36:06 genserv Out of Memory: Killed process 29522 (pickup).
Jul 29 16:36:06 genserv Out of Memory: Killed process 6777 (smbd).
Jul 29 16:36:06 genserv Out of Memory: Killed process 9465 (authdaemond).
Jul 29 16:36:06 genserv Out of Memory: Killed process 10542 (saslauthd).
Jul 29 16:36:06 genserv Out of Memory: Killed process 10642 (smbd).
Jul 29 16:36:06 genserv Out of Memory: Killed process 9504 (cupsd).
Jul 29 16:55:48 genserv printk: 1056 messages suppressed.
Jul 29 16:57:29 genserv Out of Memory: Killed process 29568 (authdaemond).
Jul 29 16:57:29 genserv Out of Memory: Killed process 10544 (saslauthd).
Jul 29 16:57:29 genserv Out of Memory: Killed process 10599 (master).
Jul 29 16:57:29 genserv Out of Memory: Killed process 9214 (sshd).
There are hundreds of pages like this over the proceding 6 hours but this is the last that was written to the log.
Jul 29 20:10:01 genserv HighMem per-cpu: empty
Jul 29 20:10:01 genserv Free pages: 5392kB (0kB HighMem)
Jul 29 20:10:01 genserv Active:1307 inactive:1276 dirty:0 writeback:0
unstable:0 free:1348 slab:122973 mapped:151 pagetables:73
Jul 29 20:10:01 genserv DMA free:2064kB min:88kB low:108kB high:132kB
active:16kB inactive:0kB present:16384kB pages_scanned:12 all_unreclaimable? no
Jul 29 20:10:01 genserv lowmem_reserve[]: 0 0 494 494
Jul 29 20:10:01 genserv DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB
inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Jul 29 20:10:01 genserv lowmem_reserve[]: 0 0 494 494
Jul 29 20:10:01 genserv Normal free:3328kB min:2800kB low:3500kB high:4200kB
active:5212kB inactive:5104kB present:506752kB pages_scanned:1551
all_unreclaimable? no
Jul 29 20:10:01 genserv lowmem_reserve[]: 0 0 0 0
Jul 29 20:10:01 genserv HighMem free:0kB min:128kB low:128kB high:128kB
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Jul 29 20:10:01 genserv lowmem_reserve[]: 0 0 0 0
Jul 29 20:10:01 genserv DMA: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB
0*512kB 0*1024kB 1*2048kB 0*4096kB = 2064kB
Jul 29 20:10:01 genserv DMA32: empty
Jul 29 20:10:01 genserv Normal: 110*4kB 27*8kB 5*16kB 3*32kB 1*64kB 3*128kB
0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3328kB
Jul 29 20:10:01 genserv HighMem: empty
Jul 29 20:10:01 genserv Swap cache: add 619226, delete 619182, find
170943/323620, race 17+2068
Jul 29 20:10:01 genserv Free swap = 2069864kB
Jul 29 20:10:01 genserv Total swap = 2072368kB
Jul 29 20:10:01 genserv Free swap: 2069864kB
Jul 29 20:10:01 genserv 130784 pages of RAM
Jul 29 20:10:01 genserv 0 pages of HIGHMEM
Jul 29 20:10:01 genserv 2549 reserved pages
Jul 29 20:10:01 genserv 2577 pages shared
Jul 29 20:10:01 genserv 44 pages swap cached
Jul 29 20:10:01 genserv 0 pages dirty
Jul 29 20:10:01 genserv 0 pages writeback
Jul 29 20:10:01 genserv 153 pages mapped
Jul 29 20:10:01 genserv 122978 pages slab
Jul 29 20:10:01 genserv 73 pages pagetables
Jul 29 20:10:01 genserv Out of Memory: Killed process 29850 (fcron).
