oom-killer just killed my haproxy instance. Anyone know if there is a way
to prioritize haproxy and have it get killed after something else? Or, any
tuning that might help.
It looked like I had plenty of swap space available when it decided to kill
haproxy.
Thanks for any advice.
Linux 3.3.7-1.fc16.x86_64
HA-Proxy version 1.4.20
# free -m
total used free shared buffers cached
Mem: 995 357 637 0 3 25
-/+ buffers/cache: 328 667
Swap: 2015 92 1923
messages:
Feb 1 15:48:03 prx2 kernel: [21556065.639023] sched: RT throttling
activated
Feb 1 15:48:03 prx2 heartbeat: [15556]: WARN: Gmain_timeout_dispatch:
Dispatch function for check for signals was delayed 1470 ms (> 1010 ms)
before being called (GSource: 0x20b4c20)
Feb 1 15:48:03 prx2 heartbeat: [15556]: info: Gmain_timeout_dispatch:
started at 2588817760 should have started at 2588817613
Feb 1 15:48:14 prx2 kernel: [21556076.952895] oom_kill_process: 997778
callbacks suppressed
Feb 1 15:48:14 prx2 kernel: [21556076.952900] haproxy invoked oom-killer:
gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
Feb 1 15:48:14 prx2 kernel: [21556076.952934] haproxy cpuset=/
mems_allowed=0
Feb 1 15:48:14 prx2 kernel: [21556076.952946] Pid: 9654, comm: haproxy Not
tainted 3.3.7-1.fc16.x86_64 #1
Feb 1 15:48:14 prx2 kernel: [21556076.952948] Call Trace:
Feb 1 15:48:14 prx2 kernel: [21556076.952978] [<ffffffff810c7811>] ?
cpuset_print_task_mems_allowed+0x91/0xa0
Feb 1 15:48:14 prx2 kernel: [21556076.952993] [<ffffffff81123cd0>]
dump_header+0x80/0x1d0
Feb 1 15:48:14 prx2 kernel: [21556076.952997] [<ffffffff81124125>]
oom_kill_process+0x85/0x290
Feb 1 15:48:14 prx2 kernel: [21556076.953000] [<ffffffff81124770>]
out_of_memory+0x1c0/0x400
Feb 1 15:48:14 prx2 kernel: [21556076.953004] [<ffffffff81129d7f>]
__alloc_pages_nodemask+0x8df/0x8f0
Feb 1 15:48:14 prx2 kernel: [21556076.953016] [<ffffffff81521652>] ?
__ip_local_out+0xa2/0xb0
Feb 1 15:48:14 prx2 kernel: [21556076.953022] [<ffffffff81160a93>]
alloc_pages_current+0xa3/0x110
Feb 1 15:48:14 prx2 kernel: [21556076.953025] [<ffffffff8152adde>]
tcp_sendmsg+0x53e/0xdf0
Feb 1 15:48:14 prx2 kernel: [21556076.953031] [<ffffffff81550e74>]
inet_sendmsg+0x64/0xb0
Feb 1 15:48:14 prx2 kernel: [21556076.953043] [<ffffffff8126dc63>] ?
selinux_socket_sendmsg+0x23/0x30
Feb 1 15:48:14 prx2 kernel: [21556076.953052] [<ffffffff814ced17>]
sock_sendmsg+0x117/0x130
Feb 1 15:48:14 prx2 kernel: [21556076.953055] [<ffffffff81521652>] ?
__ip_local_out+0xa2/0xb0
Feb 1 15:48:14 prx2 kernel: [21556076.953065] [<ffffffff81067d6e>] ?
mod_timer+0x13e/0x2f0
Feb 1 15:48:14 prx2 kernel: [21556076.953069] [<ffffffff814d220d>]
sys_sendto+0x13d/0x190
Feb 1 15:48:14 prx2 kernel: [21556076.953073] [<ffffffff810d345c>] ?
__audit_syscall_entry+0xcc/0x310
Feb 1 15:48:14 prx2 kernel: [21556076.953076] [<ffffffff810d3a76>] ?
__audit_syscall_exit+0x3d6/0x410
Feb 1 15:48:14 prx2 kernel: [21556076.953084] [<ffffffff815fc529>]
system_call_fastpath+0x16/0x1b
Feb 1 15:48:14 prx2 kernel: [21556076.953086] Mem-Info:
Feb 1 15:48:14 prx2 kernel: [21556076.953088] Node 0 DMA per-cpu:
Feb 1 15:48:14 prx2 kernel: [21556076.953198] CPU 0: hi: 0, btch:
1 usd: 0
Feb 1 15:48:14 prx2 kernel: [21556076.953200] CPU 1: hi: 0, btch:
1 usd: 0
Feb 1 15:48:14 prx2 kernel: [21556076.953203] CPU 2: hi: 0, btch:
1 usd: 0
Feb 1 15:48:14 prx2 kernel: [21556076.953205] CPU 3: hi: 0, btch:
1 usd: 0
Feb 1 15:48:14 prx2 kernel: [21556076.953206] Node 0 DMA32 per-cpu:
Feb 1 15:48:14 prx2 kernel: [21556076.953209] CPU 0: hi: 186, btch:
31 usd: 56
Feb 1 15:48:14 prx2 kernel: [21556076.953210] CPU 1: hi: 186, btch:
31 usd: 0
Feb 1 15:48:14 prx2 kernel: [21556076.953212] CPU 2: hi: 186, btch:
31 usd: 0
Feb 1 15:48:14 prx2 kernel: [21556076.953214] CPU 3: hi: 186, btch:
31 usd: 29
Feb 1 15:48:14 prx2 kernel: [21556076.953218] active_anon:138
inactive_anon:194 isolated_anon:0
Feb 1 15:48:14 prx2 kernel: [21556076.953219] active_file:24
inactive_file:80 isolated_file:0
Feb 1 15:48:14 prx2 kernel: [21556076.953220] unevictable:4373 dirty:0
writeback:213 unstable:0
Feb 1 15:48:14 prx2 kernel: [21556076.953221] free:12235
slab_reclaimable:47686 slab_unreclaimable:25122
Feb 1 15:48:14 prx2 kernel: [21556076.953222] mapped:1506 shmem:2
pagetables:725 bounce:0
Feb 1 15:48:14 prx2 kernel: [21556076.953224] Node 0 DMA free:4640kB
min:680kB low:848kB high:1020kB active_anon:44kB inactive_anon:84kB
active_file:0kB inactive_file:44kB unevictable:352kB isolated(anon):0kB
isolated(file):0kB present:15656kB mlocked:352kB dirty:0kB writeback:112kB
mapped:352kB shmem:0kB slab_reclaimable:328kB slab_unreclaimable:276kB
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:592 all_unreclaimable? yes
Feb 1 15:48:14 prx2 kernel: [21556076.953236] lowmem_reserve[]: 0 992 992
992
Feb 1 15:48:14 prx2 kernel: [21556076.953239] Node 0 DMA32 free:44300kB
min:44372kB low:55464kB high:66556kB active_anon:508kB inactive_anon:692kB
active_file:96kB inactive_file:276kB unevictable:17140kB isolated(anon):0kB
isolated(file):0kB present:1016000kB mlocked:17140kB dirty:0kB
writeback:740kB mapped:5672kB shmem:8kB slab_reclaimable:190416kB
slab_unreclaimable:100212kB kernel_stack:728kB pagetables:2900kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2692
all_unreclaimable? yes
Feb 1 15:48:14 prx2 kernel: [21556076.953251] lowmem_reserve[]: 0 0 0 0
Feb 1 15:48:14 prx2 kernel: [21556076.953254] Node 0 DMA: 19*4kB 8*8kB
14*16kB 14*32kB 7*64kB 5*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB =
4716kB
Feb 1 15:48:14 prx2 kernel: [21556076.953263] Node 0 DMA32: 436*4kB
458*8kB 376*16kB 166*32kB 105*64kB 49*128kB 15*256kB 5*512kB 4*1024kB
2*2048kB 0*4096kB = 44320kB
Feb 1 15:48:14 prx2 kernel: [21556076.953270] 1934 total pagecache pages
Feb 1 15:48:14 prx2 kernel: [21556076.953272] 348 pages in swap cache
Feb 1 15:48:14 prx2 kernel: [21556076.953274] Swap cache stats: add 52567,
delete 52219, find 6108879/6110507
Feb 1 15:48:14 prx2 kernel: [21556076.953276] Free swap = 1896624kB
Feb 1 15:48:14 prx2 kernel: [21556076.953277] Total swap = 2064380kB
Feb 1 15:48:14 prx2 kernel: [21556077.013237] 262128 pages RAM
Feb 1 15:48:14 prx2 kernel: [21556077.013239] 7290 pages reserved
Feb 1 15:48:14 prx2 kernel: [21556077.013241] 12823 pages shared
Feb 1 15:48:14 prx2 kernel: [21556077.013242] 240468 pages non-shared
Feb 1 15:48:14 prx2 kernel: [21556077.013244] [ pid ] uid tgid total_vm
rss cpu oom_adj oom_score_adj name
Feb 1 15:48:14 prx2 kernel: [21556077.013258] [ 394] 0 394 4905
228 0 -17 -1000 udevd
Feb 1 15:48:14 prx2 kernel: [21556077.013262] [ 408] 0 408 5802
150 2 0 0 systemd-stdout-
Feb 1 15:48:14 prx2 kernel: [21556077.013267] [ 612] 0 612 22825
205 0 -17 -1000 auditd
Feb 1 15:48:14 prx2 kernel: [21556077.013270] [ 661] 0 661 7428
214 1 0 0 systemd-logind
Feb 1 15:48:14 prx2 kernel: [21556077.013274] [ 688] 0 688 29598
195 3 0 0 crond
Feb 1 15:48:14 prx2 kernel: [21556077.013277] [ 689] 81 689 7453
207 0 -13 -900 dbus-daemon
Feb 1 15:48:14 prx2 kernel: [21556077.013281] [ 706] 0 706 17954
232 3 0 0 login
Feb 1 15:48:14 prx2 kernel: [21556077.013285] [ 715] 0 715 4904
98 0 -17 -1000 udevd
Feb 1 15:48:14 prx2 kernel: [21556077.013288] [ 716] 0 716 4904
97 1 -17 -1000 udevd
Feb 1 15:48:14 prx2 kernel: [21556077.013291] [ 741] 0 741 18861
218 3 -17 -1000 sshd
Feb 1 15:48:14 prx2 kernel: [21556077.013295] [ 749] 0 749 29097
207 0 0 0 bash
Feb 1 15:48:14 prx2 kernel: [21556077.013410] [ 1745] 38 1745 7670
252 0 0 0 ntpd
Feb 1 15:48:14 prx2 kernel: [21556077.013413] [ 1934] 0 1934 38932
180 3 0 0 /usr/sbin/munin
Feb 1 15:48:14 prx2 kernel: [21556077.013417] [ 2261] 997 2261 10282
151 0 0 0 nrpe
Feb 1 15:48:14 prx2 kernel: [21556077.013420] [10061] 995 10061 11423
128 2 0 0 haproxy
Feb 1 15:48:14 prx2 kernel: [21556077.013424] [17766] 0 17766 20733
94 2 0 0 sendmail
Feb 1 15:48:14 prx2 kernel: [21556077.013427] [17778] 51 17778 19627
74 0 0 0 sendmail
Feb 1 15:48:14 prx2 kernel: [21556077.013430] [10250] 995 10250 11138
130 2 0 0 haproxy
Feb 1 15:48:14 prx2 kernel: [21556077.013433] [32530] 0 32530 82406
202 2 0 0 rsyslogd
Feb 1 15:48:14 prx2 kernel: [21556077.013436] [ 9654] 995 9654 16113
368 0 0 0 haproxy
Feb 1 15:48:14 prx2 kernel: [21556077.013440] [15556] 0 15556 13754
1993 1 0 0 heartbeat
Feb 1 15:48:14 prx2 kernel: [21556077.013443] [15559] 0 15559 13720
1959 0 0 0 heartbeat
Feb 1 15:48:14 prx2 kernel: [21556077.013446] [15560] 0 15560 13719
1958 0 0 0 heartbeat
Feb 1 15:48:14 prx2 kernel: [21556077.013449] [15561] 0 15561 13719
1958 2 0 0 heartbeat
Feb 1 15:48:14 prx2 kernel: [21556077.013453] [15562] 0 15562 13719
1958 3 0 0 heartbeat
Feb 1 15:48:14 prx2 kernel: [21556077.013456] [15563] 0 15563 13719
1958 1 0 0 heartbeat
Feb 1 15:48:14 prx2 kernel: [21556077.013461] Out of memory: Kill process
9654 (haproxy) score 13 or sacrifice child
Feb 1 15:48:14 prx2 kernel: [21556077.013740] Killed process 9654
(haproxy) total-vm:64452kB, anon-rss:940kB, file-rss:532kB
Feb 1 15:53:33 prx2 systemd-logind[661]: New session 42318 of user root.
Feb 1 15:54:31 prx2 haproxy[3694]: Stopping haproxy: [ OK ]
