Hi
A few days ago one of our machines logged this:
Sep 10 10:54:29 web8 kernel: haproxy: page allocation failure. order:1,
mode:0x20
Sep 10 10:54:29 web8 kernel: Pid: 3196, comm: haproxy Not tainted
2.6.32-358.el6.x86_64 #1
Sep 10 10:54:29 web8 kernel: Call Trace:
Sep 10 10:54:29 web8 kernel: <IRQ> [<ffffffff8112c127>] ?
__alloc_pages_nodemask+0x757/0x8d0
Sep 10 10:54:29 web8 kernel: [<ffffffff81013783>] ? native_sched_clock+0x13/0x80
Sep 10 10:54:29 web8 kernel: [<ffffffff8144cfdf>] ? netif_rx+0xaf/0x160
Sep 10 10:54:29 web8 kernel: [<ffffffff811669d2>] ? kmem_getpages+0x62/0x170
Sep 10 10:54:29 web8 kernel: [<ffffffff811675ea>] ? fallback_alloc+0x1ba/0x270
Sep 10 10:54:29 web8 kernel: [<ffffffff8116703f>] ? cache_grow+0x2cf/0x320
Sep 10 10:54:29 web8 kernel: [<ffffffff81167369>] ?
____cache_alloc_node+0x99/0x160
Sep 10 10:54:29 web8 kernel: [<ffffffff811682eb>] ? kmem_cache_alloc+0x11b/0x190
Sep 10 10:54:29 web8 kernel: [<ffffffff81439b38>] ? sk_prot_alloc+0x48/0x1c0
Sep 10 10:54:29 web8 kernel: [<ffffffff8143ac12>] ? sk_clone+0x22/0x2e0
Sep 10 10:54:29 web8 kernel: [<ffffffff81489ae6>] ? inet_csk_clone+0x16/0xd0
Sep 10 10:54:29 web8 kernel: [<ffffffff814a29f3>] ?
tcp_create_openreq_child+0x23/0x450
Sep 10 10:54:29 web8 kernel: [<ffffffff814a01ed>] ?
tcp_v4_syn_recv_sock+0x4d/0x310
Sep 10 10:54:29 web8 kernel: [<ffffffff814a2796>] ? tcp_check_req+0x226/0x460
Sep 10 10:54:29 web8 kernel: [<ffffffff8149fc8b>] ? tcp_v4_do_rcv+0x35b/0x430
Sep 10 10:54:29 web8 kernel: [<ffffffff814a149e>] ? tcp_v4_rcv+0x4fe/0x8d0
Sep 10 10:54:29 web8 kernel: [<ffffffff8147f1b0>] ?
ip_local_deliver_finish+0x0/0x2d0
Sep 10 10:54:29 web8 kernel: [<ffffffff8147f28d>] ?
ip_local_deliver_finish+0xdd/0x2d0
Sep 10 10:54:29 web8 kernel: [<ffffffff8147f518>] ? ip_local_deliver+0x98/0xa0
Sep 10 10:54:29 web8 kernel: [<ffffffff8147e9dd>] ? ip_rcv_finish+0x12d/0x440
Sep 10 10:54:29 web8 kernel: [<ffffffff8147ef65>] ? ip_rcv+0x275/0x350
Sep 10 10:54:29 web8 kernel: [<ffffffff8144819b>] ?
__netif_receive_skb+0x4ab/0x750
Sep 10 10:54:29 web8 kernel: [<ffffffff814484da>] ? process_backlog+0x9a/0x100
Sep 10 10:54:29 web8 kernel: [<ffffffff8144cd43>] ? net_rx_action+0x103/0x2f0
Sep 10 10:54:29 web8 kernel: [<ffffffff81076fb1>] ? __do_softirq+0xc1/0x1e0
Sep 10 10:54:29 web8 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
Sep 10 10:54:29 web8 kernel: <EOI> [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
Sep 10 10:54:29 web8 kernel: [<ffffffff810778fa>] ? local_bh_enable_ip+0x9a/0xb0
Sep 10 10:54:29 web8 kernel: [<ffffffff815101ab>] ? _spin_unlock_bh+0x1b/0x20
Sep 10 10:54:29 web8 kernel: [<ffffffff81438c7e>] ? release_sock+0xce/0xe0
Sep 10 10:54:29 web8 kernel: [<ffffffff814b0748>] ?
inet_stream_connect+0x68/0x2c0
Sep 10 10:54:29 web8 kernel: [<ffffffff81436227>] ? sys_connect+0xd7/0xf0
Sep 10 10:54:29 web8 kernel: [<ffffffff81017888>] ?
syscall_trace_enter+0x1d8/0x1e0
Sep 10 10:54:29 web8 kernel: [<ffffffff8100b288>] ? tracesys+0xd9/0xde
It's from a RHEL machine (a Dell R610 with 48GB ECC ram - with no logged memory
errors) running 2.6.32-358.el6.x86_64 and HAProxy 1.4.22 with the following
build options:
jedc@web8:/home/jedc>$ /usr/local/sbin/haproxy -vv
HA-Proxy version 1.4.22 2012/08/09
Copyright 2000-2012 Willy Tarreau <w...@1wt.eu>
Build options :
TARGET = linux26
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing
OPTIONS = USE_LINUX_SPLICE=1 USE_REGPARM=1 USE_PCRE=1 USE_STATIC_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Available polling systems :
sepoll : pref=400, test result OK
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 4 (4 usable), will use sepoll.
Our logs show plenty of traffic up until the moment the trace was logged, and
plenty of traffic after - no restart of change of PID, so my question is:
Should I be worried?
An upgrade to 1.4.24 is planned Real Soon(TM), but I am unsure if it's a known
error that's fixed in a later version.
Regards,
Jens Dueholm Christensen
Survey IT
