Re: Page allocation failure
Hi Jens, On Mon, Oct 07, 2013 at 11:23:38AM +, Jens Dueholm Christensen wrote: A bit of followup (again sorry for topposting).. I began looking around for the first line in the stacktrace __alloc_pages_nodemask+0x757/0x8d0 and found a thread in the Linux-Kernel mailinglist: http://lkml.indiana.edu/hypermail/linux/kernel/1305.3/01761.html. Good catch! This thread left me wondering a bit, and since I'm not a C[++]-programmer, I've never dealt with handling memory allocation and the intricies it involves, so I began wondering what and how the order was affecting memory allocation.. The order indicates the log of the number of consecutive pages the system tried to allocate at once. So order 0 is 4kB, order 1 is 8kB, order 2 is 16kB etc... All the errors I've seen in our logs are logged as order-1 failures, and as far as I can understand an order-1 allocation error is not necessarily a dead end. That said it's rare. It could mean that your memory is highly fragmented due to other workloads on the same machine. According to https://www.kernel.org/doc/gorman/html/understand/understand009.html there should be a fallback to a lower-order allocation when a higher-order allocation is requested and fails. That's indeed sometimes possible (I don't know the exact conditions for this to happen). That said, the worst that can happen in your case is that some outgoing packets are dropped and will have to be retransmitted. If it happens once in a while it might be OK, but it's not pleasant to have a system log full of errors. Did you try to report the issue to the distro ? It's important to report bugs, as bugs not reported to not exist. According to the same Linux-Kernel thread some networkdrivers are buggy, and since this machine contains 6 Broadcom NetXtreme II BCM5709 1000Base-T interfaces I am currently looking into upgrading the driver. Are you running with jumbo frames or is your MTU set to 1500 ? I used to experience allocation failures with jumbo frames in the past on the e1000 driver, it could be a similar issue here. Maybe your driver allocates send buffers by large chunks, I don't know. Regards, Willy
RE: Page allocation failure
A bit of followup (again sorry for topposting).. I began looking around for the first line in the stacktrace __alloc_pages_nodemask+0x757/0x8d0 and found a thread in the Linux-Kernel mailinglist: http://lkml.indiana.edu/hypermail/linux/kernel/1305.3/01761.html. This thread left me wondering a bit, and since I'm not a C[++]-programmer, I've never dealt with handling memory allocation and the intricies it involves, so I began wondering what and how the order was affecting memory allocation.. All the errors I've seen in our logs are logged as order-1 failures, and as far as I can understand an order-1 allocation error is not necessarily a dead end. According to https://www.kernel.org/doc/gorman/html/understand/understand009.html there should be a fallback to a lower-order allocation when a higher-order allocation is requested and fails. According to the same Linux-Kernel thread some networkdrivers are buggy, and since this machine contains 6 Broadcom NetXtreme II BCM5709 1000Base-T interfaces I am currently looking into upgrading the driver. Regards, Jens Dueholm Christensen Survey IT -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Sunday, September 15, 2013 8:57 AM To: Jens Dueholm Christensen Cc: haproxy@formilux.org Subject: Re: Page allocation failure On Fri, Sep 13, 2013 at 11:47:56AM +, Jens Dueholm Christensen wrote: Hi, sorry for topposting, but Outlook is notoriously bad at inlining.. In reply to Lukas Tribus: Uptime is some 200 days jedc@web8:/home/jedc$ free -m total used free sharedbuffers cached Mem: 48253 47843410 0 2747 24841 -/+ buffers/cache: 20254 27998 Swap:32767368 32399 Acording to RHEL network, the kernel I'm running now is the latest available, so upgrading is not really an option at this time. Alas a reboot is not really an option right now (downtime scheduling, announcement etc etc..), but I will prioritize it in our plans. And better not reboot before the Red Hat support collects what the need to feed the report. Willy
Re: Page allocation failure
Hi Jens, On Fri, Sep 13, 2013 at 12:05:24PM +, Jens Dueholm Christensen wrote: Hi, sorry for topposting, but Outlook is notoriously bad at inlining.. :-) I've used it at a customer's and it's possible to make it correctly prefix existing lines with a character such as '|' or ''. After you do that, it becomes usable again. In reply to Willy Tarreau: Our sysctl settings are in this pastebin (it's a mix of sysctl.conf settings, and some commandes called in a custom iptables script): http://pastebin.com/kZDP8XuM Have you got any recommendations regarding them? No, these settings are fine and clean and should not cause any issue. When the error occurred we had a normal load, but as this normal load does involve loads of heavy (CPU and DB I/O from a remote SQL server) threads in a largeish Java VM, I can't really tell if that might have been a contributing factor. Possible, this must never be ruled out. Our MRTG graphs for memory, CPU and load at the time shows nothing unexpected, so I guess I'll report it to RedHat and see if they can give me any pointers or a fix. Yes at the moment I really think that's the best thing to do. I would have suggested to you to report it on the LKML if it were a mainline kernel, but it's not so you'd be thrown away. Best regards, Willy
RE: Page allocation failure
Hi, sorry for topposting, but Outlook is notoriously bad at inlining.. In reply to Lukas Tribus: Uptime is some 200 days jedc@web8:/home/jedc$ free -m total used free sharedbuffers cached Mem: 48253 47843410 0 2747 24841 -/+ buffers/cache: 20254 27998 Swap:32767368 32399 Acording to RHEL network, the kernel I'm running now is the latest available, so upgrading is not really an option at this time. Alas a reboot is not really an option right now (downtime scheduling, announcement etc etc..), but I will prioritize it in our plans. Med venlig hilsen, Jens Dueholm Christensen Survey IT -Original Message- From: Lukas Tribus [mailto:luky...@hotmail.com] Sent: Thursday, September 12, 2013 3:50 PM To: Jens Dueholm Christensen; haproxy@formilux.org Subject: RE: Page allocation failure Hi! A few days ago one of our machines logged this: Sep 10 10:54:29 web8 kernel: haproxy: page allocation failure. order:1, mode:0x20 The kernel has problems allocating memory to haproxy. Since we don't see the OOM killer in action, I guess your memory is heavily fragmented. I guess this box has a long uptime? How much free RAM does free -m show? Should I be worried? Memory allocation failure will lead to application failures. I would take this seriously. An upgrade to 1.4.24 is planned Real Soon(TM), but I am unsure if it's a known error that's fixed in a later version. Upgrading to 1.4.24 is important, there are several issues with 1.4.22. However, it will not fix this problem, as this is not a bug in haproxy. The problem mostly depends on your kernel. A quick fix is to reboot the box, which will make the problem go away for now. I suggest upgrading OS/kernel to a more recent version. You can read more about this here: http://www.cyberciti.biz/faq/linux-page-allocation-failure-erro/ Regards, Lukas
RE: Page allocation failure
Hi, sorry for topposting, but Outlook is notoriously bad at inlining.. In reply to Willy Tarreau: Our sysctl settings are in this pastebin (it's a mix of sysctl.conf settings, and some commandes called in a custom iptables script): http://pastebin.com/kZDP8XuM Have you got any recommendations regarding them? When the error occurred we had a normal load, but as this normal load does involve loads of heavy (CPU and DB I/O from a remote SQL server) threads in a largeish Java VM, I can't really tell if that might have been a contributing factor. Our MRTG graphs for memory, CPU and load at the time shows nothing unexpected, so I guess I'll report it to RedHat and see if they can give me any pointers or a fix. Thanks! Regards, Jens Dueholm Christensen Survey IT -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Thursday, September 12, 2013 10:21 PM To: Lukas Tribus Cc: Jens Dueholm Christensen; haproxy@formilux.org Subject: Re: Page allocation failure On Thu, Sep 12, 2013 at 03:49:33PM +0200, Lukas Tribus wrote: Hi! A few days ago one of our machines logged this: Sep 10 10:54:29 web8 kernel: haproxy: page allocation failure. order:1, mode:0x20 The kernel has problems allocating memory to haproxy. Since we don't see the OOM killer in action, I guess your memory is heavily fragmented. It is possible the problem is even deeper. haproxy called connect(), which happened to enable irqs and was suddenly interrupted by a softirq pertaining to a pending incoming ACK packet completing a pending incoming connection. The TCP receive path was called from the softirq to create a real connection from a connection request, and at this exact point failed a memory allocation to receive a packet. I guess this box has a long uptime? How much free RAM does free -m show? Also please check your sysctl.conf in case you would have changed some of them based on the advices from random sites (we often find wrong settings causing 4096 times too much RAM being allocated to the network stack). And you should check if you were not running some backups or anything I/O intensive at the same time, as it could fill the RAM with a lot of cached data if the tuning is not that good. If you didn't do it and still have some RAM, I'd suggest reporting that to Red Hat who may be interested in investigating this issue as it could be very specific to their kernel. Should I be worried? Memory allocation failure will lead to application failures. I would take this seriously. In this case it did not hit haproxy (otherwise it would have been killed). However one incoming connection was destroyed and we don't know why. So yes it could be a very serious issue. An upgrade to 1.4.24 is planned Real Soon(TM), but I am unsure if it?s a known error that?s fixed in a later version. Upgrading to 1.4.24 is important, there are several issues with 1.4.22. However, it will not fix this problem, as this is not a bug in haproxy. The problem mostly depends on your kernel. Clearly. A quick fix is to reboot the box, which will make the problem go away for now. I suggest upgrading OS/kernel to a more recent version. I believe that 2.6.32-358 is a reasonably recent one, though I may be wrong. Cheers, Willy
RE: Page allocation failure
Hi! A few days ago one of our machines logged this: Sep 10 10:54:29 web8 kernel: haproxy: page allocation failure. order:1, mode:0x20 The kernel has problems allocating memory to haproxy. Since we don't see the OOM killer in action, I guess your memory is heavily fragmented. I guess this box has a long uptime? How much free RAM does free -m show? Should I be worried? Memory allocation failure will lead to application failures. I would take this seriously. An upgrade to 1.4.24 is planned Real Soon(TM), but I am unsure if it’s a known error that’s fixed in a later version. Upgrading to 1.4.24 is important, there are several issues with 1.4.22. However, it will not fix this problem, as this is not a bug in haproxy. The problem mostly depends on your kernel. A quick fix is to reboot the box, which will make the problem go away for now. I suggest upgrading OS/kernel to a more recent version. You can read more about this here: http://www.cyberciti.biz/faq/linux-page-allocation-failure-erro/ Regards, Lukas
Re: Page allocation failure
On Thu, Sep 12, 2013 at 03:49:33PM +0200, Lukas Tribus wrote: Hi! A few days ago one of our machines logged this: Sep 10 10:54:29 web8 kernel: haproxy: page allocation failure. order:1, mode:0x20 The kernel has problems allocating memory to haproxy. Since we don't see the OOM killer in action, I guess your memory is heavily fragmented. It is possible the problem is even deeper. haproxy called connect(), which happened to enable irqs and was suddenly interrupted by a softirq pertaining to a pending incoming ACK packet completing a pending incoming connection. The TCP receive path was called from the softirq to create a real connection from a connection request, and at this exact point failed a memory allocation to receive a packet. I guess this box has a long uptime? How much free RAM does free -m show? Also please check your sysctl.conf in case you would have changed some of them based on the advices from random sites (we often find wrong settings causing 4096 times too much RAM being allocated to the network stack). And you should check if you were not running some backups or anything I/O intensive at the same time, as it could fill the RAM with a lot of cached data if the tuning is not that good. If you didn't do it and still have some RAM, I'd suggest reporting that to Red Hat who may be interested in investigating this issue as it could be very specific to their kernel. Should I be worried? Memory allocation failure will lead to application failures. I would take this seriously. In this case it did not hit haproxy (otherwise it would have been killed). However one incoming connection was destroyed and we don't know why. So yes it could be a very serious issue. An upgrade to 1.4.24 is planned Real Soon(TM), but I am unsure if it?s a known error that?s fixed in a later version. Upgrading to 1.4.24 is important, there are several issues with 1.4.22. However, it will not fix this problem, as this is not a bug in haproxy. The problem mostly depends on your kernel. Clearly. A quick fix is to reboot the box, which will make the problem go away for now. I suggest upgrading OS/kernel to a more recent version. I believe that 2.6.32-358 is a reasonably recent one, though I may be wrong. Cheers, Willy
