Haproxy - Large number of Req Resp Errors, 504 bad gateway
Hi Guys, I have 2 different HAProxy setups 1xLB, 2xWebServers 1xDB. The first of the setups has been running for 2+years, the second has been running for 2-3 weeks. Both have recently had a number of users are reporting 504 bad gateway errors from multiple isp's. All I have noticed is that In the HAProxy stats pages for both, a number of Req Resp Errors are showing up. They seem to slowly build up and up over the hours. I am not too sure where to start for tracking down these Req Resp errors the 504 issues for that matter and if the two are related. Any tips? Syd
LB Layout Question
Hi There, I've setup a few small load balanced environments with haproxy usually 2 LB's, 2+ webservers, 1 db server. However, I now have a client who needs the above but with an aditional file storage server for user uploads. So I'm arranging for an extra dedicated server with several TB that will be on private network with the 2 webservers. The client uses a custom coded CMS which allows for a path to be specified for an upload folder for user file storage. Any simple advice for the best method to connect a file server to the web servers? I'm guessing an an NFS share from the 2 webservers to the 1 fileserver. However, from a bit of research with load balanced magento setups there seems to be a lot of negative comments about using NFS in this way.
Re: External Monitoring of https on LB's
Hi Willy BAptiste, I've been running stunnel-4.44 already patched with xforwarded-for-diff from that link. How should I set the listenqueue param? via stunnel.conf? Out of the detailed logs available from HyperSpin, they have 20 or so servers which connect to test at random. The same 3 servers connect fine, and the same 17 servers fail. Which makes me think its some strange difference between those servers that work, and don't work and my environment, rather than reaching some limit or something. I have verbose logging enabled (debug = 7), however it doesn't seem to be logging anything to /var/log/stunnel.log so i'll have to look into that. Cheers, Syd Hi, I could suspect something else. Did you patch your stunnel ? By default it has a very tiny listen queue of only 5 entries which can cause exactly this issue if there is even a moderate load on it. A patch to change this is available here if you want : http://www.exceliance.fr/download/free/patches/stunnel/ It adds a listenqueue parameter allowing you to increase the backlog. I would really not be surprised if this was the issue. Regards, Willy
Re: External Monitoring of https on LB's
Hi Willy, The only listen-queue patch is for 4.20 - if im running stunnel 4.44 with the 4.44-xforwarded-for patch, can i use this? Quoting Willy Tarreau w...@1wt.eu: On Mon, Aug 27, 2012 at 04:04:52PM +1000, s...@summerwinter.com wrote: Hi Willy BAptiste, I've been running stunnel-4.44 already patched with xforwarded-for-diff from that link. How should I set the listenqueue param? via stunnel.conf? yes, but once you've applied the listen-queue patch from the link. Out of the detailed logs available from HyperSpin, they have 20 or so servers which connect to test at random. The same 3 servers connect fine, and the same 17 servers fail. Which makes me think its some strange difference between those servers that work, and don't work and my environment, rather than reaching some limit or something. Interesting. It may also be some of their servers not using the same network path as others. You should enable network captures for their IPs so that you see what's happening. Regards, Willy
External Monitoring of https on LB's
Hi there, Forgive me if this is the wrong place for advice, but I figure a lot of people here must use a similar setup. I've got 2 LB's setup with haproxy, heartbeat stunnel. Http https is working correctly. I am using HyperSpin.com for external monitoring to receive alerts based on ping, http https on the float IP. Ping http work without issue. However, 75% of there 20 or so global monitoring servers appear return errors 'couldn't connect to port 443', so every 10-15minutes a server that can't connect on 443 tests it, fails and my inbox fills. There is no firewall on the LB, nothing I can tell that would be blocking access to 443. I've received the following logs from HyperSpin on a server that is unable to connect: - We do not know the cause of the problem, but we can confirm it is a SSL issue. We logged to our Singapore server and tried using curl and wget to access your website. Both returned errors. === [admin@sg ~]$ curl https://floatip curl: (35) Unknown SSL protocol error in connection to floatip:443 [admin@sg ~]$ wget -O - https://floatip --21:44:16-- https://floatip = `-' Connecting to floatip:443... connected. Unable to establish SSL connection. === I thought it may be an issue with the intermediate certificate, but I have tacked that on at the end of the ssl.crt file i'm using. Any ideas?
