Hi there, Ok I disabled selinux and increased check inter to 30s. I enabled an http check of an asphx file because ASP is critical to the operation of the site. It was already there but I disabled it earlier because of the problems we were having: option httpchk HEAD /testip.ashx HTTP/1.1\r\nHost:\ www.oursite.com
With regards to free, I'm ashamed to say that yes I did go after the first line. I also did a yum upgrade but will postpone 1.4rc1 until I see how this change responds. Will put the LB back online when the traffic is not that heavy as I cannot risk another outage and hence my job :) Will post a reply tomorrow afternoon. Thank you so much you've been great. On 7 February 2010 02:06, Hank A. Paulson <[email protected]>wrote: > You have selinux on, so it may be unhappy with some part of haproxy - the > directory it uses, the socket listeners, etc. Turn it off (if you can) until > you get everything working ok. Turning it off requires a reboot. > > To see if it is on: > # sestatus > google for how to turn it off > > I would back off the check inter to 30s or so and make it an http check of > a file that you know exists, if you can have any static files on your > servers. This will allow you to see that haproxy is able to find that file, > get a 200 response and verify that the server is up. > > Also, when you say "free mem going down to 45Mb" are you looking at the > first line of "free" or the second line? Ignore the first line, it is > designed to cause panic. eg: > > $ free -m > total used free shared buffers cached > Mem: 32244 32069 174 0 0 19578 > -/+ buffers/cache: 12490 19753 > Swap: 4095 0 4095 > > OMG, I only have 174MB of my 32GB of memory available!?! > - no, really 19.75 GB is still available. > > On your haproxy config, if you log errors separately then you can tail -f > that error-only log and watch it as you start up haproxy. And why not do > http logging if you are doing http mode? Maybe I am missing something. > > I would back off the check inter to 30s or so and make it an http check of > a file that you know exists, if you can have any static files on your > servers. This will allow you to see that haproxy is able to find that file, > get a 200 response and verify that the server is really is up and responding > fully, not just opening a socket. If you can switch to 1.4rc1 then you get > alot more info about the health check/health status on the stats page and > you can do set log-health-checks as an addition aid to troubleshooting. > > > global > log 127.0.0.1 local0 > log 127.0.0.1 local1 notice > #log loghost local0 info > option log-separate-errors > > maxconn 4096 > chroot /var/lib/haproxy > user haproxy > group haproxy > daemon > # debug > #quiet > > defaults > log global > mode http > # option httplog > option dontlognull > retries 3 > option redispatch > maxconn 4096 > contimeout 5s > clitimeout 30s > srvtimeout 30s > > > listen loadbalancer :80 > mode http > balance roundrobin > option forwardfor except 10.0.1.50 > option httpclose > option httplog > option httpchk HEAD /favicon.ico > > cookie SERVERID insert indirect nocache > server WEB01 10.0.1.108:80 cookie A check inter 30s > server WEB05 10.0.1.109:80 cookie B check inter 30s > > > listen statistics 10.0.1.50:8080 > stats enable > stats auth stats:stats > stats uri / > > [BTW, Did you do a yum upgrade - not yum update after your install of F12?, > "yum update" misses certain kinds of packaging changes, "yum upgrade" covers > all updates, even if the name of a package changes - yum upgrade should be > the default used in yum examples - I ask because many people don't do this > and there are many security fixes and other package bug fixes that have been > posted] > > > On 2/6/10 6:59 AM, Peter Griffin wrote: > >> Hi Will, >> Yes X-Windows is installed, but the default init is runlevel 3 and I >> have not started X for the past couple of days. The video card is an >> addon card so I rule out shared memory. >> >> With regards to eth1 I ran iptraf and can see that there is no traffic >> on eth1 so I'd rule this out as well. I thought about listening for >> stunnel requests on eth1 10.0.1.51 and connecting to haproxy on >> 10.0.1.50, but maybe this will cause more problems... >> I had already ftp'd a file some 70MB to another machine on the same Vlan >> and I did not see any problems whatsoever. What I'm planning to do now >> is to setup the LB in another environment with another 2 Web servers and >> 1 DB server and stress the hell out of it. Then I can also test the >> network traffic using Iperf. >> Will report back in a few days, thank you once more. >> >> >> >> >> On 6 February 2010 14:29, Willy Tarreau <[email protected] <mailto:[email protected]>> >> wrote: >> >> On Sat, Feb 06, 2010 at 01:16:00PM +0100, Peter Griffin wrote: >> > Both http & https. Also both web servers started to take it in >> turns to >> > report as DOWN but more frequently the second one than the first. >> > >> > I ran ethtool eth0 and can verify that it's full-duplex 1Gbps: >> >> OK. >> >> > I'm attaching dmesg, I don't understand most of it. >> >> well, it shows some video driver issues, which are unrelated (did you >> start a graphics environment on your LB ?). It seems it's reserving >> some memory (64 or 512MB, I don't understand well) for the video. I >> hope it's not a card with shared memory, as the higher the resolution, >> the lower the remaining memory bandwidth for normal work. >> >> But I don't see any iptables related issue there, so that's fine. >> >> Stupid question, are you sure that your traffic passes via eth0 (the >> gig one) ? I'm asking, because eth1 is a cheap 100 Mbps realtek 8139, >> and if you got the routing wrong, it could explain a lot of networking >> issues ! >> >> > I'll try to send a file >> > in both directions to saturate the link as you suggested. >> >> OK. >> >> When doing that, don't bench the disks, just the network. For that, >> create "sparse files", which are empty files for which the kernel >> produces zeroes on the fly, and send them files to /dev/null. Eg >> with ftp : >> >> machine1$ dd if=/dev/null bs=1M count=0 seek=1024 of=1g.bin >> >> machine2$ ftp machine1 >> > recv 1g.bin /dev/null >> >> >> Regards, >> Willy >> >> >> >
