You have selinux on, so it may be unhappy with some part of haproxy - the
directory it uses, the socket listeners, etc. Turn it off (if you can) until
you get everything working ok. Turning it off requires a reboot.
To see if it is on:
# sestatus
google for how to turn it off
I would back off the check inter to 30s or so and make it an http check of a
file that you know exists, if you can have any static files on your servers.
This will allow you to see that haproxy is able to find that file, get a 200
response and verify that the server is up.
Also, when you say "free mem going down to 45Mb" are you looking at the first
line of "free" or the second line? Ignore the first line, it is designed to
cause panic. eg:
$ free -m
total used free shared buffers cached
Mem: 32244 32069 174 0 0 19578
-/+ buffers/cache: 12490 19753
Swap: 4095 0 4095
OMG, I only have 174MB of my 32GB of memory available!?!
- no, really 19.75 GB is still available.
On your haproxy config, if you log errors separately then you can tail -f that
error-only log and watch it as you start up haproxy. And why not do http
logging if you are doing http mode? Maybe I am missing something.
I would back off the check inter to 30s or so and make it an http check of a
file that you know exists, if you can have any static files on your servers.
This will allow you to see that haproxy is able to find that file, get a 200
response and verify that the server is really is up and responding fully, not
just opening a socket. If you can switch to 1.4rc1 then you get alot more info
about the health check/health status on the stats page and you can do set
log-health-checks as an addition aid to troubleshooting.
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
#log loghost local0 info
option log-separate-errors
maxconn 4096
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
# debug
#quiet
defaults
log global
mode http
# option httplog
option dontlognull
retries 3
option redispatch
maxconn 4096
contimeout 5s
clitimeout 30s
srvtimeout 30s
listen loadbalancer :80
mode http
balance roundrobin
option forwardfor except 10.0.1.50
option httpclose
option httplog
option httpchk HEAD /favicon.ico
cookie SERVERID insert indirect nocache
server WEB01 10.0.1.108:80 cookie A check inter 30s
server WEB05 10.0.1.109:80 cookie B check inter 30s
listen statistics 10.0.1.50:8080
stats enable
stats auth stats:stats
stats uri /
[BTW, Did you do a yum upgrade - not yum update after your install of F12?,
"yum update" misses certain kinds of packaging changes, "yum upgrade" covers
all updates, even if the name of a package changes - yum upgrade should be the
default used in yum examples - I ask because many people don't do this and
there are many security fixes and other package bug fixes that have been posted]
On 2/6/10 6:59 AM, Peter Griffin wrote:
Hi Will,
Yes X-Windows is installed, but the default init is runlevel 3 and I
have not started X for the past couple of days. The video card is an
addon card so I rule out shared memory.
With regards to eth1 I ran iptraf and can see that there is no traffic
on eth1 so I'd rule this out as well. I thought about listening for
stunnel requests on eth1 10.0.1.51 and connecting to haproxy on
10.0.1.50, but maybe this will cause more problems...
I had already ftp'd a file some 70MB to another machine on the same Vlan
and I did not see any problems whatsoever. What I'm planning to do now
is to setup the LB in another environment with another 2 Web servers and
1 DB server and stress the hell out of it. Then I can also test the
network traffic using Iperf.
Will report back in a few days, thank you once more.
On 6 February 2010 14:29, Willy Tarreau <[email protected] <mailto:[email protected]>>
wrote:
On Sat, Feb 06, 2010 at 01:16:00PM +0100, Peter Griffin wrote:
> Both http & https. Also both web servers started to take it in
turns to
> report as DOWN but more frequently the second one than the first.
>
> I ran ethtool eth0 and can verify that it's full-duplex 1Gbps:
OK.
> I'm attaching dmesg, I don't understand most of it.
well, it shows some video driver issues, which are unrelated (did you
start a graphics environment on your LB ?). It seems it's reserving
some memory (64 or 512MB, I don't understand well) for the video. I
hope it's not a card with shared memory, as the higher the resolution,
the lower the remaining memory bandwidth for normal work.
But I don't see any iptables related issue there, so that's fine.
Stupid question, are you sure that your traffic passes via eth0 (the
gig one) ? I'm asking, because eth1 is a cheap 100 Mbps realtek 8139,
and if you got the routing wrong, it could explain a lot of networking
issues !
> I'll try to send a file
> in both directions to saturate the link as you suggested.
OK.
When doing that, don't bench the disks, just the network. For that,
create "sparse files", which are empty files for which the kernel
produces zeroes on the fly, and send them files to /dev/null. Eg
with ftp :
machine1$ dd if=/dev/null bs=1M count=0 seek=1024 of=1g.bin
machine2$ ftp machine1
> recv 1g.bin /dev/null
Regards,
Willy
