Hi all,
I've just installed my first loadbalancing setup and it works perfectly
but for 1 strange (to me, hopefully not to you) error. The ldirectord
keeps sending me e-mails that a real server is inaccessible, however, I
believe it's trying to check the fallback server and failing. The setup
is as follows (pardon for the amount of info but better too much than
too little because I have no idea where my problem stems from):
loadbalancer is lb1, web nodes are web1 and web2, all are debian lenny
5.0.1 with all latest updates. Following is the setup as complete as
possible:
---
lb1
---
lo 127.0.0.1/8
eth0 192.168.0.3/24
eth0:0 192.168.0.6/24
lb1:~# netstat -tulpen
tcp 0 0 192.168.0.3:80 0.0.0.0:*
LISTEN 0 64025 16923/apache2
tcp 0 0 192.168.0.6:80 0.0.0.0:*
LISTEN 0 64023 16923/apache2
tcp 0 0 127.0.0.1:80 0.0.0.0:*
LISTEN 0 64021 16923/apache2
My full /etc/ha.d/ldirectord.cf:
checktimeout = 10
checkinterval = 2
autoreload = yes
logfile = "local0"
emailalert = "ee...@procolix.com"
emailalertfreq = 60
quiescent = no
virtual=192.168.0.6:80
service = http
real = 192.168.0.4:80 gate 500
real = 192.168.0.5:80 gate 500
fallback = 127.0.0.1;80 gate
request = "alive"
receive = "yes"
scheduler = wlc
protocol = tcp
checkport = 80
checktype = negotiate
lb1:~# dpkg -s ldirectord
Package: ldirectord
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 276
Maintainer: Simon Horman <ho...@debian.org>
Architecture: all
Source: heartbeat
Version: 2.1.3-6lenny1
Depends: perl, ipvsadm, libauthen-radius-perl, libcrypt-ssleay-perl,
libdbi-perl, libdigest-hmac-perl, libdigest-md5-perl,
libmail-pop3client-perl, libmailtools-perl, libnet-dns-perl,
libnet-imap-simple-perl, libnet-imap-simple-ssl-perl, libnet-ldap-perl,
libnet-perl, libwww-perl
Recommends: rsyslog | system-log-daemon, logrotate
Conflicts: libpils0 (<< 2.0.8-3), libstonith0 (<< 2.0.8-3), stonith (<<
2.0.8-3)
Conffiles:
/etc/init.d/ldirectord afbe41a5bd19f9ed8c770b6e2bd38771
/etc/default/ldirectord 9b553aebc0d11d5cc551ed98dfd4d8a3
/etc/logrotate.d/ldirectord ff7728efb8bc46c6e66da07ac5474928
Description: Monitors virtual services provided by LVS
ldirectord is a stand-alone daemon to monitor services of real
for virtual services provided by The Linux Virtual Server (LVS).
and works with Linux-HA's heartbeat package.
Python-Version: current
----
web1
----
eth0 192.168.0.4/24
lo:0 192.168.0.6/32
sysctl -A output:
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
----
web2
----
eth0 192.168.0.5/24
lo:0 192.168.0.6/32
sysctl -A output:
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
lb1:~# /etc/init.d/ldirectord start
From the logs:
Jun 25 10:50:05 lb1 ldirectord[21699]: Invoking ldirectord invoked as:
/usr/sbin/ldirectord start
Jun 25 10:50:05 lb1 ldirectord[21699]: Starting Linux Director
v1.186-ha-2.1.3 as daemon
Jun 25 10:50:05 lb1 ldirectord[21701]: Added virtual server: 192.168.0.6:80
Jun 25 10:50:05 lb1 ldirectord[21701]: Added fallback server:
127.0.0.1:80 (192.168.0.6:80) (Weight set to 1)
Jun 25 10:50:05 lb1 ldirectord[21701]: emailalert: Added fallback
server: 127.0.0.1:80 (192.168.0.6:80) (Weight set to 1)
Jun 25 10:50:05 lb1 ldirectord[21701]: Added real server: 192.168.0.4:80
(192.168.0.6:80) (Weight set to 500)
Jun 25 10:50:05 lb1 ldirectord[21701]: emailalert: Added real server:
192.168.0.4:80 (192.168.0.6:80) (Weight set to 500)
Jun 25 10:50:05 lb1 ldirectord[21701]: Deleted fallback server:
127.0.0.1:80 (192.168.0.6:80)
Jun 25 10:50:05 lb1 ldirectord[21701]: emailalert: Deleted fallback
server: 127.0.0.1:80 (192.168.0.6:80)
Jun 25 10:50:05 lb1 ldirectord[21701]: Added real server: 192.168.0.5:80
(192.168.0.6:80) (Weight set to 500)
Jun 25 10:50:05 lb1 ldirectord[21701]: emailalert: Added real server:
192.168.0.5:80 (192.168.0.6:80) (Weight set to 500)
So everything is fine, the real servers are added and are reachable, the
fallback server is subsequently deleted.
lb1:~# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.6:80 wlc
-> 192.168.0.5:80 Route 500 0 2
-> 192.168.0.4:80 Route 500 0 2
lb1:~# ipvsadm -Lnc
IPVS connection entries
pro expire state source virtual destination
TCP 01:35 FIN_WAIT 192.168.0.101:1810 192.168.0.6:80 192.168.0.4:80
TCP 01:35 FIN_WAIT 192.168.0.101:1808 192.168.0.6:80 192.168.0.5:80
TCP 01:34 FIN_WAIT 192.168.0.101:1806 192.168.0.6:80 192.168.0.4:80
TCP 01:34 FIN_WAIT 192.168.0.101:1804 192.168.0.6:80 192.168.0.5:80
So everything is working, my connections are beautifully balanced and
all is OK, I'm receiving my http pages from the 2 web nodes.
However, now my logs start filling up with the actual problem i'm having:
Jun 25 10:51:05 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 127.0.0.1:80 (192.168.0.6:80)
Jun 25 10:52:06 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 127.0.0.1:80 (192.168.0.6:80)
Jun 25 10:53:07 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 127.0.0.1:80 (192.168.0.6:80)
Jun 25 10:54:07 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 127.0.0.1:80 (192.168.0.6:80)
Jun 25 10:55:08 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 127.0.0.1:80 (192.168.0.6:80)
Jun 25 10:56:09 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 127.0.0.1:80 (192.168.0.6:80)
Jun 25 10:57:09 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 127.0.0.1:80 (192.168.0.6:80)
Jun 25 10:58:10 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 127.0.0.1:80 (192.168.0.6:80)
It seems like the ldirectord is trying to somehow check the fallback
server on 127.0.0.1 and not having any result for that. I've checked the
apache logs on the fallback server but i don't see any access in my
webserver logs on lb1 so it's not 'really' checking 127.0.0.1:80, at
least, not that I can tell on the webserver side of things.
Removing all real servers brings the fallback server online and the
fallback server works correctly (i receive my error page from the
fallback when i go to the VIP). After this the erroneous e-mail alerts
disappear, the log on lb1 now reads:
Jun 25 11:11:52 lb1 ldirectord[21701]: Deleted real server:
192.168.0.4:80 (192.168.0.6:80)
Jun 25 11:11:52 lb1 ldirectord[21701]: emailalert: Deleted real server:
192.168.0.4:80 (192.168.0.6:80)
Jun 25 11:12:19 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 127.0.0.1:80 (192.168.0.6:80)
Jun 25 11:12:23 lb1 ldirectord[21701]: Deleted real server:
192.168.0.5:80 (192.168.0.6:80)
Jun 25 11:12:23 lb1 ldirectord[21701]: emailalert: Deleted real server:
192.168.0.5:80 (192.168.0.6:80)
Jun 25 11:12:23 lb1 ldirectord[21701]: Added fallback server:
127.0.0.1:80 (192.168.0.6:80) (Weight set to 1)
Jun 25 11:12:23 lb1 ldirectord[21701]: emailalert: Added fallback
server: 127.0.0.1:80 (192.168.0.6:80) (Weight set to 1)
Jun 25 11:12:53 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 192.168.0.4:80 (192.168.0.6:80)
Jun 25 11:13:23 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 192.168.0.5:80 (192.168.0.6:80)
Jun 25 11:13:54 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 192.168.0.4:80 (192.168.0.6:80)
Jun 25 11:14:24 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 192.168.0.5:80 (192.168.0.6:80)
Jun 25 11:14:54 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 192.168.0.4:80 (192.168.0.6:80)
Jun 25 11:15:25 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 192.168.0.5:80 (192.168.0.6:80)
Jun 25 11:15:55 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 192.168.0.4:80 (192.168.0.6:80)
Jun 25 11:16:25 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 192.168.0.5:80 (192.168.0.6:80)
Jun 25 11:16:56 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 192.168.0.4:80 (192.168.0.6:80)
Jun 25 11:17:26 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 192.168.0.5:80 (192.168.0.6:80)
So the behaviour of ldirectord is now correct, it's sending me alerts
about the real servers being offline, but the alerts about 127.0.0.1
being offline disappear. After re-adding a real server however the
fallback get's removed and the errors start again:
Jun 25 11:21:09 lb1 ldirectord[21701]: Added real server: 192.168.0.5:80
(192.168.0.6:80) (Weight set to 500)
Jun 25 11:21:09 lb1 ldirectord[21701]: emailalert: Added real server:
192.168.0.5:80 (192.168.0.6:80) (Weight set to 500)
Jun 25 11:21:09 lb1 ldirectord[21701]: Deleted fallback server:
127.0.0.1:80 (192.168.0.6:80)
Jun 25 11:21:09 lb1 ldirectord[21701]: emailalert: Deleted fallback
server: 127.0.0.1:80 (192.168.0.6:80)
Jun 25 11:21:59 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 192.168.0.4:80 (192.168.0.6:80)
Jun 25 11:22:09 lb1 ldirectord[21701]: emailalert: Inaccessible real
server: 127.0.0.1:80 (192.168.0.6:80)
I hope anyone can shed some light on this because I'm at a loss as to
why this is happening.
Thanks in advance.
Best regards,
Eelco Jepkema
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-requ...@linuxvirtualserver.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
