Wow, i'm going to try all this and tell you tomorrow how did it go. You're a genious by the way Thank you a lot
L.S. Keijser-2 wrote: > > Hi, > > I'm replying to you and to the lvs-users list so everyone can > participate and help. It's a fairly long reply with detailed > instructions on how to get it working. I could just point you to the > HOWTO (as i already did) but i'm in a good mood. Also, since it's so > long and detailed, i hope i got it right else it's a waste of time :P > > Oh and i'm going to make a lot of assumptions while replying, so please > correct me if i'm wrong: > > On Thu, 2009-10-22 at 12:38 -0700, [email protected] wrote: >> Thank you Léon for the reply. I apreciate it so much. >> Yes i actually have 3 public ip's like (none are with 192..) >> XX.XX.XX.234 (this is the lvs..) > > by 'the lvs' i assume you mean 'the director' > >> XX.XX.XX.235 real server (web ngix) >> YYY.YYY.YYY.163 real server (web apache) - I don't really care about >> this one, i can move it into the > > into the .. ? cat caught your tongue? :P > >> we will use XX.XX.XX.236 as virtual..(there isn't an ip on the net with >> that number up). of course it will be simpler to use a 192.. but, i have >> tried that also ,and no luck > > I assume that by 'virtual' you mean Virtual IP configured on the > director. > >> same subnet as the first 2 ones, i just want to make it work from the 234 >> -> 235..but it gives me a timeout on the browser... > > You mean from .236 -> .235 because clients won't connect to the IP of > the director. Instead they'll connect to the VIP. > >> Here's what i did: >> >> [r...@linux ~]# cat /etc/ha.d/ldirectord.cf >> checktimeout=3 >> checkinterval=10 >> autoreload=yes >> logfile="/var/log/ldirectord.log" >> quiescent=no > > I'm not sure 'quiescent=no' is a valid option in ldirectord.cf .. What > were you trying to accomplish here? > >> virtual=XX.XX.XX.236:80 >> fallback=127.0.0.1:80 >> real=XX.XX.XX.235:80 gate >> real=YYY.YYY.YYY.235:80 gate > > Where does the YYY.YYY.YYY.235 come from? Assuming X != Y, this will > never work as the two realservers are in different subnets. Again i'm > assuming this because of your ifconfig post later in this mail. Either > move the 2nd realserver into the subnet, leave it out, or extend your > subnet to include it (probably not possible). > >> service=http >> request="test.html" >> receive="Still alive" >> scheduler=rr #here i've tried with172.18.24.15 wlr as well > > The scheduler is irrelevant for now. Let's just first try to get it > working period. Anyway, 'wlr'? I'm not even sure this is a valid > scheduler. Don't you mean 'wlc' or 'wrr'? :) > >> protocol=tcp >> checktype=negotiate >> >> [r...@linux ~]# /usr/sbin/ldirectord -d /etc/ha.d/ldirectord.cf start > > It's better (to learn LVS) if you don't use ldirectord for now, but ok, > let's try it anyway :P > > -snip heartbeat output- > >> If i go to the webserver i can see that ldirector is actually testing the >> test.html..every 10 seconds like in the conf > > Good, at least the director can access one of the realservers. > >> I am sure that is because of the configurations of the IPs and the >> aditional eth0:0 and lo:0 and that's why it doesn't work, i will paste >> everything that i did, maybe, just maybe you can help me out on this one, >> i'm really stuck..probably because i don't know lots of stuff on how the >> OSI layer is build and how arp works > > Okay, tiny summary of what's supposed to happen in LVS-DR: > > 1) client sends request to VIP > 2) director receives packet, checks LVS table for available realserver > and forwards the packet to it without changing destination_ip > 3) realserver receives packet and sees destination_ip matches the > configured ip on its loopback device > 4) realserver handles request and replies to src_address (the original > client) thereby using its default gateway, bypassing the director > 5) client receives reply > > What probably happened is that you didn't solve the ARP problem and > instead of receiving a reply from the IP configured on the realserver's > loopback device (that's the VIP, the same one configured on the > director), the realserver's RIP replies to the client. Your client never > sent a request to _that_ IP so it drops the packet, endlessly waiting > for a reply from the VIP. > >> Aditional Network conf.. >> XX.XX.XX.234 (this is the lvs..) is spawned on eth1.. >> so i spawned another eth1:0 >> [r...@linux ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1:0 >> DEVICE=eth1:0 >> IPADDR=XX.XX.XX.236 # this is from the same subnet and i doesn't >> conflict with an existent one >> NETMASK=255.255.255.0 >> NETWORK=XX.XX.XX.0 >> BROADCAST=XX.XX.XX.255 >> GATEWAY=XX.XX.XX.233 # i've put the same gateway as the default ip >> XX.XX.XX.234 had >> ONBOOT=yes >> >> i also enabled port forwarding > > Port forwarding isn't required for LVS-DR. > >> net.ipv4.ip_forward = 1 >> net.ipv4.conf.default.rp_filter = 1 >> net.ipv4.conf.default.accept_source_route = 0 >> kernel.sysrq = 0 >> kernel.core_uses_pid = 1 >> net.ipv4.tcp_syncookies = 1 >> kernel.msgmnb = 65536 >> kernel.msgmax = 65536 >> kernel.shmmax = 68719476736 >> kernel.shmall = 4294967296 >> >> And i did the modprobe with all those modules for IPV >> >> modprobe ip_vs_dh >> modprobe ip_vs_ftp >> modprobe ip_vs_dh >> modprobe ip_vs_ftp >> modprobe ip_vs >> modprobe ip_vs_lblc >> modprobe ip_vs_lblcr >> modprobe ip_vs_lc >> modprobe ip_vs_nq >> modprobe ip_vs_rr >> modprobe ip_vs_sed >> modprobe ip_vs_sh >> modprobe ip_vs_wlc >> modprobe ip_vs_wrr > > IPVS will load the module it requires for the configured scheduler > automatically. It's not necessary to load any module manually. > >> and that's all i did for the LVS server.. >> >> Now for the webserver >> XX.XX.XX.235 (this is the lvs..) is spawned on eth0.. >> r...@linux ~]# cat /etc/sysconfig/network-scripts/ifcfg-lo:0 >> DEVICE=lo:0 >> IPADDR=XX.XX.XX.236 >> NETMASK=255.255.255.255 >> NETWORK=XX.XX.XX.XX.0 >> BROADCAST=XX.XX.XX.255 >> ONBOOT=yes >> NAME=loopback > > AFAIK it's best not to use the sysconfig scripts to create the loopback > device on the realserver. RedHat does an ARPING to determine if the > interface you're trying to configure is already up. It might confuse > things. You could just as easily configure everything > from /etc/rc.local. See below the next bit: > >> and >> >> net.ipv4.ip_forward = 0 >> net.ipv4.conf.lo.arp_ignore = 1 #here i have tried with eth0 instead of >> lo, no luck.. >> net.ipv4.conf.lo.arp_announce = 2 #here i have tried with eth0 instead >> of lo, no luck.. >> net.ipv4.conf.all.arp_ignore = 1 >> net.ipv4.conf.all.arp_announce = 2 > > Okay, you're starting okay in trying to solve the ARP problem. But > you're really not... Configure arp ignore/announce and the loopback > device from within /etc/rc.local like this: > > # solve the 'ARP problem' > echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore > echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore > echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce > echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce > /sbin/ifconfig lo:0 XX.XX.XX.236 netmask 255.255.255.255 up > > (reboot your realserver after this, or bring down lo:0 and > run /etc/rc.local) > >> net.ipv4.conf.default.rp_filter = 1 >> net.ipv4.conf.default.accept_source_route = 0 >> kernel.sysrq = 0 >> kernel.core_uses_pid = 1 >> net.ipv4.tcp_syncookies = 1 >> kernel.msgmnb = 65536 >> kernel.msgmax = 65536 >> kernel.shmmax = 68719476736 >> kernel.shmall = 4294967296 >> >> here i've tried with ip forward 0 and 1 , no luck the requests simply >> don't reach this server only the direct ones > > You don't configure ip_forward on the realservers. > >> >> Now i've understand that this is an ARP problem, and as CentOS doesn't >> support the arp hidden flag on sysctl , i tried with >> /etc/init.d/arptables_jf > > -snip arptables output- > > Again, assuming you have a fairly recent kernel, you don't need > arptables. Just for fun, please post your kernel version. > >> ----------------------------------------------- >> >> i've probed with ipvsadm or something like it, to see the active >> connections, and they are always to 0 >> >> i've modprobed here the same modules, no luck.. >> >> So from this point i'm really stuck and don't know what to do... >> >> Here's the ifconfig from both servers if that helps >> from the LVS (xx.234) >> >> eth1 Link encap:Ethernet HWaddr 00:1B:21:46:3E:A9 >> inet addr:XX.XX.XX.234 Bcast:XX.XX.XX.239 >> Mask:255.255.255.248 > > Okay, this pretty much rules out the possibility that both your > realservers are in the same subnet, so fix that first by either removing > one that isn't in the same subnet as the director, or by moving it into > the subnet. > >> eth1:0 Link encap:Ethernet HWaddr 00:1B:21:46:3E:A9 >> inet addr:XX.XX.XX.236 Bcast:XX.XX.XX.255 Mask:255.255.255.0 > > Your configured VIP is in a different subnet than your real IP. Probably > doesn't matter much in this case though, but still.. > > -snip rest of output- > >> here's the one from the webserver >> >> eth0 Link encap:Ethernet HWaddr 00:24:1D:72:61:AB >> inet addr:XX.XX.XX.235 Bcast:XX.XX.XX.239 >> Mask:255.255.255.248 > > -snip rest of eth0 and lo output- > >> lo:0 Link encap:Local Loopback >> inet addr:XX.XX.XX.236 Mask:255.255.255.255 >> UP LOOPBACK RUNNING MTU:16436 Metric:1 > > Good, it has the VIP configured. > > So after all these changes/checks/reboots, run this command from a > client (not the director or any of the realservers!) : > > $ arping XX.XX.XX.236 > > You should get a reply from 00:1B:21:46:3E:A9 (MAC on the director). If > you get any reply from 00:24:1D:72:61:AB (MAC on the realserver) you > haven't solved the ARP problem yet. Assuming (here we go again) that you > have now configured the lo:0 on the realserver correctly (and removed it > from /etc/sysconfig/network-scripts/ifcfg-lo:0 !!), clear the arp cache > on the client: > > $ arp -d XX.XX.XX.236 > > and try the arping command again. > > Also, disable any firewalls on both director and realservers while > setting things up. You can always later lock it down (while keeping an > eye on functionality). This way it won't disturb setting up LVS. > > And if you rather not use ldirectord (recommended for first-time usage): > > $ service ldirectord stop > $ ipvsadm -A -t XX.XX.XX.236:80 -s rr > $ ipvsadm -a -t XX.XX.XX.236:80 -r XX.XX.XX.235:80 -g -w 1 > > Then from a client (not the director or realserver) try it out: > > $ telnet XX.XX.XX.236 80 > > If you get a response like this: > > Trying XX.XX.XX.236... > Connected to XX.XX.XX.236. > Escape character is '^]'. > > Then it's working. > > Good luck :) > > -- > Léon > > > _______________________________________________ > Please read the documentation before posting - it's available at: > http://www.linuxvirtualserver.org/ > > LinuxVirtualServer.org mailing list - [email protected] > Send requests to [email protected] > or go to http://lists.graemef.net/mailman/listinfo/lvs-users > -- View this message in context: http://www.nabble.com/LdirectorD-LVS-and-CentOS-Fedora-RedHat-tp26004219p26018186.html Sent from the LVS mailing list archive at Nabble.com. _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - [email protected] Send requests to [email protected] or go to http://lists.graemef.net/mailman/listinfo/lvs-users
