Ok - I finally solved this, and I'd like to offer it up as a suggestion for another resolution to the LVS-NAT RealServers as Clients notes. While this does route all traffic back through the director, it is an entirely generic solution so it could work for as many VIPs and RIPs as necessary. As a fringe benefit and result of going round-about via your default gateway through the director, all connections follow the same load balancing you've established for the system. This might be important if you have a lot of these cases. This setup also provides the added benefit that realservers can communicate to anywhere for outbound connections the normal network gateway.
My work is largely inspired by Jonathan and Karen's blog post here: http://developingrapids.blogspot.com/2006/05/conning-mark-multiwan-connections.html I'm describing redhat/centos based configs, slight variations may be required on other systems that I'm not familiar with. The assumption is that you have 2 internal VLANs, one for "management" traffic and one for LVS traffic. I'll call those: InternalNet 192.168.1.x LVSNet 192.168.2.x The LVS director is 192.168.2.100 On realservers: - Use your normal default gateway for your network, NOT the LVS Director, eg: 192.168.1.1 - Use mangle table prerouting firewall marks to mark any LVS traffic to fwmark 2 - Use mangle table prerouting firewall marks to mark any non-LVS traffic to fwmark 1 (may be redundant) - Use ip rules to direct any traffic received with fwmark 2 via the LVSNet table (192.168.2.100 gateway) - Use ip rules to direct any traffic received on 192.168.2.x via the LVSNet table (192.168.2.100 gateway - may be redundant) - Use ip rules to direct any traffic received with fwmark 1 via the InternalNet table (192.168.1.1 gateway - may be redundant to the default route) - Use ip rules to direct any traffic received on 192.168.1.x via the InternalNet table (192.168.1.1 gateway - may be redundant) Configuration: =========== 1) Add two tables to your /etc/iproute2/rt_tables 200 InternalNet 201 LVSNet 2) Set your default gateway to 192.168.1.1 (/etc/sysconfig/network) GATEWAY=192.168.1.1 3) In your iptables mangle table (for real server 192.168.1.112), add the following. In different realservers, replace the IP address 192.168.1.112 with that of the current real server: -A PREROUTING -j CONNMARK --restore-mark -A PREROUTING -m mark --mark 0x1 -j ACCEPT -A PREROUTING -d 192.168.1.112 -j MARK --set-mark 0x1 -A PREROUTING -m mark --mark 0x2 -j ACCEPT -A PREROUTING -d 192.168.2.112 -j MARK --set-mark 0x2 -A PREROUTING -j CONNMARK --save-mark 4) Setup IP routes and rules (for each real server you change the IP address in the rules below to that of the real server IPs) routes: /etc/sysconfig/network-scripts/route-eth1: default table InternalNet via 192.168.1.1 default table LVSNet via 192.168.2.100 rules: /etc/sysconfig/network-scripts/rule-eth1 from 192.168.1.112 table InternalNet from 192.168.2.112 table LVSNet fwmark 1 lookup InternalNet fwmark 2 lookup LVSNet ------ DONE ---- Net result is that LVS-NAT based inbound connections route back out through the LVS Director appropriately, even though your default gateway is on 192.168.1.1. All other real server requests route through your normal LAN default gateway. If that outbound request is to one of your LVS VIPs (real server as client), the connection goes via your normal LAN to the front end of the LVS director, and the reply comes back the same path. --------------------- Verify configuration: You end up with: [r...@rack112 ~]# route eth1 Link encap:Ethernet inet addr:192.168.1.112 Bcast:192.168.1.255 Mask:255.255.255.0 eth1:1 Link encap:Ethernet inet addr:192.168.2.112 Bcast:192.168.2.255 Mask:255.255.255.0 [r...@rack112 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth1 [r...@rack112 ~]# ip rule show 0: from all lookup 255 32759: from all fwmark 0x2 lookup LVSNet 32760: from all fwmark 0x1 lookup InternalNet 32761: from 192.168.2.112 lookup LVSNet 32762: from 192.168.1.112 lookup InternalNet 32763: from all fwmark 0x2 lookup main 32766: from all lookup main 32767: from all lookup default Best, James _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - [email protected] Send requests to [email protected] or go to http://lists.graemef.net/mailman/listinfo/lvs-users
