Hi LVS List! IPVS (on XEN) with two Bridges fails.
We have the following setup: A Xen Machine with one external and one internal physical interfaces peth0 and peth2. The dom0 is connected via eth0 to the internet and via eth2 to a local subnet. On this local subnet a machine Vzope1 acts as one realserver. internet -> peth0 -|> Xen server -|> peth2 -> switch -> realserver We use the bridged Xen Setup with multible Bridges (in our case two). Incoming Requests were received from the external interface peth0 and sent via eth2 to a realserver. This ist setup in more detail: World | XEN-Dom0 running ipvs | Request -> peth0 -> xenbr0 -|> eth0 -> ipvs -> eth2 -> xenbr2 -|> peth2 -> physical switch -> realserver (Vzope1) * First the Request-Package enters peth0 the pysical interface of the XEN-Server. * The Bridge xenbr0 then delivers the Package to eth0 which is in the Scope of the Virtual-Server running ipvs. * According to http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/images/nf-lvs.png ipvs takes the packet out of the LOCAL_IN Chain Here the commented tcpdump: peth0: 02:14:09.948636 IP dslb-088-074-161-106.pools.arcor-ip.net.35322 > 193.239.28.158.www: S 2531256350:2531256350(0) win 5840 <mss 1452,sackOK,timestamp 14032446 0,nop,wscale 7> xenbr0: 02:14:09.948653 IP dslb-088-074-161-106.pools.arcor-ip.net.35322 > 193.239.28.158.www: S 2531256350:2531256350(0) win 5840 <mss 1452,sackOK,timestamp 14032446 0,nop,wscale 7> eth0: 02:14:09.948653 IP dslb-088-074-161-106.pools.arcor-ip.net.35322 > 193.239.28.158.www: S 2531256350:2531256350(0) win 5840 <mss 1452,sackOK,timestamp 14032446 0,nop,wscale 7> ipvs: DNAT performed succesfully eth2: 02:14:09.948670 IP dslb-088-074-161-106.pools.arcor-ip.net.35322 > 10.1.0.124.www: S 2531256350:2531256350(0) win 5840 <mss 1452,sackOK,timestamp 14032446 0,nop,wscale 7> xenbr2: 02:14:09.948670 IP dslb-088-074-161-106.pools.arcor-ip.net.35322 > 10.1.0.124.www: S 2531256350:2531256350(0) win 5840 <mss 1452,sackOK,timestamp 14032446 0,nop,wscale 7> peth2: 02:14:09.948684 IP dslb-088-074-161-106.pools.arcor-ip.net.35322 > 10.1.0.124.www: S 2531256350:2531256350(0) win 5840 <mss 1452,sackOK,timestamp 14032446 0,nop,wscale 7> Going out to realserver: All ok this way. Let's have a look at the response packets: World | Server running ipvs | Request <- peth0 <- xenbr0 <- eth0 <- ipvs <- eth2 <- xenbr2 <- peth2 <- realserver We Received a Syn Ack Packet from the realserver at peth2: 02:14:09.948885 IP 10.1.0.124.www > dslb-088-074-161-106.pools.arcor-ip.net.35322: S 1725501155:1725501155(0) ack 2531256351 win 5792 <mss 1460,sackOK,timestamp 956956465 14030196,nop,wscale 7> Then it undergoes SNAT by ipvs to be delivered to xenbr2: 02:14:09.948898 IP 193.239.28.158.www > dslb-088-074-161-106.pools.arcor-ip.net.35322: S 1725501155:1725501155(0) ack 2531256351 win 5792 <mss 1460,sackOK,timestamp 956956465 14030196,nop,wscale 7> hitting eth2: 02:14:09.948898 IP 193.239.28.158.www > dslb-088-074-161-106.pools.arcor-ip.net.35322: S 1725501155:1725501155(0) ack 2531256351 win 5792 <mss 1460,sackOK,timestamp 956956465 14030196,nop,wscale 7> The response goes the folowing way Realserver -> peth2 -> xenbr2 -> ipvs SNAT -> xenbr2 -> eth2 -> Martian source resulting in something like Jan 22 20:46:31 charyptis kernel: martian source 88.74.158.52 from 193.239.28.158, on dev eth2 Jan 22 20:46:31 charyptis kernel: ll header: 00:e0:81:49:aa:af:00:16:3e:76:ff:ec:08:00 We do ipvs for a long time. So please expect us not to make beginners mistakes. We checked : * the interfaces * the bridges * the routes * the netfilter We do not have a single problem. Every thing runs fine. We reproduced this tests on a second XEN machine with exactily the same results. We read the dokumentation. We checked for typical pitfalls like tcp checksum error under XEN. We read the code of ipvs. We read to the discussion about "route_me_harder()" We suspect the problem the area of "route_me_harder()" since we stumbled over the following comment in route_me_harder(): /* some non-standard hacks like ipt_REJECT.c:send_reset() can cause * packets with foreign saddr to appear on the NF_IP_LOCAL_OUT hook. */ How can we track the problem further? May anyone be capable to confirm our findings? Best Regards Volker -- ==================================================== inqbus it-consulting +49 ( 341 ) 5643800 Dr. Volker Jaenisch http://www.inqbus.de Herloßsohnstr. 12 0 4 1 5 5 Leipzig N O T - F Ä L L E +49 ( 170 ) 3113748 ==================================================== _______________________________________________ LinuxVirtualServer.org mailing list - [email protected] Send requests to [EMAIL PROTECTED] or go to http://lists.graemef.net/mailman/listinfo/lvs-users
