RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs
Thanks Martin. My rp_filter was disabled on server box B: # cat /proc/sys/net/ipv4/conf/all/rp_filter 0 I got it to work however, along the lines you were suggesting like this: On Box A: iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 on box B, I finally found ACK packets. I do not know why I could not see them before. It turns out that box B was trying to route ACKs via its WAN interface eth0 directly to the client, in some sort of asymmetric router fashion. This did not work. So on B, I routed ACKs back to A: #ip rule add from $BOX_B_ETH1 lookup 3 #ip route add default via $BOX_A_ETH1 table 3 Now it works, with DNAT on A and without anything else. I have 1 small follow-up question: 1) when I add custom rules like #ip rule add from $BOX_B_ETH1 lookup 3 it does not take effect for at least 1 minute, perhaps 2-3. Why is that? This is confusing, since it makes one think that the rule does not work, while in reality it just has not taken effect. Thanks a lot for your help, Alec. -Original Message- From: Martin A. Brown [mailto:[EMAIL PROTECTED] Sent: Thursday, March 08, 2007 8:40 PM To: Alec Matusis Cc: lartc@mailman.ds9a.nl Subject: RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Alec, : I tried implementing DNAT as you indicated: : : iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 : : After that, I can see SYN packets arriving on BOX_B_ETH1, having : the original client's IP. OK, that means DNAT + routing on your BOX_A is working correctly. : Only half of the connection gets established after this: I cannot : see ACK packets from box B anywhere (neither on BOX_B_ETH0, nor : on BOX_A_ETH0). This is where your problem lies now. You need to find out why the SYN (which you said was transmitted to BOX_B_ETH1) is not getting accepted by this IP stack. * reverse path filtering (net.ipv4.conf.*.rp_filter) * packet filtering rules on BOX_B? : I think the reason is that since box A never sends a SYN packet : itself, it never classifies the connection as ESTABLISHED, so all : further traffic gets rejected. It's still a mystery to me what : happens to SYN packets from be in this scenario however. BOX_A will never have a socket in ESTABLISHED state. BOX_A will have a state entry in the /proc/net/ip_conntrack table. Examine /proc/net/ip_conntrack after sending a SYN to BOX_B. : It turns out that I have to supplement DNAT with SNAT for this to work : correctly. : On box A: : iptables -t nat -i eth0 -p tcp -m tcp --dport $SERVER_PORT -j DNAT : --to-destination $BOX_B_ETH1 : iptables -t nat -A POSTROUTING -d $BOX_B_ETH1 -p tcp -m tcp --dport : $SERVER_PORT -j SNAT --to-source $BOX_A_ETH1 If this works, then I think you problem may be reverse path filtering. : in this case, the clients can connect, however the server on B : sees only IP of BOX_A_ETH1, not the original client IPs. [ tproxy recommendation snipped ] - -Martin - -- Martin A. Brown http://linux-ip.net/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/) iD8DBQFF8OURHEoZD1iZ+YcRAoenAJ9XCZyMf4K7TVCTs28bzIGeu3EEewCg07Cw Spk8a+T/th+ESyPN4hSTjYs= =k+5E -END PGP SIGNATURE- ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello, : 1) when I add custom rules like : #ip rule add from $BOX_B_ETH1 lookup 3 : it does not take effect for at least 1 minute, perhaps 2-3. : : Why is that? It's quite simple. There is a routing cache. Recently used routes are stored here for faster access. If you would like to empty the routing cache, then you must make your changes to the routing tables and then empty the routing cache: ip route flush cache Be very careful not to omit the word cache. You will have a nice little surprise if you forget the word cache. : This is confusing, since it makes one think that the rule does : not work, while in reality it just has not taken effect. If you'd like to view the route cache: ip route show cache Do you want to see a particular entry? ip route show cache 72.14.203.104 - -Martin - -- Martin A. Brown http://linux-ip.net/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/) iD8DBQFF8YC/HEoZD1iZ+YcRAkfFAJ9zYzVRCVMTGE619avs4hZVe+yi2gCgtGRi iCnX/HpQS3PiGIvlaJi7nlo= =S+EQ -END PGP SIGNATURE- ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs
I have not considered bridge and ebtables yet. I assume I'd have to implement this in the routing box A. Is ebtables CPU-intensive? The goal of setting up this forwarding is to reduce the load on the router box A. Also, is it very surprising that there is no way to build a transparent proxy/router with standard iproute2 and iptables tools ONLY? It would seem that transparent forwarding of TCP connections to another machine is a very common task. From: Robb Bossley [mailto:[EMAIL PROTECTED] Sent: Thursday, March 08, 2007 9:13 AM To: Alec Matusis Subject: Re: [LARTC] routing TCP to another box preserving ORIGINAL client IPs Have you considered putting a bridge in and using ebtables? That might be what you are looking for. On 3/7/07, Alec Matusis alecm at chatango.com wrote: Hello Martin, I tried implementing DNAT as you indicated: iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 After that, I can see SYN packets arriving on BOX_B_ETH1, having the original client's IP. Only half of the connection gets established after this: I cannot see ACK packets from box B anywhere (neither on BOX_B_ETH0, nor on BOX_A_ETH0). I think the reason is that since box A never sends a SYN packet itself, it never classifies the connection as ESTABLISHED, so all further traffic gets rejected. It's still a mystery to me what happens to SYN packets from be in this scenario however. It turns out that I have to supplement DNAT with SNAT for this to work correctly. On box A: iptables -t nat -i eth0 -p tcp -m tcp --dport $SERVER_PORT -j DNAT --to-destination $BOX_B_ETH1 iptables -t nat -A POSTROUTING -d $BOX_B_ETH1 -p tcp -m tcp --dport $SERVER_PORT -j SNAT --to-source $BOX_A_ETH1 in this case, the clients can connect, however the server on B sees only IP of BOX_A_ETH1, not the original client IPs. Regarding tproxy: I am currently running the TCP server on box A. I would like to move it to box B to reduce the load on A. Other services on A are bound to the same IP address as the server that I need to move, so simply moving that IP address to BOX_B_ETH0 is impossible. Box A has 2.6.11 kernel. Does tproxy install over netfilter? I am sort of scared to tamper with netfilter installation on A, since it is currently running live services. It it possible to implement this scenario: * the client thinks it's connected to Box A * Box A knows its connected to the client * Box A uses client's source address to initiate traffic to Box B * Box B thinks it's connected to client with advanced routing and iptables only? Thanks Alec. -Original Message- From: Martin A. Brown [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 07, 2007 8:24 PM To: Alec Matusis Cc: lartc@mailman.ds9a.nl Subject: Re: [LARTC] routing TCP to another box preserving ORIGINAL client IPs -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Greetings Alec, : My TCP clients connect to box A. I need to forward those : connections to a server on box B, such that the original client : IPs are visible to the server on B. : : Each box has two Ethernet ports. One port on each box is : connected to WAN, and they are cross-connected in a LAN via : remaining ports: : : --- --- : WAN -- |eth0 Box A eth1|---LAN---|eth1 Box B eth0| -- WAN : --- --- : : : Is there a way to do this with iproute2 and iptables tools ONLY? : Can you provide an example? Nothing in Google after more than a : week of searching. An additional requirement is to reduce the : load on box A as much as possible (I guess the server on B would : still have to reply to the client via A, not using B's own WAN : interface however..) You need to provide us a bit more information to help you figure out the right way to solve this problem. Why is DNAT not sufficient? Given your description, you should simply be able to: iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 If it were that simple, though, you shouldn't have spent a week looking for the answer. Do you have a TCP service on Box A which is providing services to the client across the WAN? If so, then you are looking for something called transparent proxying in the Linux networking world. You will want to examine the tproxy patches to iptables [0]. If you go with the transparent proxying method, it's helpful to remember: * the client thinks it's connected to Box A * Box A knows its connected to the client * Box A uses client's source address to initiate traffic to Box B * Box B thinks it's connected to client In either case, you are correct about routing. Box B must send its traffic back to Box A to forward back across the LAN. Good luck, - -Martin [0] http://www.balabit.com/products/oss/tproxy/ http://www.balabit.com/downloads/tproxy/linux-2.4/ - -- Martin A. Brown http://linux-ip.net/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment
RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs
Are you sure? Take a look into: http://tldp.org/HOWTO/TransparentProxy-6.html Regards El Vie, 9 de Marzo de 2007, 1:05, Alec Matusis escribió: I have not considered bridge and ebtables yet. I assume I'd have to implement this in the routing box A. Is ebtables CPU-intensive? The goal of setting up this forwarding is to reduce the load on the router box A. Also, is it very surprising that there is no way to build a transparent proxy/router with standard iproute2 and iptables tools ONLY? It would seem that transparent forwarding of TCP connections to another machine is a very common task. From: Robb Bossley [mailto:[EMAIL PROTECTED] Sent: Thursday, March 08, 2007 9:13 AM To: Alec Matusis Subject: Re: [LARTC] routing TCP to another box preserving ORIGINAL client IPs Have you considered putting a bridge in and using ebtables? That might be what you are looking for. On 3/7/07, Alec Matusis alecm at chatango.com wrote: Hello Martin, I tried implementing DNAT as you indicated: iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 After that, I can see SYN packets arriving on BOX_B_ETH1, having the original client's IP. Only half of the connection gets established after this: I cannot see ACK packets from box B anywhere (neither on BOX_B_ETH0, nor on BOX_A_ETH0). I think the reason is that since box A never sends a SYN packet itself, it never classifies the connection as ESTABLISHED, so all further traffic gets rejected. It's still a mystery to me what happens to SYN packets from be in this scenario however. It turns out that I have to supplement DNAT with SNAT for this to work correctly. On box A: iptables -t nat -i eth0 -p tcp -m tcp --dport $SERVER_PORT -j DNAT --to-destination $BOX_B_ETH1 iptables -t nat -A POSTROUTING -d $BOX_B_ETH1 -p tcp -m tcp --dport $SERVER_PORT -j SNAT --to-source $BOX_A_ETH1 in this case, the clients can connect, however the server on B sees only IP of BOX_A_ETH1, not the original client IPs. Regarding tproxy: I am currently running the TCP server on box A. I would like to move it to box B to reduce the load on A. Other services on A are bound to the same IP address as the server that I need to move, so simply moving that IP address to BOX_B_ETH0 is impossible. Box A has 2.6.11 kernel. Does tproxy install over netfilter? I am sort of scared to tamper with netfilter installation on A, since it is currently running live services. It it possible to implement this scenario: * the client thinks it's connected to Box A * Box A knows its connected to the client * Box A uses client's source address to initiate traffic to Box B * Box B thinks it's connected to client with advanced routing and iptables only? Thanks Alec. -Original Message- From: Martin A. Brown [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 07, 2007 8:24 PM To: Alec Matusis Cc: lartc@mailman.ds9a.nl Subject: Re: [LARTC] routing TCP to another box preserving ORIGINAL client IPs -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Greetings Alec, : My TCP clients connect to box A. I need to forward those : connections to a server on box B, such that the original client : IPs are visible to the server on B. : : Each box has two Ethernet ports. One port on each box is : connected to WAN, and they are cross-connected in a LAN via : remaining ports: : : --- --- : WAN -- |eth0 Box A eth1|---LAN---|eth1 Box B eth0| -- WAN : --- --- : : : Is there a way to do this with iproute2 and iptables tools ONLY? : Can you provide an example? Nothing in Google after more than a : week of searching. An additional requirement is to reduce the : load on box A as much as possible (I guess the server on B would : still have to reply to the client via A, not using B's own WAN : interface however..) You need to provide us a bit more information to help you figure out the right way to solve this problem. Why is DNAT not sufficient? Given your description, you should simply be able to: iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 If it were that simple, though, you shouldn't have spent a week looking for the answer. Do you have a TCP service on Box A which is providing services to the client across the WAN? If so, then you are looking for something called transparent proxying in the Linux networking world. You will want to examine the tproxy patches to iptables [0]. If you go with the transparent proxying method, it's helpful to remember: * the client thinks it's connected to Box A * Box A knows its connected to the client * Box A uses client's source address to initiate traffic to Box B * Box B thinks it's connected to client In either case, you are correct about routing. Box B must send its traffic back to Box A to forward back across the LAN. Good luck
RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs
I tried method 2 multiple times for 2 weeks, and it does not work. Perhaps someone can tell me what the problem is? Maybe I am missing some kernel option? How do I check that? Router box A (wan ip x.x.x.83 on eth0) : # iptables -t mangle -L -n Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- 0.0.0.0/0x.x.x.83tcp dpt:5224 MARK set 0x3 Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination #iptables -t filter -L -n Chain INPUT (policy ACCEPT) ... # ip ru l 0: from all lookup local 32765: from all fwmark 0x3 lookup 2 32766: from all lookup main 32767: from all lookup default # ip ro l t 2 default via 10.18.1.1 dev eth1 # cat /proc/sys/net/ipv4/ip_forward 1 checking a few things from A: # telnet 10.18.1.1 5224 Trying 10.18.1.1... Connected to 10.18.1.1 (10.18.1.1). Escape character is '^]'. --success. It means that box B is ready to accept connections on 5224 via LAN (it's INPUT chain's policy in filter is ACCEPT for simplicity). From an outside client Outside client / telnet x.x.x.83 5224 Trying x.x.x.83 ... telnet: connect to address x.x.x.83: Connection refused This suggests that the problem lies within box A. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of ArcosCom Linux User Sent: Thursday, March 08, 2007 4:49 PM To: lartc@mailman.ds9a.nl Subject: RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs Are you sure? Take a look into: http://tldp.org/HOWTO/TransparentProxy-6.html Regards El Vie, 9 de Marzo de 2007, 1:05, Alec Matusis escribió: I have not considered bridge and ebtables yet. I assume I'd have to implement this in the routing box A. Is ebtables CPU-intensive? The goal of setting up this forwarding is to reduce the load on the router box A. Also, is it very surprising that there is no way to build a transparent proxy/router with standard iproute2 and iptables tools ONLY? It would seem that transparent forwarding of TCP connections to another machine is a very common task. From: Robb Bossley [mailto:[EMAIL PROTECTED] Sent: Thursday, March 08, 2007 9:13 AM To: Alec Matusis Subject: Re: [LARTC] routing TCP to another box preserving ORIGINAL client IPs Have you considered putting a bridge in and using ebtables? That might be what you are looking for. On 3/7/07, Alec Matusis alecm at chatango.com wrote: Hello Martin, I tried implementing DNAT as you indicated: iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 After that, I can see SYN packets arriving on BOX_B_ETH1, having the original client's IP. Only half of the connection gets established after this: I cannot see ACK packets from box B anywhere (neither on BOX_B_ETH0, nor on BOX_A_ETH0). I think the reason is that since box A never sends a SYN packet itself, it never classifies the connection as ESTABLISHED, so all further traffic gets rejected. It's still a mystery to me what happens to SYN packets from be in this scenario however. It turns out that I have to supplement DNAT with SNAT for this to work correctly. On box A: iptables -t nat -i eth0 -p tcp -m tcp --dport $SERVER_PORT -j DNAT --to-destination $BOX_B_ETH1 iptables -t nat -A POSTROUTING -d $BOX_B_ETH1 -p tcp -m tcp --dport $SERVER_PORT -j SNAT --to-source $BOX_A_ETH1 in this case, the clients can connect, however the server on B sees only IP of BOX_A_ETH1, not the original client IPs. Regarding tproxy: I am currently running the TCP server on box A. I would like to move it to box B to reduce the load on A. Other services on A are bound to the same IP address as the server that I need to move, so simply moving that IP address to BOX_B_ETH0 is impossible. Box A has 2.6.11 kernel. Does tproxy install over netfilter? I am sort of scared to tamper with netfilter installation on A, since it is currently running live services. It it possible to implement this scenario: * the client thinks it's connected to Box A * Box A knows its connected to the client * Box A uses client's source address to initiate traffic to Box B * Box B thinks it's connected to client with advanced routing and iptables only? Thanks Alec. -Original Message- From: Martin A. Brown [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 07, 2007 8:24 PM To: Alec Matusis Cc: lartc@mailman.ds9a.nl Subject: Re: [LARTC] routing TCP to another box preserving ORIGINAL client IPs -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Greetings Alec, : My TCP clients connect to box A. I need to forward those : connections to a server on box B
RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs
Hi Alex, Perhaps another test to see if it is really working is to connect to the public address (x.x.x.83) from box A. The other thing perhaps you might want to check is if there is a firewall running in between that is blocking 5224. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Alec Matusis Sent: Friday, March 09, 2007 4:13 PM To: [EMAIL PROTECTED]; lartc@mailman.ds9a.nl Subject: RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs I tried method 2 multiple times for 2 weeks, and it does not work. Perhaps someone can tell me what the problem is? Maybe I am missing some kernel option? How do I check that? Router box A (wan ip x.x.x.83 on eth0) : # iptables -t mangle -L -n Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- 0.0.0.0/0x.x.x.83tcp dpt:5224 MARK set 0x3 Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination #iptables -t filter -L -n Chain INPUT (policy ACCEPT) ... # ip ru l 0: from all lookup local 32765: from all fwmark 0x3 lookup 2 32766: from all lookup main 32767: from all lookup default # ip ro l t 2 default via 10.18.1.1 dev eth1 # cat /proc/sys/net/ipv4/ip_forward 1 checking a few things from A: # telnet 10.18.1.1 5224 Trying 10.18.1.1... Connected to 10.18.1.1 (10.18.1.1). Escape character is '^]'. --success. It means that box B is ready to accept connections on 5224 via LAN (it's INPUT chain's policy in filter is ACCEPT for simplicity). From an outside client Outside client / telnet x.x.x.83 5224 Trying x.x.x.83 ... telnet: connect to address x.x.x.83: Connection refused This suggests that the problem lies within box A. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of ArcosCom Linux User Sent: Thursday, March 08, 2007 4:49 PM To: lartc@mailman.ds9a.nl Subject: RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs Are you sure? Take a look into: http://tldp.org/HOWTO/TransparentProxy-6.html Regards El Vie, 9 de Marzo de 2007, 1:05, Alec Matusis escribió: I have not considered bridge and ebtables yet. I assume I'd have to implement this in the routing box A. Is ebtables CPU-intensive? The goal of setting up this forwarding is to reduce the load on the router box A. Also, is it very surprising that there is no way to build a transparent proxy/router with standard iproute2 and iptables tools ONLY? It would seem that transparent forwarding of TCP connections to another machine is a very common task. From: Robb Bossley [mailto:[EMAIL PROTECTED] Sent: Thursday, March 08, 2007 9:13 AM To: Alec Matusis Subject: Re: [LARTC] routing TCP to another box preserving ORIGINAL client IPs Have you considered putting a bridge in and using ebtables? That might be what you are looking for. On 3/7/07, Alec Matusis alecm at chatango.com wrote: Hello Martin, I tried implementing DNAT as you indicated: iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 After that, I can see SYN packets arriving on BOX_B_ETH1, having the original client's IP. Only half of the connection gets established after this: I cannot see ACK packets from box B anywhere (neither on BOX_B_ETH0, nor on BOX_A_ETH0). I think the reason is that since box A never sends a SYN packet itself, it never classifies the connection as ESTABLISHED, so all further traffic gets rejected. It's still a mystery to me what happens to SYN packets from be in this scenario however. It turns out that I have to supplement DNAT with SNAT for this to work correctly. On box A: iptables -t nat -i eth0 -p tcp -m tcp --dport $SERVER_PORT -j DNAT --to-destination $BOX_B_ETH1 iptables -t nat -A POSTROUTING -d $BOX_B_ETH1 -p tcp -m tcp --dport $SERVER_PORT -j SNAT --to-source $BOX_A_ETH1 in this case, the clients can connect, however the server on B sees only IP of BOX_A_ETH1, not the original client IPs. Regarding tproxy: I am currently running the TCP server on box A. I would like to move it to box B to reduce the load on A. Other services on A are bound to the same IP address as the server that I need to move, so simply moving that IP address to BOX_B_ETH0 is impossible. Box A has 2.6.11 kernel. Does tproxy install over netfilter? I am sort of scared to tamper with netfilter installation on A, since it is currently running live services. It it possible to implement this scenario: * the client thinks it's connected to Box A * Box A knows its connected to the client * Box A uses client's source address to initiate traffic to Box B * Box B
RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs
Hi Ranghi: The connection to the public IP x.x.x.83 of box A from box A itself fails as well, just like from a remote client. On A, in table filter, both INPUT and FORWARD policies are ACCEPT. I start suspecting that some options (e.g. IP advanced router) are not compiled into the kernel (2.6.11), but I do not know how to check that? -Original Message- From: Rangi Biddle [mailto:[EMAIL PROTECTED] Sent: Thursday, March 08, 2007 8:32 PM To: 'Alec Matusis'; lartc@mailman.ds9a.nl Subject: RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs Hi Alex, Perhaps another test to see if it is really working is to connect to the public address (x.x.x.83) from box A. The other thing perhaps you might want to check is if there is a firewall running in between that is blocking 5224. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Alec Matusis Sent: Friday, March 09, 2007 4:13 PM To: [EMAIL PROTECTED]; lartc@mailman.ds9a.nl Subject: RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs I tried method 2 multiple times for 2 weeks, and it does not work. Perhaps someone can tell me what the problem is? Maybe I am missing some kernel option? How do I check that? Router box A (wan ip x.x.x.83 on eth0) : # iptables -t mangle -L -n Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- 0.0.0.0/0x.x.x.83tcp dpt:5224 MARK set 0x3 Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination #iptables -t filter -L -n Chain INPUT (policy ACCEPT) ... # ip ru l 0: from all lookup local 32765: from all fwmark 0x3 lookup 2 32766: from all lookup main 32767: from all lookup default # ip ro l t 2 default via 10.18.1.1 dev eth1 # cat /proc/sys/net/ipv4/ip_forward 1 checking a few things from A: # telnet 10.18.1.1 5224 Trying 10.18.1.1... Connected to 10.18.1.1 (10.18.1.1). Escape character is '^]'. --success. It means that box B is ready to accept connections on 5224 via LAN (it's INPUT chain's policy in filter is ACCEPT for simplicity). From an outside client Outside client / telnet x.x.x.83 5224 Trying x.x.x.83 ... telnet: connect to address x.x.x.83: Connection refused This suggests that the problem lies within box A. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of ArcosCom Linux User Sent: Thursday, March 08, 2007 4:49 PM To: lartc@mailman.ds9a.nl Subject: RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs Are you sure? Take a look into: http://tldp.org/HOWTO/TransparentProxy-6.html Regards El Vie, 9 de Marzo de 2007, 1:05, Alec Matusis escribió: I have not considered bridge and ebtables yet. I assume I'd have to implement this in the routing box A. Is ebtables CPU-intensive? The goal of setting up this forwarding is to reduce the load on the router box A. Also, is it very surprising that there is no way to build a transparent proxy/router with standard iproute2 and iptables tools ONLY? It would seem that transparent forwarding of TCP connections to another machine is a very common task. From: Robb Bossley [mailto:[EMAIL PROTECTED] Sent: Thursday, March 08, 2007 9:13 AM To: Alec Matusis Subject: Re: [LARTC] routing TCP to another box preserving ORIGINAL client IPs Have you considered putting a bridge in and using ebtables? That might be what you are looking for. On 3/7/07, Alec Matusis alecm at chatango.com wrote: Hello Martin, I tried implementing DNAT as you indicated: iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 After that, I can see SYN packets arriving on BOX_B_ETH1, having the original client's IP. Only half of the connection gets established after this: I cannot see ACK packets from box B anywhere (neither on BOX_B_ETH0, nor on BOX_A_ETH0). I think the reason is that since box A never sends a SYN packet itself, it never classifies the connection as ESTABLISHED, so all further traffic gets rejected. It's still a mystery to me what happens to SYN packets from be in this scenario however. It turns out that I have to supplement DNAT with SNAT for this to work correctly. On box A: iptables -t nat -i eth0 -p tcp -m tcp --dport $SERVER_PORT -j DNAT --to-destination $BOX_B_ETH1 iptables -t nat -A POSTROUTING -d $BOX_B_ETH1 -p tcp -m tcp --dport $SERVER_PORT -j SNAT --to-source $BOX_A_ETH1 in this case, the clients can connect, however the server on B sees only IP of BOX_A_ETH1, not the original client IPs. Regarding tproxy: I am currently running the TCP server on box A. I would like to move it to box B
RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Alec, : I tried implementing DNAT as you indicated: : : iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 : : After that, I can see SYN packets arriving on BOX_B_ETH1, having : the original client's IP. OK, that means DNAT + routing on your BOX_A is working correctly. : Only half of the connection gets established after this: I cannot : see ACK packets from box B anywhere (neither on BOX_B_ETH0, nor : on BOX_A_ETH0). This is where your problem lies now. You need to find out why the SYN (which you said was transmitted to BOX_B_ETH1) is not getting accepted by this IP stack. * reverse path filtering (net.ipv4.conf.*.rp_filter) * packet filtering rules on BOX_B? : I think the reason is that since box A never sends a SYN packet : itself, it never classifies the connection as ESTABLISHED, so all : further traffic gets rejected. It's still a mystery to me what : happens to SYN packets from be in this scenario however. BOX_A will never have a socket in ESTABLISHED state. BOX_A will have a state entry in the /proc/net/ip_conntrack table. Examine /proc/net/ip_conntrack after sending a SYN to BOX_B. : It turns out that I have to supplement DNAT with SNAT for this to work : correctly. : On box A: : iptables -t nat -i eth0 -p tcp -m tcp --dport $SERVER_PORT -j DNAT : --to-destination $BOX_B_ETH1 : iptables -t nat -A POSTROUTING -d $BOX_B_ETH1 -p tcp -m tcp --dport : $SERVER_PORT -j SNAT --to-source $BOX_A_ETH1 If this works, then I think you problem may be reverse path filtering. : in this case, the clients can connect, however the server on B : sees only IP of BOX_A_ETH1, not the original client IPs. [ tproxy recommendation snipped ] - -Martin - -- Martin A. Brown http://linux-ip.net/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/) iD8DBQFF8OURHEoZD1iZ+YcRAoenAJ9XCZyMf4K7TVCTs28bzIGeu3EEewCg07Cw Spk8a+T/th+ESyPN4hSTjYs= =k+5E -END PGP SIGNATURE- ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] routing TCP to another box preserving ORIGINAL client IPs
My TCP clients connect to box A. I need to forward those connections to a server on box B, such that the original client IPs are visible to the server on B. Each box has two Ethernet ports. One port on each box is connected to WAN, and they are cross-connected in a LAN via remaining ports: --- --- WAN -- |eth0 Box A eth1|---LAN---|eth1 Box B eth0| -- WAN --- --- Is there a way to do this with iproute2 and iptables tools ONLY? Can you provide an example? Nothing in Google after more than a week of searching. An additional requirement is to reduce the load on box A as much as possible (I guess the server on B would still have to reply to the client via A, not using B's own WAN interface however..) ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] routing TCP to another box preserving ORIGINAL client IPs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Greetings Alec, : My TCP clients connect to box A. I need to forward those : connections to a server on box B, such that the original client : IPs are visible to the server on B. : : Each box has two Ethernet ports. One port on each box is : connected to WAN, and they are cross-connected in a LAN via : remaining ports: : : --- --- : WAN -- |eth0 Box A eth1|---LAN---|eth1 Box B eth0| -- WAN : --- --- : : : Is there a way to do this with iproute2 and iptables tools ONLY? : Can you provide an example? Nothing in Google after more than a : week of searching. An additional requirement is to reduce the : load on box A as much as possible (I guess the server on B would : still have to reply to the client via A, not using B's own WAN : interface however..) You need to provide us a bit more information to help you figure out the right way to solve this problem. Why is DNAT not sufficient? Given your description, you should simply be able to: iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 If it were that simple, though, you shouldn't have spent a week looking for the answer. Do you have a TCP service on Box A which is providing services to the client across the WAN? If so, then you are looking for something called transparent proxying in the Linux networking world. You will want to examine the tproxy patches to iptables [0]. If you go with the transparent proxying method, it's helpful to remember: * the client thinks it's connected to Box A * Box A knows its connected to the client * Box A uses client's source address to initiate traffic to Box B * Box B thinks it's connected to client In either case, you are correct about routing. Box B must send its traffic back to Box A to forward back across the LAN. Good luck, - -Martin [0] http://www.balabit.com/products/oss/tproxy/ http://www.balabit.com/downloads/tproxy/linux-2.4/ - -- Martin A. Brown http://linux-ip.net/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/) iD8DBQFF74/JHEoZD1iZ+YcRAlRaAJ4wf2fIc3oBJnGstjUBdpKWn1wOsQCbB2Ee 5Q7zrssGkA02Pq+298i9tEA= =O3sf -END PGP SIGNATURE- ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] routing TCP to another box preserving ORIGINAL client IPs
Hello Martin, I tried implementing DNAT as you indicated: iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 After that, I can see SYN packets arriving on BOX_B_ETH1, having the original client's IP. Only half of the connection gets established after this: I cannot see ACK packets from box B anywhere (neither on BOX_B_ETH0, nor on BOX_A_ETH0). I think the reason is that since box A never sends a SYN packet itself, it never classifies the connection as ESTABLISHED, so all further traffic gets rejected. It's still a mystery to me what happens to SYN packets from be in this scenario however. It turns out that I have to supplement DNAT with SNAT for this to work correctly. On box A: iptables -t nat -i eth0 -p tcp -m tcp --dport $SERVER_PORT -j DNAT --to-destination $BOX_B_ETH1 iptables -t nat -A POSTROUTING -d $BOX_B_ETH1 -p tcp -m tcp --dport $SERVER_PORT -j SNAT --to-source $BOX_A_ETH1 in this case, the clients can connect, however the server on B sees only IP of BOX_A_ETH1, not the original client IPs. Regarding tproxy: I am currently running the TCP server on box A. I would like to move it to box B to reduce the load on A. Other services on A are bound to the same IP address as the server that I need to move, so simply moving that IP address to BOX_B_ETH0 is impossible. Box A has 2.6.11 kernel. Does tproxy install over netfilter? I am sort of scared to tamper with netfilter installation on A, since it is currently running live services. It it possible to implement this scenario: * the client thinks it's connected to Box A * Box A knows its connected to the client * Box A uses client's source address to initiate traffic to Box B * Box B thinks it's connected to client with advanced routing and iptables only? Thanks Alec. -Original Message- From: Martin A. Brown [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 07, 2007 8:24 PM To: Alec Matusis Cc: lartc@mailman.ds9a.nl Subject: Re: [LARTC] routing TCP to another box preserving ORIGINAL client IPs -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Greetings Alec, : My TCP clients connect to box A. I need to forward those : connections to a server on box B, such that the original client : IPs are visible to the server on B. : : Each box has two Ethernet ports. One port on each box is : connected to WAN, and they are cross-connected in a LAN via : remaining ports: : : --- --- : WAN -- |eth0 Box A eth1|---LAN---|eth1 Box B eth0| -- WAN : --- --- : : : Is there a way to do this with iproute2 and iptables tools ONLY? : Can you provide an example? Nothing in Google after more than a : week of searching. An additional requirement is to reduce the : load on box A as much as possible (I guess the server on B would : still have to reply to the client via A, not using B's own WAN : interface however..) You need to provide us a bit more information to help you figure out the right way to solve this problem. Why is DNAT not sufficient? Given your description, you should simply be able to: iptables -t nat -i eth0 [...] -j DNAT --to-destination $BOX_B_ETH1 If it were that simple, though, you shouldn't have spent a week looking for the answer. Do you have a TCP service on Box A which is providing services to the client across the WAN? If so, then you are looking for something called transparent proxying in the Linux networking world. You will want to examine the tproxy patches to iptables [0]. If you go with the transparent proxying method, it's helpful to remember: * the client thinks it's connected to Box A * Box A knows its connected to the client * Box A uses client's source address to initiate traffic to Box B * Box B thinks it's connected to client In either case, you are correct about routing. Box B must send its traffic back to Box A to forward back across the LAN. Good luck, - -Martin [0] http://www.balabit.com/products/oss/tproxy/ http://www.balabit.com/downloads/tproxy/linux-2.4/ - -- Martin A. Brown http://linux-ip.net/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/) iD8DBQFF74/JHEoZD1iZ+YcRAlRaAJ4wf2fIc3oBJnGstjUBdpKWn1wOsQCbB2Ee 5Q7zrssGkA02Pq+298i9tEA= =O3sf -END PGP SIGNATURE- ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
