Re: CARP+pf+pfsync redundant firewalls running active/active doable?
Jason Stubbs wrote:
From what I understand of the theory, it should work but I was hoping
to get a yes, I'm doing it from somebody. Unless there's a reason it
won't work, I'll be having a go and getting it set up in the first week
of March and will write back with the results.
Ok, I had troubles and then looked at the supported solutions, but am
still having the same problems. For reference, I now have a testing
setup that is the same as the something bigger on the following page:
http://www.countersiege.com/doc/pfsync-carp/#big
However, pfsync is syncing too slow causing problems with state
handling. With one client, one server and the two arp-balanced firewalls
in between, essentially what's happening is:
* Client sends SYN packet to server public ip via firewall 1
* Firewall 1 switches the destination to the server's private ip and
forwards the packet
* Server receives the packet and sends a SYN/ACK packet back to the
client via firewall 2
* Firewall 2 forwards the packet as is to the client
* Client sends a RST to the server's private IP (which gets forwarded
elsewhere due to the private ip being unroutable)
* Firewall 2 receives the state update from firewall 1
* Client sends another SYN packet to server public ip via firewall 1
* Firewall 1 NATs it, sends it to the server which replies with a
SYN/ACK going via firewall 2 as before
* Firewall 2 now has the state and so un-NAT's it and sends it back to
the client.
* Client ACK's the SYN/ACK and the connection is set up
I don't fully understand the reasons, but even though the connection is
set up, the state on each firewall is now out of sync. On firewall 1 the
state is CLOSED:SYN_SENT and on firewall 2 the state is
SYN_SENT:ESTABLISHED.
If I turn off arp-balance, only the one firewall is used and the states
are correctly synced on the other firewall. Connections are maintained
when rebooting either firewall and fully synced again after booting.
Configuration on firewall 1 is as follows:
/etc/hostname.carp0
inet 192.168.1.201 255.255.255.0 192.168.1.255 vhid 1 pass carp0dev
carpdev fxp0 advskew 0
/etc/hostname.carp1
inet 192.168.1.201 255.255.255.0 192.168.1.255 vhid 2 pass carp0dev
carpdev fxp0 advskew 50
/etc/hostname.carp2
inet 10.0.0.1 255.255.0.0 10.0.255.255 vhid 3 pass carp2dev carpdev em1
advskew 0
/etc/hostname.carp3
inet 10.0.0.1 255.255.0.0 10.0.255.255 vhid 4 pass carp2dev carpdev em1
advskew 50
/etc/hostname.em0
inet 10.255.255.2 255.255.255.0 NONE
/etc/hostname.em1
inet 10.0.255.1 255.255.0.0 NONE
/etc/hostname.fxp0
inet 192.168.1.203 255.255.255.0 NONE
/etc/hostname.pfsync0
up syncdev em0
/etc/pf.conf
ext_if=fxp0
syn_if=em0
int_if=em1
srv_ip=192.168.1.201
table fw { 192.168.1.201 192.168.1.203 192.168.1.204 10.0.0.1 }
rdr on $ext_if proto tcp from any to 192.168.2.1 port ssh - 10.0.1.1
pass quick on { $int_if $ext_if } proto carp
pass quick on { $syn_if } proto pfsync
pass in on $ext_if from !fw to 10.0.1.1 keep state
/etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet.carp.preempt=1
net.inet.carp.arpbalance=1
Configuration on firewall 2 is almost identical. The 0 and 50 are
toggled on the carp devices. em0, em1 and fxp0 have their IPs
incremented by 1. The firewall is wide open for testing. Closing
everything but that which is absolutely necessary produces the same
out-of-sync issue.
While I would ordinarily assume that the above is expected behaviour,
the countersiege example indicates that it should be doable. Is there
some step that I'm missing?
--
Jason Stubbs
Re: CARP+pf+pfsync redundant firewalls running active/active doable?
Joseph C. Bender wrote: Jason Stubbs wrote: Hi, I'm looking to set up redundant firewalls in pretty much the same way as is detailed in the PF FAQ. For discussion purposes, I've reproduced the basic network layout below. From your description and questions below, it looks like you're not trying to do it the same way, and your understanding may be incomplete. Apologies. I stated what I want to do but forgot to state the why. We're moving to a new data center. Originally, I was looking at setting up stock redundant firewalls but we will be charged almost as much for the inactive line to the data center as the active line costs. While inbound traffic isn't such a problem, output can reach up to 60Mbps during peak times. Hence, what I'd like to do is run 2 active 30Mbps lines and balance outgoing traffic between them rather than having active/passive 60Mbps lines. [Snip Layout] Firewall External IP addresses 10.0.0.1 nat'ed to sv1 with fw1 being the master 10.0.0.2 nat'ed to sv2 with fw2 being the master Firewall Internal IP addresses 192.168.0.1 with fw1 being the master 192.168.0.2 with fw2 being the master Are these CARP'd addresses, as in you have multiple CARP interfaces per NIC? If so, why? CARP'd addresses, yes. The external addresses are those of the services being ran on sv1 and sv2 (which are in fact LVS'd Linux farms). The multiple internal addresses are for the internal servers to round-robin outgoing traffic to. Now with sv1's default route being set to 192.168.0.1 and sv2's default route being set to 192.168.0.2 all should work fine (at least as far as documentation goes). However, what I'd like to do is have both sv1 and sv2 use both 192.168.0.1 and 192.168.0.2 for routing in a round-robin fashion. With fw1 handling sv1's nat'ing, will fw2 correctly be able to un'nat and send out replies sent by sv1? I'm not going to answer this directly, mostly because I can't figure out, given you have a really kickass failover system, why you'd even want to do this. Given you're using hardware that is capable of using em cards, box loading shouldn't be an issue. Put simply, you're trying to make this harder than it really is, I think. I suggest the following, which is what we use at the office and is a heck of a lot closer to what the PF User's Guide suggests: [snip configuration details] If fw1 goes paws up or needs maintenance, and if you've done everything right, fw2 will take the load almost instantly (within milliseconds in my experience). This configuration is essentially what I'm looking at doing. The only difference is that instead of having one internal address, I'd like to have two. As I said above, the goal is to balance outgoing traffic and still have redundancy. I'm aware that when one box goes down there won't be enough bandwidth for peak times, but that's a cost/performance tradeoff that's been approved. [snip rest, as it's not relevant to my answer] My whole point is that with the CARP and pfsync redundancy, there's no need to have really complicated routes to and from your servers and their firewalls. Actually, we'd need to be looking to find a way to balance outgoing traffic anyway. We're at about 60Mbps during peak times now but that's only going to grow. As we can only get a maximum of 100Mbps out of each line, overcoming that limit is also on the agenda. From what I understand of the theory, it should work but I was hoping to get a yes, I'm doing it from somebody. Unless there's a reason it won't work, I'll be having a go and getting it set up in the first week of March and will write back with the results. -- Jason Stubbs
Re: CARP+pf+pfsync redundant firewalls running active/active doable?
Jason Stubbs wrote: Hi, I'm looking to set up redundant firewalls in pretty much the same way as is detailed in the PF FAQ. For discussion purposes, I've reproduced the basic network layout below. From your description and questions below, it looks like you're not trying to do it the same way, and your understanding may be incomplete. [Snip Layout] Firewall External IP addresses 10.0.0.1 nat'ed to sv1 with fw1 being the master 10.0.0.2 nat'ed to sv2 with fw2 being the master Firewall Internal IP addresses 192.168.0.1 with fw1 being the master 192.168.0.2 with fw2 being the master Are these CARP'd addresses, as in you have multiple CARP interfaces per NIC? If so, why? Now with sv1's default route being set to 192.168.0.1 and sv2's default route being set to 192.168.0.2 all should work fine (at least as far as documentation goes). However, what I'd like to do is have both sv1 and sv2 use both 192.168.0.1 and 192.168.0.2 for routing in a round-robin fashion. With fw1 handling sv1's nat'ing, will fw2 correctly be able to un'nat and send out replies sent by sv1? I'm not going to answer this directly, mostly because I can't figure out, given you have a really kickass failover system, why you'd even want to do this. Given you're using hardware that is capable of using em cards, box loading shouldn't be an issue. Put simply, you're trying to make this harder than it really is, I think. I suggest the following, which is what we use at the office and is a heck of a lot closer to what the PF User's Guide suggests: carp0: Assigned to em0 on both fw1 and fw2 Assigned 192.168.0.1 fw1 is the master. carp1: Assigned to em2 on both fw1 and fw2 Assigned 10.0.0.1 AND 10.0.0.2 (primary and alias). Make sure that you have carp info for the aliases (vhid and whatnot) for the alias lines. I can't remember if it's required per alias entry, but that's what we're running here and it works. Don't forget to set your advskew values properly, i.e. they should be higher on fw2 if it's the backup box. pfsync0 still on em1. (Personally I'd do em1 as the carp1 int and em2 as the pfsync, but I'm weird like that). em0 on fw1, assign 192.168.0.2 as the int's ip for management and whatnot. em0 on fw2 assign 192.168.0.3 for the same reasons. Do the same thing for em1 on both firewalls using 10.0.0.x addresses. Set up your pf rulesets, doing your rdr rules for both sv1 and sv2 on the inbound *interfaces* (this has bit me in the ass many times). Set the gateway on both sv1 and sv2 to 192.168.0.1 If fw1 goes paws up or needs maintenance, and if you've done everything right, fw2 will take the load almost instantly (within milliseconds in my experience). [snip rest, as it's not relevant to my answer] My whole point is that with the CARP and pfsync redundancy, there's no need to have really complicated routes to and from your servers and their firewalls. Hope this helps. -- Joseph C. Bender jay cee bender at bendorius dot com
