Re: pfsync interface in carp group
On 6/9/2020 1:42 PM, Markus Wernig wrote: Neither jumbo frames nor multicast will prevent group demotion when the other side of a crosslink cable goes physically down. Only not having the sync interface in the carp group will. True. But I think he was just discussing general best practices, not things specific to the issue I raised. Using jumbo frames would reduce the packet overhead if the number of states to be sent in a particular transmission would have exceeded the default MTU. In my scenario, it looks like I don't have enough traffic for that to occur, at least not on a regular basis.
Re: pfsync interface in carp group
On 6/9/20 9:25 PM, Paul B. Henson wrote: > Hmm, I had never considered using jumbo frames. ... > I guess multicast would work too Neither jumbo frames nor multicast will prevent group demotion when the other side of a crosslink cable goes physically down. Only not having the sync interface in the carp group will.
Re: pfsync interface in carp group
On 6/9/2020 7:36 AM, Stuart Henderson wrote: IME the best setup for pfsync between 2 machines is to use a dedicated cross-connect (preferably configured for jumbo frames). Obviously that's not possible with >2 machines though. Hmm, I had never considered using jumbo frames. It looks like based on the traffic level on my systems, the packets are generally below the default 1500 MTU anyway though, so it probably wouldn't help. 12:16:27.564940 lisa-bart.pbhware.com: PFSYNCv6 len 896 12:16:28.023806 lisa-bart.pbhware.com: PFSYNCv6 len 712 12:16:28.195774 bart-lisa.pbhware.com: PFSYNCv6 len 276 12:16:28.207817 lisa-bart.pbhware.com: PFSYNCv6 len 528 I'm undecided what's best to do with the group by default. But I never use syncpeer for that config, just the default with multicast, which I think is quite common - changing group based on whether or not syncpeer is used doesn't make sense to me. I guess multicast would work too for a direct peer relationship, but it just seemed more accurate to explicitly configure the two peers. Some documentation regarding the interaction of pfsync and the carp group might be helpful, along with a suggestion to remove the carp group from the pfsync interface in certain deployment scenarios. It would also be nice to document the dependency on the two rule sets being exactly identical in order to properly replicate rule specific state timeouts between them. It took me a while to sort out why that was failing. Maybe I will try to write something up; the source for the pfsync man page is in CVS, where is the source for webpages such as: https://www.openbsd.org/faq/pf/carp.html#pfsyncop Thanks…
Re: pfsync interface in carp group
On 2020-06-08, Markus Wernig wrote: > On 6/9/20 12:27 AM, Paul B. Henson wrote: > >> Yes, I am using a direct link between the two physical firewalls. > [...] >> Is this no longer a best practice? > > If it's in the documentation, I suppose it still is. > > But I have found it problematic, because taking down one firewall, or > even only its sync interface, will automatically demote the sync > interface on the other one, which then will affect the whole carp group, > if the interface is part of that group. That is exactly what Paul's suggestion would help. IME the best setup for pfsync between 2 machines is to use a dedicated cross-connect (preferably configured for jumbo frames). Obviously that's not possible with >2 machines though. I'm undecided what's best to do with the group by default. But I never use syncpeer for that config, just the default with multicast, which I think is quite common - changing group based on whether or not syncpeer is used doesn't make sense to me.
Re: pfsync interface in carp group
On 6/9/20 12:27 AM, Paul B. Henson wrote: > Yes, I am using a direct link between the two physical firewalls. [...] > Is this no longer a best practice? If it's in the documentation, I suppose it still is. But I have found it problematic, because taking down one firewall, or even only its sync interface, will automatically demote the sync interface on the other one, which then will affect the whole carp group, if the interface is part of that group. When I first tried carp in the lab many, many years ago, I vaguely remember seeing effects similar to what you describe, and have used switched sync interfaces ever since.
Re: pfsync interface in carp group
On 6/8/2020 6:29 AM, Philipp Buehler wrote: did you follow some "howto" and set net.inet.carp.preempt=1? Well, if you consider the official openBSD documentation a "how-to", then yes :). In the example in https://www.openbsd.org/faq/pf/carp.html under the section "Combining CARP and pfsync for Failover" it says: ! enable preemption and group interface failover # sysctl net.inet.carp.preempt=1 # echo 'net.inet.carp.preempt=1' >> /etc/sysctl.conf As well as in the example in 'man pfsync': The following must also be added to /etc/sysctl.conf: net.inet.carp.preempt=1 One of my firewalls has newer hardware and more power than the other, it is the primary. If I reboot it and the load fails over to the secondary, I want the load to automatically come back to the primary once it is available again. Thanks…
Re: pfsync interface in carp group
On 6/7/2020 5:21 PM, Markus Wernig wrote: I don't see that behaviour on my carp pair. Are you using a cross-link cable between the two firewalls? (You shouldn't, in my experience.) Yes, I am using a direct link between the two physical firewalls. It seems to be the configuration recommended by the documentation? https://www.openbsd.org/faq/pf/carp.html "The firewalls are connected back-to-back using a crossover cable on em1." As well as in 'man pfsync': "Only run the pfsync protocol on a trusted network - ideally a network dedicated to pfsync messages such as a crossover cable between two firewalls." "A crossover cable connects the two firewalls via their sis2 interfaces." Is this no longer a best practice?
Re: pfsync interface in carp group
Am 08.06.2020 00:29 schrieb Paul B. Henson: However, for only two firewalls, when you're using the syncpeer directive for the pfsync interface, it seems it would be better not to default to belonging to the carp group? With only two firewalls, if one of them has broken synchronization, so does the other, so is there any real point in trying to migrate away from the one that's currently master? Hi, did you follow some "howto" and set net.inet.carp.preempt=1? ciao -- pb
Re: pfsync interface in carp group
On 6/8/20 12:29 AM, Paul B. Henson wrote: > whenever I rebooted the secondary firewall, the > carp interfaces on the primary would flip to backup and then back to > master as the secondary one rebooted I don't see that behaviour on my carp pair. Are you using a cross-link cable between the two firewalls? (You shouldn't, in my experience.) best
pfsync interface in carp group
I've had a pair of redundant firewalls using pfsync for years. I've noticed in the past that whenever I rebooted the secondary firewall, the carp interfaces on the primary would flip to backup and then back to master as the secondary one rebooted. I never really noticed any issues with it, so I just ignored it. Since upgrading to 6.7 though, if I have an active ssh connection to the carp IP address on the primary when I reboot the secondary and these interface flip-flops occur, my connection is dropped, which is undesirable. It looks like this is happening because by default the pfsync interface is in the carp group, so when it goes down, it demotes all of the carp interfaces on the system. I can see why this would be useful for a setup using multicast and more than two firewalls, as if you are not synchronizing states, you are probably not the best choice to be the active owner of the virtual IP addresses. However, for only two firewalls, when you're using the syncpeer directive for the pfsync interface, it seems it would be better not to default to belonging to the carp group? With only two firewalls, if one of them has broken synchronization, so does the other, so is there any real point in trying to migrate away from the one that's currently master? I updated my configuration to remove the pfsync interface from the carp group and now when I reboot there are no issues with the carp interfaces changing state or connections being dropped. Would it make sense to not automatically include the pfsync interface in the carp group if it is using the syncpeer directive? Thanks…
