Re: calling all PFsync users for experience, gotchas, feedback, tips and tricks
On Thu, May 19, 2022 at 09:35:53AM -, Stuart Henderson wrote: > On 2022-05-19, Jordan Geoghegan wrote: > > I've run pfsync + CARP for a number of years now. One interesting > > "gotcha" I discovered when building an IPv6-only test network was that > > pfsync does not work in an IPv6-only environment. I tried both unicast > > and multicast configurations to no avail. When pfsync has a parent > > interface that only has an IPv6 address assigned (ie no IPv4 at all), no > > pfsync traffic transits the interface. Just thought I'd share this > > little tidbit since you were looking for edge cases and gotchas and > > since IPv6 support (or lack thereof) is not mentioned in the manpage. > > That sounds like a bug not an "edge case". To my knowledge nobody ever > reported that, consider writing it up for bugs@. Connectivity issues in a pure IPv6 environment are often due to NDP packets not being correctly passed. For example, the default firewall ruleset in /etc/rc is supposed to allow basic connectivity such as ssh. However, it breaks IPv6 neighour discovery protocol in at least some situations. I'm not in the office at the moment, so I can't test anything on a current system, but notes I made last year which would have been with 6.8-release: Considering a direct link between two machines with no routing or other network hardware inbetween: Output from ndp -a with the default ruleset: Neighbor Linklayer Address Netif ExpireS Flags node1(incomplete) em0 expired N node2b4:2e:99:f2:2f:67 em0 permanent R l fe80::b62e:99ff:fef2:2f67%em0b4:2e:99:f2:2f:67 em0 permanent R l The default ruleset allows neighbour solicitations out and neighbour advertisements in. Adding rules to allow neighbour solicitations in and neighbour advertisements out, fixes the problem.
Re: calling all PFsync users for experience, gotchas, feedback, tips and tricks
On 2022-05-19, Jordan Geoghegan wrote: > I've run pfsync + CARP for a number of years now. One interesting > "gotcha" I discovered when building an IPv6-only test network was that > pfsync does not work in an IPv6-only environment. I tried both unicast > and multicast configurations to no avail. When pfsync has a parent > interface that only has an IPv6 address assigned (ie no IPv4 at all), no > pfsync traffic transits the interface. Just thought I'd share this > little tidbit since you were looking for edge cases and gotchas and > since IPv6 support (or lack thereof) is not mentioned in the manpage. That sounds like a bug not an "edge case". To my knowledge nobody ever reported that, consider writing it up for bugs@.
Re: calling all PFsync users for experience, gotchas, feedback, tips and tricks
On 5/11/22 12:32, Tom Smyth wrote: Hello Folks, We are updating some course material for an upcoming PF firewall course, and I would like to put a call out to those who use PFsync in a redundant firewall cluster about your user experience, have you come across any edge cases? have you any tips or tricks about PFSync. have you come across any edge cases / minor misconfigurations / suboptimal configurations that caused problems, were there some tweaks you had to make to make your system scale ? it is likely that people who are running PFSync have more complicated firewall configs. and I would like to see what tuning other people have done in the field. I would appreciate any feedback or problem descriptions (with our without solutions) what is the largest throughput firewall you deployed with PFSync? how was your experience of running with PFsync vs without PFsync on your firewall. Thanks again, I've run pfsync + CARP for a number of years now. One interesting "gotcha" I discovered when building an IPv6-only test network was that pfsync does not work in an IPv6-only environment. I tried both unicast and multicast configurations to no avail. When pfsync has a parent interface that only has an IPv6 address assigned (ie no IPv4 at all), no pfsync traffic transits the interface. Just thought I'd share this little tidbit since you were looking for edge cases and gotchas and since IPv6 support (or lack thereof) is not mentioned in the manpage. Regards, Jordan
Re: calling all PFsync users for experience, gotchas, feedback, tips and tricks
Hello all, Thanks for the feedback it is really helpful to have peoples experiences in the wild to help feed into the training course content. and certainly better than just my humble experience I really appreciate all of your feedback. Thanks again folks, Tom Smyth Tom Smyth On Fri, 13 May 2022 at 11:20, Stuart Henderson wrote: > > On 2022-05-13, Marko Cupać wrote: > > The only problem I currently have with pfsync is the fact that it does > > not synchronise queue membership of states. > > IIRC this is meant to work but only if you have identical rulesets, > after expanding interface addresses etc. This will require some care in > constructing pf.conf - interface groups instead of interface names if > nic hw is different - "(self)" or list the addresses of both firewalls > instead of using "self" - avoid "antispoof". > > -- Kindest regards, Tom Smyth.
Re: calling all PFsync users for experience, gotchas, feedback, tips and tricks
On 2022-05-13, Marko Cupać wrote: > The only problem I currently have with pfsync is the fact that it does > not synchronise queue membership of states. IIRC this is meant to work but only if you have identical rulesets, after expanding interface addresses etc. This will require some care in constructing pf.conf - interface groups instead of interface names if nic hw is different - "(self)" or list the addresses of both firewalls instead of using "self" - avoid "antispoof".
Re: calling all PFsync users for experience, gotchas, feedback, tips and tricks
Nick Holland writes: > Wrote a little script which, when run: Good grief, man! Just put the pf.conf in CVS and push it with rdist. We do that for all our carped firewall pairs and it works a treat. The following 'special' command in the Distfile will give you a failsafe reload of the pf rules: special files-hc1/etc/pf.conf " pfctl -f /etc/pf.conf || mv /etc/pf.conf.OLD /etc/pf.conf" ; --lyndon
Re: calling all PFsync users for experience, gotchas, feedback, tips and tricks
Hi Tom On 5/11/22 21:32, Tom Smyth wrote: We are updating some course material for an upcoming PF firewall course, and I would like to put a call out to those who use PFsync in a redundant firewall cluster The one thing that immediately comes to mind is to NOT use a crossover cable for the pfsync connection (even though that seems to be kind of recommended in the pfsync(4) man page). Doing so will lead to a change of the other firewall's carp demotion counter on its pfsync interface if one peer is rebooted or shut down (and thus causing a link down event on the cabled interface on the other side). It also gives you three chained single points of failure at the same time (nic1, cable, nic2), which I would rather avoid (do the math). I do of course agree with the intention of the suggestion (only run pfsync over a secure link). Since I am in the position where I only run my PF firewalls in a trusted environment, where I also control the switches (no shared cloud etc. infrastructure), I have found that running pfsync over a dedicated VLAN interface on a pair of trunk(4)ed NICs on 2 trusted switches sufficiently satisfies that requirement. Best, Markus
Re: calling all PFsync users for experience, gotchas, feedback, tips and tricks
On 5/11/22 3:32 PM, Tom Smyth wrote: Hello Folks, We are updating some course material for an upcoming PF firewall course, and I would like to put a call out to those who use PFsync in a redundant firewall cluster about your user experience, have you come across any edge cases? have you any tips or tricks about PFSync. have you come across any edge cases / minor misconfigurations / suboptimal configurations that caused problems, were there some tweaks you had to make to make your system scale ? it is likely that people who are running PFSync have more complicated firewall configs. and I would like to see what tuning other people have done in the field. It's been a few years since I managed a firewall cluster with pfsync, but one thing I came up with fairly early on is we needed a way to manage rule changes between the two devices, and I came up with something that I think is pretty cool, and yet haven't seen anyone else describe something similar. Wrote a little script which, when run: * Compared this script on "this" FW with "Other" FW * generated a diff between the /etc/pf.conf file on both systems (other box assumed to be "old", "this" box assumed to be new) * Put the diff into a file along with the user ID of the administrator who made the changes, prompted the user to enter a description for the change above the diff, who approved it, etc. * If the administrator enters a change log and saves the file: * save that file to disk, with a clear date and time stamp. * Copy "this" FW's pf.conf file to the "other" system * pfctl -f /etc/pf.conf on both systems * scp the change log file to the other system * Probably should look for changes in hostname.*, and deal with their changes, too, but I didn't implement that at the time, so I'd be lying if I told you I did. But I recall wishing I had! :D This way, you have a log of every change made to the system, plus administrator comments as to why the change was made. EITHER FW can push changes to the other, both boxes have a full history, either box can be used to rebuild the other. IF you find a problem, a diff to undo it is easily found. It makes change control almost a pleasure. If someone made a change and forgets to push it to the other, you can see that the diff is more complicated than you expected (or you made a typo and blew something out!). I've used similar scripts for other fully redundant systems, like DNS servers. Yes, I'm sure you can do similar things with system management applications like puppet, etc., but this is completely self-contained, no extra hw or packages required. (and yes, DNS has the master/slave config with zone transfers, but I'd argue this is a better system.) Nick.
calling all PFsync users for experience, gotchas, feedback, tips and tricks
Hello Folks, We are updating some course material for an upcoming PF firewall course, and I would like to put a call out to those who use PFsync in a redundant firewall cluster about your user experience, have you come across any edge cases? have you any tips or tricks about PFSync. have you come across any edge cases / minor misconfigurations / suboptimal configurations that caused problems, were there some tweaks you had to make to make your system scale ? it is likely that people who are running PFSync have more complicated firewall configs. and I would like to see what tuning other people have done in the field. I would appreciate any feedback or problem descriptions (with our without solutions) what is the largest throughput firewall you deployed with PFSync? how was your experience of running with PFsync vs without PFsync on your firewall. Thanks again, -- Kindest regards, Tom Smyth.
