I had a tool for ipfilter that would simulate packets hitting it, and then make sure the reaction was the same as the last edit and the whole thing was driven by make.
You're basically asking a similar question to "does this program do what I want?" which is unsolvable. Asking "does this program ever stop?" is even unsolvalble in the general case. Basically the best you can do is code up test cases around likely problem spots, and make sure each of those does what you want. But IIUC you can't even test pf rules without real packets, the way you could with ipfilter. I wanted something that would look at all the rules and automatically create packets for both sides of an inequality (or all three, if it were an equality or range) and show a venn diagram with the results, but this became tricker to visualize with stateful matching and only two dimensions for output. For the most basic test, run nmap from an untrusted host. Or visit any of the sites that will scan your IP for you (e.g. "Shields Up!" at grc.com). Beyond that, there's a trillion packet creation tools (I like scapy) but I'm afraid it's not a simple answer to implement. Amusing firewall anecdote: Once long ago, I had an OpenBSD firewall, and two SMC ISA ethernet cards. I am not sure how, perhaps a packet arrived at an inopportune moment, but the cards autoprobed in the opposite order of normal. I was using a bogus internal domain name, and syslogging the blocked packets to something like loghost.bogus.com, except that someone registered that bogus domain name in the meantime, and their syslog server was named loghost and so all my syslog information got sent to them. Also the firewall was essentially unprotected. I was lucky to have been on the console running tcpdump at the time, and saw syslog stuff flowing out to the Internet, so I caught it quickly. A reboot fixed it and it never happened again. I now use a bogus TLD internally. The end. -- Security Guru for Hire http://www.lightconsulting.com/~travis/ -><- GPG fingerprint: 9D3F 395A DAC5 5CCC 9066 151D 0A6B 4098 0C55 1484
