Don Ng <[EMAIL PROTECTED]> wrote: > Hi a newbie here but I have heard claims that >any software based firewall bandwidth throughput etc >is >inherently unreliable when dealing with the short >durations of latency through firewalls.
It's a bit more subtle than that. The first rule in good benchmarking is to make sure that what you are measuring is related to the value proposition you're trying to test. For a firewall, its value is that it provides a presumed level and type of security. So to measure a firewall's primary purpose, you want to measure _security_ not _thruput_. Although thruput is absolutely definitely a factor worth considering, it's really easy to get focused on a red herring either accidentally or on purpose. "NEW! Firewall-2000: the industry's heaviest firewall, in pounds per cubic inch!!!" Thruput's definitely something interesting to look at, but again it's hard. You want to test the firewall with the most realistic possible looking loads. Which generally means real traffic! It's hard (people pay a LOT of money for accurate load generators) to simulate traffic well. It's even harder with security traffic because if you're not careful you may wind up simulating something that is neither real traffic nor an attack. There's a few old (ancient, really!) tools and papers for firewall load testing that Andrew Molitor and Chris Kostick and I worked on back in the dawn of the firewall age of Internet security. They're on: http://web.ranum.com/pubs/fwperf/index.htm beware that they are so old they may not port cleanly to newer versions of UNIX that ignore backwards compatibility (linux...) I recently published a paper on IDS benchmarking that goes into some of the issues about attack traffic simulation and how difficult it is. Rather than repeating big chunks of it here, the URL is: http://www.nfr.com/forum/publications.html see "Benchmarking IDS" -- it's amazingly hard to do it well, it turns out. >> 1. Sustained TCP connections, thoughput & number. >> Eg. >> FTP You probably want to measure max live connections at any one time, and also the effect on thruput as the number increases. Note that this has nothing to do with the security properties of the firewall but it is still interesting. You may also want to specify not just connections but connections by type. I suspect very few firewalls do anything with FTP (your example!) but handle the callback connections and gateway the data through. Why not HTTP connections instead? Some firewalls might actually do HTTP traffic/content analysis, so they'll have a different performance window, there. IMPORTANT POINT: in the case above, a firewall that _does_ content analysis will be SLOWER than on that doesn't. If all you publish is thruput results you are actually penalizing the more secure firewall!!! A hub will outperform most firewalls and can handle lots of connections but are you measuring the right thing? >> 2. Short-lived TCP connections, throughput, number, >> connection establishment and tear-down time. Eg. >> SMTP/HTTP This is an interesting number again but what does it have to do with security? >> 3. Sustanied UDP connections (although UDP is >> connectionless), throughput & number. Eg. Streaming >> video/audio. Interesting number again, but - what does it have to do with security? I could probably make a defensible argument that a firewall that doesn't let UDP through at all is a better firewall than one that doesn't. So... Which brings me to the next point... Your measures here are predicated on network-level traffic. What about a proxy firewall? It's probably a lot more secure than a network-level firewall but some of the measures you are describing above are absolutely meaningless for a proxy. This is one of the things Molitor and I had to deal with. The conclusion we came to was that the only effective measure was of real traffic. I.e.: you can't measure arbitrary UDP/DNS packet performance, you have to measure DNS query performance against a live nameserver - which means you have to eliminate caching effects and performance of the backend/frontend nameserver/nameclient code. >> 4. Short-lived UDP communication, number. Eg. DNS. Packet-oriented measure. I'd argue that the most secure firewall out there won't even admit this test; it'd score a big fat zero. :) >> 5. ICMP RTT at diferent load levels. What kind of firewall lets ICMP through it at all? The good ones will score a zero here. The bad ones will show throughput... >> 6. SYN Flood test Detecting and blocking them, or letting them through? ;) Are you measuring security here or throughput... >> 7. Connection establishment time wrt to number of >> rules on the firewall. >> >> 8. Filtering and fragmentation >> - Reaction of the firewall on receiving a TCP packet >> with the RST or ACK flag set. >> - IP fragmentation re-assembly test. >> - Overlap recognition What about a proxy firewall? >> 9. Are existing checksums for IP, TCP and UDP >> verified? >> >> 10. A portscan of the firewall IP. Of the servers >> behind the firewall. What does this show? How do you factor in NAT? How do you factor in the fact that a decent firewall isn't going to react to this at all? _OR_ it may show every port as "live" just to mess with you - I wrote one of those once... It's going to be nearly impossible to get an apples-to-apples comparison unless you're extremely careful in setting your criteria. >> 11. Nessus tests on the firewall IP and the servers >> behind the firewall. What does Nessus show? It's a vulnerability scanner. Vulnerability scanners do inherently different things from actual attack tools. Or at least they should. Are you measuring the firewall's ability to be scanned, or to block attacks? What about a proxy firewall? It's going to react completely differently to a network-level firewall. Could you even tell the difference? >> 12. All the tests repeated with static NAT enabled. >> >> 13. All the tests repeated with IPSec. With attacks over IPSec (interior domain) or off it? (exterior) Are you measuring the firewall's ability to validate and decrypt IP, or to throw it away? Those are the only 2 things such a test would measure. >> 14. Effect of logging on the these tests. "Effect" is not something you can meaningfully benchmark. Do you mean performance effect, security effect, or effect on disk I/O, or what? >> Can you think of more tests for >> stressing/penetrating >> the firewall. Also, what methodology should be >> adopted >> to measure the various test results? I think the methodology you've described above and the questions you've listed above are so rudimentary that you won't even be able to draw a useful conclusion from them except for seat-of-the-pants subjective measures. There are huge differences in fundamentals of how various firewalls can work (though most of what's out there now are just glorified cheap sh*t packet filters...) that your criteria don't take into account. Your focus is over-weighted toward performance measures that have nothing to do with security (because it's easier to measure thruput than security) which risks establishing a preference for the firewall that is fast, rather than a good firewall. Frankly, I think you need to go back to the drawing board and specify whether you are measuring security or thruput. Start with that decision and work your way back from there. Building a good test criteria is _incredibly_ hard and is not a lightweight exercise to jump into without a lot of thought and some heavy-duty research into testing methods. Every time someone runs a poorly-designed test and publishes the results, they do the community a disservice, so please be as rigorous as you can afford to be! Good luck! mjr. --- Marcus J. Ranum Chief Technology Officer, NFR Security, Inc. Work: http://www.nfr.com Personal: http://www.ranum.com _______________________________________________ Firewalls mailing list [EMAIL PROTECTED] http://lists.gnac.net/mailman/listinfo/firewalls
