Ok, I've sat on the sidelines on this one as long as I could - I can't
take it any longer!!! :)
While IMNSHO there is clearly no ONE correct answer to this debate, I
encourage members on this list to consider a few things that I (and others
at Neohapsis) have learned over the years of testing signature-based
security gateway products. Maybe some of this will help...or at least,
help until the next time this thread gets created (about every 2 years or
so, yeah?):
1. Philosophically, as a tester I believe it is important to decide
whether you want to validate that a device is detecting actual attacks vs.
detecting a specific set of packets. They may *look* the same at times,
but they are unequivocally NOT the same thing. Think about this one for a
second before going further. (I totally agree with Ivan on this point.)
2. I am pretty confident that at least 90% of the products on the
commercial market have made their way through our lab at one point or
another, and I can say WITHOUT HESITATION or DOUBT that almost ALL OF THEM
have botched AT LEAST one signature or two - sometimes more. As in, they
thought sig x worked, we ran an actual exploit, and the device came up
with a big nadda. I can also say - without hesitation - that many of the
QA efforts deployed by many of the
IDS/IPS/UTM/whatever-we-call-it-these-days are based on replay methods,
and sometimes those packet captures (used for the replay process) are
flawed themselves. Which leads me to:
3. I can also say that we've seen replay-based tools put traffic on the
wire that is utter BS. As in, it's a bunch of junk that no IDS should
ever flag because it isn't actually hostile traffic. But rarely do I see
testers actually busting out Ethereal and their exploit code to validate
that what went down on the wire is in fact, something the IDS should flag.
I dislike replay in general for reasons stated here (and other reasons,
too - I won't submit you to any more rambling than necessary!), but it is
especially important to take into consideration that just because a replay
/ injection tool says it does X doesn't mean it really does X correctly.
And do we really want our signature-based devices flagging things that
aren't actually hostile?
4. I like real attack traffic if I didn't state this already. :)
5. Put another way, it's one thing for an engineer to say "Hey, I don't
think that packet capture represents a legit attack stream" but it's kind
of hard to argue a root shell prompt staring you in the face. I think
I've saved DAYS of my life by preemptively ending finger pointing thanks
to mr. shell; the eloquent simplicity that is a remote root shell remains
a hard point to argue!
6. I think MetaSploit, CORE's product, and ImmunitySec's Canvas are good
foundations for executing real attacks. NOTE THAT THESE ARE NOT
VULNERABILITY SCANNERS; DON'T USE THOSE - THEIR METHODS TYPICALLY DON'T
EXPLOIT ACTUAL TARGETS! (See recurring thread on this one, too..)
However, without sufficient LEGIT background traffic (see Spirent's
WebAvalanche for more info here) this exploit-only method of testing is
limited in value because:
7. Signature recognition capabilities are often tied to specific product
engine thresholds. Perhaps one of the points I see most often missed by
testers is identifying / knowing the difference between engine flex
testing and attack validation. Running an HTTP-based attack with no HTTP
background traffic what-so-ever and validating whether or not the device
detects the attack is one thing. Not real-world at all, limited in value,
but it is ONE test. However, validating that same attack while you start
incrementally loading on legit traffic (and in turn, stressing different
parts of the product) is a lot closer to most real world conditions, and a
lot more valuable of a test. There are people on this list that know IDS
internals way better than I ever will, but I suspect any engine developers
on this list will attest to the, er, threshold challenges. :) This is a
much bigger subject, but the short version is NO BACKGROUND TRAFFIC = TEST
IS LIMITED IN VALUE.
8. Driving this point one step further, using replay to put attack
patterns on the wire is one thing, using it to put background traffic on
the wire is another thing, but trying to test whether a product can
perform accurate attack detection at sustained rates of 500Mbps? 700Mbps?
1Gbps+? Doing the math on the pcap sizes you'll need for replaying
background traffic at these higher speeds drives the point home pretty
fast...
9. All IDSs break. Period. Or at least, I've never tested one that
couldn't be broken - but IMO, that's really not material. What is
material is that us consumers understand where that breaking point is, and
whether or not that breaking point is relevant to our environment. (i.e.
I'm not going to sweat my IDS freaking out at 500Mbps when I have it
inspecting a DS3) In short, know what your testing methods validate...and
know what they DO NOT validate.
10. I like using real background traffic, and real attack injection.
(Wait, did I say that already?) Ok, final point: if you don't have a well
equipped lab the ideal scenario for many orgs is piloting a device using
their own internal traffic, and then using controlled attack and victim
boxes. (The metasploit / vmware combo is a good one, IMO) As public
testers we've worked very hard to learn how to synthetically generate
traffic extremely close to real world conditions, but we will rarely (if
ever) be able to *exactly* replicate the real thing in the lab. (We can
get a lot closer now, however!) When in doubt, pilot.
Ok - wheh - I feel better now! (hope this helps)
Greetings from Chicago,
-Greg
------------------------------------------------------------------------
Test Your IDS
Is your IDS deployed correctly?
Find out quickly and easily by testing it
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
------------------------------------------------------------------------
