> -----Original Message-----
> From: Paul D. Robertson [mailto:[EMAIL PROTECTED]]
> Sent: Monday, September 10, 2001 11:08 PM
> To: Ben Nagy
> Cc: Daniel Crichton; [EMAIL PROTECTED]
> Subject: RE: Firewall testing recommendations
>
>
> On Mon, 10 Sep 2001, Ben Nagy wrote:
>
> > But any "analysis" process should include external
> verification - ie
> > that the box is doing what you told it to do, [...]
> > This is quite distinct from the traditional pen-test in
> that it isn't
> > blind.
>
> Yep, I'd _love_ to launch a jihad upon blind pentests *or*
> blind vulnerability tests. Anyone who's trying to play
> "stump the chump" with their security vendors is losing a
> significant ammount of value, and moving from a model where
> the test is cooperative to one where it's combatative.
[...]
When I review people's organisational security, and especially when I've
actually set up some or all of the infrastructure, I do my own verification.
However, afterwards I also like to advise them to begin a regular "blind"
audit regime, often using a direct and hostile competitor (it's not
you-scratch-my-back arrangement, the other guys _want_ the accounts). I
think that there is a point where one if too close to the architecture, and
teeny little problems start to look like major truck-sized holes. For that
reason, blind pen-tests are useful (IMO, of course) for drawing a line that
says "You Must Be This Smart to Hack This Network". Pentesters think like
hackers - they do the same stuff, run the same sort of probes and scans and
miss the same flaws.
I did note the point about companies that place restrictions on the
penetration testers, or rule out certain classes of attack. That's Just
Dumb, and removes pretty much _all_ value from the process.
[...]
> I'm curious- how does a blind test hold value versus a more
> open test? The only thing I can see (and trust me- I see it
> pretty clearly) is that it's significantly less expensive to
> implement for _most_ cases.
Ok, looking at it that way, while I'm pretty confident that having a third
party audit your work is a Good Thing for the customer (although
commercially risky for the vendor) I can't see any disadvantage from a
security POV to making the 3rd party test open as well. I guess it could
lead to risk inflation and thus over engineering the site security, since
one would expect most risks to be exposed in that kind of scenario.
> > Also, do you find vulnerability testing (like running nessus)
> to be about the same as pentesting in your definition, or
> more or less valuable?
Much less. I like to look at the OSSTMM[1] as a template for a penetration
test. I consider what you call vulnerability testing to be a strict subset
of a pen-test.
> > need to be perfect - one just needs to know quite accurately how
> > imperfect they are.
>
> I'm not sure you can know that accurately when blind. That's
> actually probably my biggest problem with blind tests- the
> tester doesn't get to see the configuration file that could
> contain the backdoor from hell.[...]
I don't see that blind pen-testing can help much with a priori risk
assessment. What it _can_ help with is verification of the risk levels that
have been assigned to various problems. If you had "Hack the Webserver" as
"Trivial" and the pen-testers couldn't do it, either they're morons or you
overestimated the risk.
[...]
> > The way I see it, the ultimate goal of the FD Pentest is to break
> > systems. Contrast this with the goal of the blind pentest,
> which is to
> > find out if systems will break under a fairly "normal" attack
> > scenario.[...]
>
> I think this is where we diverge somewhat. As I see it (and
> I like the term verification testing) verification testing is
> the sharp pointy end of what an audit should be- something
> between validation that people are following procedures and
> assurance that equipment is operating correctly with a pinch
> of architecture review thrown in.
>
> Typically audits fall more on the procedure part and
> penetration tests and vulnerability scans fall into a
> slightly different category which I tend to think of as
> "poking sticks through known holes." They're only part of
> what should be the bigger picture, and they don't include
> architecture reviews that can show what sorts of things might
> let those stick through.
OK, I like this bit. You're pointing at a hole between process audits and
pentests, where there should be something like a "best practice audit". This
won't be covered by a pen-test, because that just tells you about what your
security is, and not about if you did it the right way. While some people
may believe that the final outcome is all that's important (and there's an
argument for that) I like to make design recommendations based on General
Solution Goodness. I get lots of people asking "What extra security to you
get doing it your way?" and I have to say "None that I can think of right
now - but that's the point. I know it's more correct, so in general it will
be more secure.".
I think I'm also now convinced that your definition of "Verification
Testing" above makes lots of sense for maximising system security. I'm still
not convinced that blind pen-testing isn't useful for refining the _current_
real-world accuracy of risk assessments.
[...]
> Certainly the entire "cost of compromise verses loss from
> security" thing should be someone's part of the security
> equation.
[...]
> I might just have to steal the term verification testing
> though, it's growing on me :)
>
> Paul
Cheers,
[1] www.ideahamster.org
--
Ben Nagy
Network Security Specialist
Marconi Services Australia Pty Ltd
Mb: +61 414 411 520 PGP Key ID: 0x1A86E304
_______________________________________________
Firewalls mailing list
[EMAIL PROTECTED]
http://lists.gnac.net/mailman/listinfo/firewalls
