[SC-L] ddj: beyond the badnessometer
Hi all, Is penetration testing good or bad? http://ddj.com/dept/security/18951 gem company www.cigital.com podcast www.cigital.com/silverbullet book www.swsec.com This electronic message transmission contains information that may be confidential or privileged. The information contained herein is intended solely for the recipient and use by any other party is not authorized. If you are not the intended recipient (or otherwise authorized to receive this message by the intended recipient), any disclosure, copying, distribution or use of the contents of the information is prohibited. If you have received this electronic message transmission in error, please contact the sender by reply email and delete all copies of this message. Cigital, Inc. accepts no responsibility for any loss or damage resulting directly or indirectly from the use of this email or its contents. Thank You. ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php
Re: [SC-L] ddj: beyond the badnessometer
On Thu, 13 Jul 2006, Gary McGraw wrote: Hi all, Is penetration testing good or bad? http://ddj.com/dept/security/18951 It's great, but penetration testing of the network assesment types is useless as it takes a picture of what the network look slike TODAY, while tomorrow it's a different network with different vulnerabilities. Automating the process is the way to go. As to software testing, it considerably depends on what you use. If you test with SATAN-comparable tools, well, you won't get far. gem company www.cigital.com podcast www.cigital.com/silverbullet book www.swsec.com This electronic message transmission contains information that may be confidential or privileged. The information contained herein is intended solely for the recipient and use by any other party is not authorized. If you are not the intended recipient (or otherwise authorized to receive this message by the intended recipient), any disclosure, copying, distribution or use of the contents of the information is prohibited. If you have received this electronic message transmission in error, please contact the sender by reply email and delete all copies of this message. Cigital, Inc. accepts no responsibility for any loss or damage resulting directly or indirectly from the use of this email or its contents. Thank You. ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php
Re: [SC-L] ddj: beyond the badnessometer
On Thu, Jul 13, 2006 at 07:56:16AM -0400, Gary McGraw wrote: Is penetration testing good or bad? http://ddj.com/dept/security/18951 Test coverage is an issue that penetration testers have to deal with, without a doubt. Pen-tests can never test every possible attack vector, which means that pen-tests can not always falsify a security assertion. Ok. But... First, pen-testers are highly active. The really good ones spend alot of time in the hacker community keeping up with the latest attack types, methods, and tools. Hence, the relevance of the test coverage you get from a skilled pen-tester is actually quite good. In addition, the tests run are similar to real attacks you're likely to see in the wild. Also, pen-testsing is often intelligent, focused, and highly motivated. After all, how would you like to have to go back to your customer with a blank report? And, the recommendations you get can be quite good because pen-testers tend to think about the entire deployment environment, instead of just the code. So, they can help you use technologies you already have to fix problems instead of having to write lots and lots of new code to fix them. All of these make pen-testing a valuable exercise for software environments to go through. Second, every software application in deployment has an associated level of acceptable risk. In many cases, the level of acceptable risk is low enough that penetration testing provides all the verificaton capabilities needed. In some cases, the level of acceptable risk is really low and even pen-testing is overkill. I do mostly code review work these days, but I find that pen-testing has more general applicability to my customers. There are exceptions, but not that many. Third, pen-tests also have real business advantages that don't directly address risk mitigation. Pen-test reports are typically more down to earth. That is, they can be read more easily and the attacks can be demonstrated more easily to business leaders, executives, and other stakeholders. In my experience, recommendations from both pen-tests and code reviews are commonly ignored. But, a good pen-test gets the executive blood flowing in a way that code-oriented security evaluations just don't. Fourth, assertion falsification isn't always what you're after. Being able to falsify the statement, this app is secure enough, is a common objective, but it's not really that useful for most businesses. What exactly is secure enough? How do you define it? How do you measure it? How much accuracy do you need? How do you get more accuracy, if you want it? How much do you trust your expert's opinion? Sometimes, it's better to simply demonstrate a positive assertion, such as: - This application is not subject to known, automatic attacks. - This application demonstrates the same security profile in all supported deployment environments. - This application demonstrates different security profiles, depending upon the deployment environment. - The latest MS patch does not affect the testable security profile of this application. These are all assertions that pen-testing is arguably pretty good for demonstrating. In some cases it might even be better than code anlaysis--e.g., the effects of new environments or upgrades to low-level libraries, virtual machines, operating systems. Finally, my freind Sam pointed out that only during some kind of pen-testing can you really identify what the actual attack surface of an application looks like in its final deployment environment. This is especially relevant in today's world because applications are now made as much through integration of existing, off-the-shelf components as through new development. A new application might only have a few thousand lines of original code, but might be resting on top of a software stack that has millions. Whether it's J2EE, .NET, or LAMP, all those environments are only really practical to test using some form of pen-test. Every security assessment methodology has its limits. Pen-testing has limited falsification capabilities. Code review, various kinds of code analysis, unit testing, whatever else. These methods can all have practical financial limitations and information accesibility problems. Of course, all these are good approaches and a wise security manager will employ as wide a variety of assessment methods as he can afford so that they compliment each other. But, affordability is a real concern for most busineses and pen-testing is pretty affordable. In the end, no assessment methodology produces results that are as good as having a skilled Security Developer on your team during the application design stage. Getting a security architecture in place that matches your risk tolerance and functional requirements is the single best way to prevent intrusions, bar none. nash e. foster Stratum Security, LLC -- the lyf so short, the craft so long to lerne. -
RE: [SC-L] ddj: beyond the badnessometer
Excellent post nash. Thanks!
I agree with you for the most part. You have a view of pen testing that
is quite sophisticated (especially compared to the usual drivel). I
agree with you so much that I included pen testing as the third most
important touchpoint in my new book Software Security www.swsec.com.
It is the subject of chapter 6. All the code review and architectural
risk analysis in the world can still be completely sidestepped by poor
decisions regarding the fielded software. Pen testing is ideal for
looking into that.
But there are two things I want to reiterate:
1) pen testing is a bad way to *start* working on software
security...you'll get much better traction with code review and
architectural risk assessment. {Of course, what nash says about the
power of a live sploit is true, and that kind of momentum creation may
be called for in a completely new situation where biz execs need basic
clue.}
2) pen testing can't tell you anything about how good your security is,
only how bad it is.
3) never use the results of a pen test as a punch list to attain
security
gem
company www.cigital.com
podcast www.cigital.com/silverbullet
book www.swsec.com
This electronic message transmission contains information that may be
confidential or privileged. The information contained herein is intended
solely for the recipient and use by any other party is not authorized. If
you are not the intended recipient (or otherwise authorized to receive this
message by the intended recipient), any disclosure, copying, distribution or
use of the contents of the information is prohibited. If you have received
this electronic message transmission in error, please contact the sender by
reply email and delete all copies of this message. Cigital, Inc. accepts no
responsibility for any loss or damage resulting directly or indirectly from
the use of this email or its contents.
Thank You.
___
Secure Coding mailing list (SC-L)
SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
RE: [SC-L] ddj: beyond the badnessometer
Although pentesting isn't perfect, I think in the right scope it has the
potential of acting in a vital role in the development lifecycle of a
project.
Building known attack patterns into a library which can be run against a
codebase has some merrit, as long as you understand what the resulting
expectations will be. As an example, I would consider automated
vulnerability assessment tools built to do input validation fuzzing to
be part of pentesting. And most pentesters out there use said tools for
that very reason.
I agree that pentesting at the START of a project is futile. Evaluating
the risks to the software by understanding its architecture, inputs and
flows goes much deeper and allows you to better find design flaws
instead of implementation bugs. But I am not so sure I would dismiss the
act of pentesting because of the badness-ometer factor. If we did, we
would also be dismissing things like static code analysis tools as they
may show similar results to many out there.
On a different tangent to this thread though, I don't think that the
BEST use of pentesting is for determining how secure your code is in the
first place. It is much more suited to allow you to stress test failure
code paths for different implementation configurations. No matter how
safe you write your code, pentesting can ferret out different scenarios
that come from deployment configuration problems. That is, if the
pentest tools and the user(s) of said tools know how to run through this
properly. As you mentioned in your article, too many people pass
themselves off as pentesting experts when they aren't. Just because they
CAN run Nessus doesn't mean they are good pentesters.
Its about using the right tool for the right job. As pentest tools
mature I think we will be able to use the growing attack libraries to
test against known patterns to eliminate the brain dead security bugs,
while allowing the tools to go deeper and ferret out problems as it
reaches more code coverage. The more we can automate that process and
make it part of our daily tests... the quicker it can expose 'known
problems' in a code base.
Can anyone recommend a tool that CAN tell you how good your security is?
I thought not.
Here I go quoting Spaf again.
When the code is incorrect, you can't really talk about security. When
the code is faulty, it cannot be safe.
Problem is, no tool in the world is going to show green blinky lights to
tell you the code is safe. Human heuristics come into play here, and we
have to leverage what assets we have, both manual and automatic, to find
the faulty code and eliminate it. And pentesting is just another one of
those tools in the arsenal to help.
Regards,
Dana Epp
[Microsoft Security MVP]
http://silverstr.ufies.org/blog/
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Gary McGraw
Sent: Thursday, July 13, 2006 8:05 AM
To: Nash
Cc: Secure Coding Mailing List
Subject: RE: [SC-L] ddj: beyond the badnessometer
Excellent post nash. Thanks!
I agree with you for the most part. You have a view of pen testing that
is quite sophisticated (especially compared to the usual drivel). I
agree with you so much that I included pen testing as the third most
important touchpoint in my new book Software Security www.swsec.com.
It is the subject of chapter 6. All the code review and architectural
risk analysis in the world can still be completely sidestepped by poor
decisions regarding the fielded software. Pen testing is ideal for
looking into that.
But there are two things I want to reiterate:
1) pen testing is a bad way to *start* working on software
security...you'll get much better traction with code review and
architectural risk assessment. {Of course, what nash says about the
power of a live sploit is true, and that kind of momentum creation may
be called for in a completely new situation where biz execs need basic
clue.}
2) pen testing can't tell you anything about how good your security is,
only how bad it is.
3) never use the results of a pen test as a punch list to attain
security
gem
company www.cigital.com
podcast www.cigital.com/silverbullet
book www.swsec.com
This electronic message transmission contains information that may be
confidential or privileged. The information contained herein is
intended solely for the recipient and use by any other party is not
authorized. If you are not the intended recipient (or otherwise
authorized to receive this message by the intended recipient), any
disclosure, copying, distribution or use of the contents of the
information is prohibited. If you have received this electronic message
transmission in error, please contact the sender by reply email and
delete all copies of this message. Cigital, Inc. accepts no
responsibility for any loss or damage resulting directly or indirectly
from the use of this email or its contents.
Thank You.
