"Before anyone talks about vulnerabilities to test for, we have to figure out 
what the business cares about and why. What could go wrong? Who cares? What 
would the impact be? Answers to those questions drive our testing strategy, and 
ultimately our test plans and test cases." 

We have to figure out what the __customer__ cares about and why.  Often times, 
the business areas don't have a clue about their customers.  The business areas 
throw web applications into the webiverse and hope someone will bite. What is 
going to keep customers? What is going to drive customers away? 

My 2 cents

Dave
 
David Wieneke, MSIT-IS, GSEC
 
IT Security Engineer
Security Operations and Administration
CUNA Mutual Group
dave.wien...@cunamutual.com
 

-----Original Message-----
From: sc-l-boun...@securecoding.org [mailto:sc-l-boun...@securecoding.org] On 
Behalf Of Paco Hope
Sent: Wednesday, February 04, 2009 1:18 PM
To: SC-L@securecoding.org
Subject: Re: [SC-L] Security in QA is more than exploits

All,

I just read Robert's blog entry about "re-aligning training expectations for 
QA." (http://bit.ly/157Pc3) It has some useful points that both developers and 
so-called "security people" need to hear. I disagree with some implicit biases, 
however, and I think we need to get past some stereotypes that sneak out in the 
article.

Bias #1, obviously, is the focus on the web. Despite its omnipresence, there is 
more non-web software than web software in the world, and non-web software does 
more important stuff than all the web software combined. The role of security 
in _software_ testing is vital, and the presence or absence of web technologies 
does not change that. Despite writing a recent book on Web Security Testing, I 
know my place in the universe. Quality assurance and software testing are 
disciplines far older than the web, and their mission is so much bigger than 
finding vulnerabilities.

Bias #2 is vulnerabilities ├╝ber alles. By talking about weaving vulnerabilities 
into security test plans, we've overlooked the first place where security goes 
into the QA process: test strategy. Look at any of the prominent folks in QA 
(Jon Bach, Michael Bolton, Rex Black, Cem Kaner), the people I'm privileged to 
share podiums with at QA conferences like STAR West, STAR East, and Better 
Software, and you'll see that security is part of the overall risk-based 
testing strategy. Risk-based testing has been around for a really long time. 
Longer than the web.

Before anyone talks about vulnerabilities to test for, we have to figure out 
what the business cares about and why. What could go wrong? Who cares? What 
would the impact be? Answers to those questions drive our testing strategy, and 
ultimately our test plans and test cases.

Bias #3 is that idea that a bunch of web vulnerabilities are equivalent in 
impact to the business. That is, you just toss as many as you can into your 
test plan and test for as much as you can. This isn't how testing is 
prioritized.

You don't organize testing based on which top X vulnerabilities are likely to 
affect your organization (as the blog suggests). Likelihood is one part of the 
puzzle. Business impact is the part that is missing. You prioritize security 
tests by risk severity-that marriage of likelihood and impact to the business. 
If I have a whole pile of very likely attacks that are all low or negligible 
impact, and I have a few moderately likely attacks that have high impact, I 
should prioritize my testing effort around the ones with greater impact to my 
business.

Bias #4 is the treatment of testers like second class citizens. In the blog 
article, developers are "detail oriented" have a "deep understanding of flows." 
Constrast this with QA who merely understand "what is provided to them." They 
sound impotent, as if all they can do is what they're told. Software testing, 
despite whatever firsthand experience the author may have, is a mature 
discipline. It is older and more formalized than "security" as a discipline. 
Software testing is older than the Internet or the web. If software testing as 
a discipline has adopted security too slowly, given security's rise to the 
forefront in the marketplace, that might be a legitimate criticism. But I don't 
approve of the slandering QA by implying that they just take what's given them 
and execute it. QA is hard and there are some really bright minds working in 
that field.

As someone who has been training in risk-based security testing for several 
years now, I totally agree with some points, but very much disagree with 
others. I agree that the "bug parade" (as we call it) of top X vulnerabilities 
to find is the wrong way to teach security testing. Risk management, though, 
has been a fundamental part of mainstream QA for a very long time. Likewise, 
risk management is the same technique that good "security people" use to 
prioritize their results. Risk management is certainly how the business is 
going to make decisions about which issues to remediate and when. Risk 
management is what ties this all together.

If there's something that QA needs to learn that they're not already learning, 
it's the weaving of "security" into the risk management techniques they already 
know how to do. If testers fall short in their ability to apply risk management 
techniques, then they are falling short against the QA yardstick, there's 
nothing particularly security-related in this observation. If they are applying 
mature QA practices with modern risk management, but are not adequately 
addressing the software-induced business risks facing their stakeholders, then 
some security training might be in order. That security training should be 
built on the foundation of modern QA practice, including risk-based testing.

So, in some ways we agree: speak the lingo of QA. But in other ways we 
disagree. I think the original article fails to give credit to the decades of 
substantial research and practice in QA. In other words, it's a lot more than 
speaking the language. It is standing on the shoulders of giants, not their 
toes.

My $0.02.

Paco
_______________________________________________
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
_______________________________________________


_______________________________________________
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
_______________________________________________

Reply via email to