"Before anyone talks about vulnerabilities to test for, we have to figure out
what the business cares about and why. What could go wrong? Who cares? What
would the impact be? Answers to those questions drive our testing strategy, and
ultimately our test plans and test cases."
We have to figure out what the __customer__ cares about and why. Often times,
the business areas don't have a clue about their customers. The business areas
throw web applications into the webiverse and hope someone will bite. What is
going to keep customers? What is going to drive customers away?
My 2 cents
Dave
David Wieneke, MSIT-IS, GSEC
IT Security Engineer
Security Operations and Administration
CUNA Mutual Group
dave.wien...@cunamutual.com
-Original Message-
From: sc-l-boun...@securecoding.org [mailto:sc-l-boun...@securecoding.org] On
Behalf Of Paco Hope
Sent: Wednesday, February 04, 2009 1:18 PM
To: SC-L@securecoding.org
Subject: Re: [SC-L] Security in QA is more than exploits
All,
I just read Robert's blog entry about "re-aligning training expectations for
QA." (http://bit.ly/157Pc3) It has some useful points that both developers and
so-called "security people" need to hear. I disagree with some implicit biases,
however, and I think we need to get past some stereotypes that sneak out in the
article.
Bias #1, obviously, is the focus on the web. Despite its omnipresence, there is
more non-web software than web software in the world, and non-web software does
more important stuff than all the web software combined. The role of security
in _software_ testing is vital, and the presence or absence of web technologies
does not change that. Despite writing a recent book on Web Security Testing, I
know my place in the universe. Quality assurance and software testing are
disciplines far older than the web, and their mission is so much bigger than
finding vulnerabilities.
Bias #2 is vulnerabilities über alles. By talking about weaving vulnerabilities
into security test plans, we've overlooked the first place where security goes
into the QA process: test strategy. Look at any of the prominent folks in QA
(Jon Bach, Michael Bolton, Rex Black, Cem Kaner), the people I'm privileged to
share podiums with at QA conferences like STAR West, STAR East, and Better
Software, and you'll see that security is part of the overall risk-based
testing strategy. Risk-based testing has been around for a really long time.
Longer than the web.
Before anyone talks about vulnerabilities to test for, we have to figure out
what the business cares about and why. What could go wrong? Who cares? What
would the impact be? Answers to those questions drive our testing strategy, and
ultimately our test plans and test cases.
Bias #3 is that idea that a bunch of web vulnerabilities are equivalent in
impact to the business. That is, you just toss as many as you can into your
test plan and test for as much as you can. This isn't how testing is
prioritized.
You don't organize testing based on which top X vulnerabilities are likely to
affect your organization (as the blog suggests). Likelihood is one part of the
puzzle. Business impact is the part that is missing. You prioritize security
tests by risk severity-that marriage of likelihood and impact to the business.
If I have a whole pile of very likely attacks that are all low or negligible
impact, and I have a few moderately likely attacks that have high impact, I
should prioritize my testing effort around the ones with greater impact to my
business.
Bias #4 is the treatment of testers like second class citizens. In the blog
article, developers are "detail oriented" have a "deep understanding of flows."
Constrast this with QA who merely understand "what is provided to them." They
sound impotent, as if all they can do is what they're told. Software testing,
despite whatever firsthand experience the author may have, is a mature
discipline. It is older and more formalized than "security" as a discipline.
Software testing is older than the Internet or the web. If software testing as
a discipline has adopted security too slowly, given security's rise to the
forefront in the marketplace, that might be a legitimate criticism. But I don't
approve of the slandering QA by implying that they just take what's given them
and execute it. QA is hard and there are some really bright minds working in
that field.
As someone who has been training in risk-based security testing for several
years now, I totally agree with some points, but very much disagree with
others. I agree that the "bug parade" (as we call it) of top X vulnerabilities
to find is the wrong way to teach security testing. Risk management, though,
has been a fundamental part of mainstream QA for a very long time. Likewise,
risk management is the same technique that good "security people" use to
prioritize their results. Risk management is certainly how the business is
going to make decisions
