"Before anyone talks about vulnerabilities to test for, we have to figure out what the business cares about and why. What could go wrong? Who cares? What would the impact be? Answers to those questions drive our testing strategy, and ultimately our test plans and test cases."
We have to figure out what the __customer__ cares about and why. Often times, the business areas don't have a clue about their customers. The business areas throw web applications into the webiverse and hope someone will bite. What is going to keep customers? What is going to drive customers away? My 2 cents Dave David Wieneke, MSIT-IS, GSEC IT Security Engineer Security Operations and Administration CUNA Mutual Group dave.wien...@cunamutual.com -----Original Message----- From: sc-l-boun...@securecoding.org [mailto:sc-l-boun...@securecoding.org] On Behalf Of Paco Hope Sent: Wednesday, February 04, 2009 1:18 PM To: SC-L@securecoding.org Subject: Re: [SC-L] Security in QA is more than exploits All, I just read Robert's blog entry about "re-aligning training expectations for QA." (http://bit.ly/157Pc3) It has some useful points that both developers and so-called "security people" need to hear. I disagree with some implicit biases, however, and I think we need to get past some stereotypes that sneak out in the article. Bias #1, obviously, is the focus on the web. Despite its omnipresence, there is more non-web software than web software in the world, and non-web software does more important stuff than all the web software combined. The role of security in _software_ testing is vital, and the presence or absence of web technologies does not change that. Despite writing a recent book on Web Security Testing, I know my place in the universe. Quality assurance and software testing are disciplines far older than the web, and their mission is so much bigger than finding vulnerabilities. Bias #2 is vulnerabilities über alles. By talking about weaving vulnerabilities into security test plans, we've overlooked the first place where security goes into the QA process: test strategy. Look at any of the prominent folks in QA (Jon Bach, Michael Bolton, Rex Black, Cem Kaner), the people I'm privileged to share podiums with at QA conferences like STAR West, STAR East, and Better Software, and you'll see that security is part of the overall risk-based testing strategy. Risk-based testing has been around for a really long time. Longer than the web. Before anyone talks about vulnerabilities to test for, we have to figure out what the business cares about and why. What could go wrong? Who cares? What would the impact be? Answers to those questions drive our testing strategy, and ultimately our test plans and test cases. Bias #3 is that idea that a bunch of web vulnerabilities are equivalent in impact to the business. That is, you just toss as many as you can into your test plan and test for as much as you can. This isn't how testing is prioritized. You don't organize testing based on which top X vulnerabilities are likely to affect your organization (as the blog suggests). Likelihood is one part of the puzzle. Business impact is the part that is missing. You prioritize security tests by risk severity-that marriage of likelihood and impact to the business. If I have a whole pile of very likely attacks that are all low or negligible impact, and I have a few moderately likely attacks that have high impact, I should prioritize my testing effort around the ones with greater impact to my business. Bias #4 is the treatment of testers like second class citizens. In the blog article, developers are "detail oriented" have a "deep understanding of flows." Constrast this with QA who merely understand "what is provided to them." They sound impotent, as if all they can do is what they're told. Software testing, despite whatever firsthand experience the author may have, is a mature discipline. It is older and more formalized than "security" as a discipline. Software testing is older than the Internet or the web. If software testing as a discipline has adopted security too slowly, given security's rise to the forefront in the marketplace, that might be a legitimate criticism. But I don't approve of the slandering QA by implying that they just take what's given them and execute it. QA is hard and there are some really bright minds working in that field. As someone who has been training in risk-based security testing for several years now, I totally agree with some points, but very much disagree with others. I agree that the "bug parade" (as we call it) of top X vulnerabilities to find is the wrong way to teach security testing. Risk management, though, has been a fundamental part of mainstream QA for a very long time. Likewise, risk management is the same technique that good "security people" use to prioritize their results. Risk management is certainly how the business is going to make decisions about which issues to remediate and when. Risk management is what ties this all together. If there's something that QA needs to learn that they're not already learning, it's the weaving of "security" into the risk management techniques they already know how to do. If testers fall short in their ability to apply risk management techniques, then they are falling short against the QA yardstick, there's nothing particularly security-related in this observation. If they are applying mature QA practices with modern risk management, but are not adequately addressing the software-induced business risks facing their stakeholders, then some security training might be in order. That security training should be built on the foundation of modern QA practice, including risk-based testing. So, in some ways we agree: speak the lingo of QA. But in other ways we disagree. I think the original article fails to give credit to the decades of substantial research and practice in QA. In other words, it's a lot more than speaking the language. It is standing on the shoulders of giants, not their toes. My $0.02. Paco _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________ _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________
