[SC-L] OWASP Podcast #6
Hello SC-L I just pushed OWASP Podcast #6 live at http://www.owasp.org/index.php/Podcast_6 - an OWASP Roundtable with Brian Holyfield, Marcin Wielgoszewski, Andre Gironda and myself, Jim Manico. Our focus was WAF's. Thanks and I hope you enjoy, Jim Manico ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___
Re: [SC-L] Security in QA is more than exploits
For starters I believe you misinterpreted my comments on QA. I was in no way slamming their abilities. With this in mind comments below. Before anyone talks about vulnerabilities to test for, we have to figure ou= t what the business cares about and why. What could go wrong? Who cares? Wh= at would the impact be? Answers to those questions drive our testing strate= gy, and ultimately our test plans and test cases. We absolutely agree here. At the same time an externally exploitable sql injection needs to get fixed. The way qa/development is informed to its impact is through education likely via training. Not a single company with average hiring/skill requirements will have everybody (who needs to) know what sql injection is, and why it is bad. Bias #3 is that idea that a bunch of web vulnerabilities are equivalent in = impact to the business. That is, you just toss as many as you can into your= test plan and test for as much as you can. This isn't how testing is prior= itized. I said A better approach in my opinion is to identify the top 10/25/x attacks/weaknesses/vulnerabilities that are likely to affect your own organization These would be associated to customer or business impacts likely to affect you. Perhaps this could have been articulated better. As someone who has been training in risk-based security testing for several= years now, I totally agree with some points, but very much disagree with o= thers. I agree that the bug parade (as we call it) of top X vulnerabiliti= es to find is the wrong way to teach security testing. Risk management, tho= ugh, has been a fundamental part of mainstream QA for a very long time. Lik= ewise, risk management is the same technique that good security people us= e to prioritize their results. Risk management is certainly how the busines= s is going to make decisions about which issues to remediate and when. Risk= management is what ties this all together. We agree. If there's something that QA needs to learn that they're not already learni= ng, it's the weaving of security into the risk management techniques they= already know how to do. If testers fall short in their ability to apply ri= In your experience do you find average QA people doing risk management? In my experiences a Sr QA person/Team lead identifies what is going to be tested for a given release, and usually are the ones writing/tracking the test plans. So, in some ways we agree: speak the lingo of QA. But in other ways we disa= gree. I think the original article fails to give credit to the decades of s= ubstantial research and practice in QA. In other words, it's a lot more tha= n speaking the language. It is standing on the shoulders of giants, not the= ir toes. Actually the main goal of the article is that information security people need to set appropriate expectations as to what QA cares about as their primary business function. They need to factor in that the majority of QA people don't care about security as a primary job function, and that if infosec wants them to care they had better be prepared - to speak their language and understand their needs - to customize and prioritize the security testing they may be doing instead of solely using generic top x lists Paco Have a fantastic day Paco! Regards, - Robert ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___
Re: [SC-L] Security in QA is more than exploits
On Wed, Feb 4, 2009 at 11:17 AM, Paco Hope p...@cigital.com wrote: Before anyone talks about vulnerabilities to test for, we have to figure out what the business cares about and why. What could go wrong? Who cares? What would the impact be? Answers to those questions drive our testing strategy, and ultimately our test plans and test cases. Paco, I don't really read what Robert wrote this way. I think what this general risk management approach misses is that certain things are always going to be defects, bugs, etc. Sure there are differences per-business and per-application. All bugs aren't created equal. But I think we lose something when we start saying everything is relative. Each application, each business, each org needs a testing plan, strategy, and a definition of what they care about. At the same time there are going to be common types of tests that everyone performs. All Robert is pointing out is that if certain classes of vulnerabilities are important to you, then you want to have a common testing process for them. Bias #3 is that idea that a bunch of web vulnerabilities are equivalent in impact to the business. That is, you just toss as many as you can into your test plan and test for as much as you can. This isn't how testing is prioritized. Again, I don't think he's saying this at all. Where I work every XSS is absolutely critical, and we get them fixed immediately. this might not be the case elsewhere. Some folks don't really worry about XSS that much. Because I can find differences though doesn't mean that everything is relative. Authentication bypass, SQL Injection, these types of things tend to rate HIGH/P1/Major for almost everyone, and I think. You don't organize testing based on which top X vulnerabilities are likely to affect your organization (as the blog suggests). Likelihood is one part of the puzzle. Business impact is the part that is missing. You prioritize security tests by risk severity—that marriage of likelihood and impact to the business. If I have a whole pile of very likely attacks that are all low or negligible impact, and I have a few moderately likely attacks that have high impact, I should prioritize my testing effort around the ones with greater impact to my business. Again - fair enough. But at the same time you also prioritize around effort to test and avoid, right? Bias #4 is the treatment of testers like second class citizens. In the blog article, developers are detail oriented have a deep understanding of flows. Constrast this with QA who merely understand what is provided to them. They sound impotent, as if all they can do is what they're told. Software testing, despite whatever firsthand experience the author may have, is a mature discipline. It is older and more formalized than security as a discipline. Software testing is older than the Internet or the web. If software testing as a discipline has adopted security too slowly, given security's rise to the forefront in the marketplace, that might be a legitimate criticism. But I don't approve of the slandering QA by implying that they just take what's given them and execute it. QA is hard and there are some really bright minds working in that field. I don't think Robert's comments were about the general field/discipline of QA. His commentary was more about the types of QA organizations he has come across. My own experience (albeit limited as well) has found a relative lack of highly skilled QA folks as well. There are people responsible for quality that are at the level you're talking about but I still bet they are more the exception than the rule. Most QA organizations are staffed with people writing relatively simple tests, running through positive functional testing, etc. I think the point here is that you have to tailor expectations to the organization you have. Much in the same way that if you have mostly junior programmers who are lucky to get their code to compile you're probably not going to have a lot of luck training them on formal proofs, rigorous design, etc. -- Andy Steingruebl stein...@gmail.com ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___
Re: [SC-L] Security in QA is more than exploits
For starters I believe you misinterpreted my comments on QA. I was in no way slamming their abilities. With this in mind comments below. Sorry about that. I am sensitive to the bias. I went to a very small company once (10 people total) and as I looked around I saw offices with big LCDs (I assumed management) and cubicles with multi-core, multi-monitor setups (I assumed developers). Then I saw this old wooden table with 3 5-year-old HP desktops and a 15 tube monitor. I said to my host, the dev manager, that's the QA workstation. He looked surprised and said it was. He asked how I knew. I said because it's a piece of junk! This plays out a lot in industry, both big and little. So I apologize if I misread and found bias where none was intended. I'm ever vigilant against it. Before anyone talks about vulnerabilities to test for, we have to figure out what the business cares about and why. What could go wrong? Who cares? What would the impact be? Answers to those questions drive our testing strategy, and ultimately our test plans and test cases. We absolutely agree here. At the same time an externally exploitable sql injection needs to get fixed. Let me shock and appall people by saying not necessarily. It is commonly believed that some bugs are so horrific that we can say, without considering the business context, they must be fixed. 10 years ago we said this about buffer overflows. If you find a buffer overflow, you *must* fix it immediately. Then we went into industry and found out that there were times where missing a market window was far more costly than releasing a known buffer overflow. Ditto for the most horrendous web vulnerability you can think of. I resist absolute statements like this. As Andy Steingruebl pointed out you also prioritize around effort to test and avoid, right? Of course. We all agree that the cost of the fix is weighed against the benefits of not fixing and an estimate of the impact of successful exploitation. And that's why it's always possible you'll find a bug that sounds horrible, but is released anyways. Andy also said I think we lose something when we start saying 'everything is relative.' I think we lose something more important if we try to impose abolutes: we lose the connection to the business. No business operates on absolutes and blind imperatives. Few, if any, profit-focused businesses dogmatically fix all remotely exploitable SQL injections. Every business looks pragmatically at these things. Fixing the bug might cause the release of the product to slip by 6 weeks or a major customer to buy a competitor's product this quarter instead of waiting for the release. It's always a judgment call by the business. Even if their goal and their track record is fixing 100% of sev 1 issues before release, you know that each sev 1 issue was considered in terms of its cost, impact, schedule delay and so on. In your experience do you find average QA people doing risk management? Not all of them. Actually our experiences parallel nicely. My point is that any weak QA practitioners we're seeing in the marketplace are not QA folks who are short on security training. We're seeing QA folks who are short on QA training. When I find QA folks who are up-to-date on the state-of-the-practice in modern QA, teaching them a little security is a lot easier. When we go to teach security to folks who are already behind in their basics, we're building a castle on shaky ground. Actually the main goal of the article is that information security people need to set appropriate expectations as to what QA cares about as their primary business function. They need to factor in that the majority of QA people don't care about security as a primary job function, and that if infosec wants them to care they had better be prepared to speak their language and understand their needs So I'll continue to violently agree with you. :) QA is a process of taking inputs in the form of requirements (use cases, stories, etc.) and producing evidence of correct behavior (in both expected and unexpected situations). If infosec wants to give QA something they can consume and use directly, security requirements would be a great artifact. They fit the QA workflow, render explicit the security expectations, and foster traceability and test case development. It is an outstanding idea for infosec guys to provide security test cases, or the framework for them, to QA. That beats the heck out of what they usually do. However, a bunch of test cases for XSS, CSRF, SQL injection and so on will not map easily to requirements or to the QA workflow. At what priority do they execute? When the business (inevitably) squeezes testing to try to claw back a day or two on the slipped schedule, can any of these security tests be left out? Why or why not? Without hanging them into the QA workflow with clear traceability, QA will struggle to prioritize them correctly and maintain them. Security requirements would make
Re: [SC-L] Security in QA is more than exploits
On Wed, Feb 4, 2009 at 7:26 PM, Paco Hope p...@cigital.com wrote: Andy also said I think we lose something when we start saying 'everything is relative.' I think we lose something more important if we try to impose abolutes: we lose the connection to the business. No business operates on absolutes and blind imperatives. Few, if any, profit-focused businesses dogmatically fix all remotely exploitable SQL injections. Every business looks pragmatically at these things. Fixing the bug might cause the release of the product to slip by 6 weeks or a major customer to buy a competitor's product this quarter instead of waiting for the release. It's always a judgment call by the business. Even if their goal and their track record is fixing 100% of sev 1 issues before release, you know that each sev 1 issue was considered in terms of its cost, impact, schedule delay and so on. The ppint here though is that repeatable processes do matter. Having a standard of what constitutes a given severity of bug standardized in a policy statement is a good thing. Sure that is hard as every application is different, but you need a starting place. And so while my standards don't say XSS always equals P1 they do say XSS that can be discovered in an external facing application or even slightly more generically than that. So my bug priority matrix does talk about business impact because that is what matters, but I still have to give real world examples to folks who aren't expert security testers of how to handle a bug when they come across it. And we need to provide clear guidance in standards because every single bug shouldn't require an ad-hoc trage process. It is an outstanding idea for infosec guys to provide security test cases, or the framework for them, to QA. That beats the heck out of what they usually do. However, a bunch of test cases for XSS, CSRF, SQL injection and so on will not map easily to requirements or to the QA workflow. At what priority do they execute? When the business (inevitably) squeezes testing to try to claw back a day or two on the slipped schedule, can any of these security tests be left out? Why or why not? Without hanging them into the QA workflow with clear traceability, QA will struggle to prioritize them correctly and maintain them. Security requirements would make that priority and maintenance straightforward. At this point I'm not disagreeing with you, but taking your good approach and extending it a step farther. I undertsand this, but handing security requirements to QA folks without a set of reeatable test cases for doing them isn't going to help much, in mos organizations. James Whittaker doesn't work for me :) . And if you're developing web applications you're probably going to have some set of standardized testing you do. You need to have a repository of test cases for certain things, and I think testing for certain type of attacks is probably a decent starting point. Sure you want QA to own those, but if you're worried about buffer overflows you've going to have a bunch of standard test cases, test scenarios, test data (long input strings, inputs with null bytes in them, etc) that you're going to reuse a bunch of times so that each tester isn't starting from scratch when they see the security requirments - Application must handle input properly and not crash. I don't think we're far off here in what we're saying, but repeatability is key. Leaving the interface with QA at the level of security requriements in a functional spec isn't going to cut it. And, you're probably going to have some standardized set of security requirements for a whole swath of your applications that you might not want to repeat ad-naseum in every single product/feature spec. This is the place for standards, policies, and testing guidelines so that this becomes just part of the regular QA cycle. -- Andy Steingruebl stein...@gmail.com ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___
