All, I'm the editor of the Top 25 list. Thanks to Ken and others on SC-L who provided some amazing feedback before its publication. I hope we were able to address most of your concerns and am sorry that we couldn't address all of them.
Note that MITRE's site for the Top 25 is more technically detailed and has lots of supporting documents than the SANS site, which is really a jumping-off point. See http://cwe.mitre.org/top25/. Also, a process document and changelog is on that site. Here are some topics that arose during the construction of the Top 25. I thought these might make some interesting points of debate or discussion on this list: 1) The inclusion of output encoding (CWE-116) in conjunction with input validation (CWE-20) generated a lot of mixed reviews. Part of it seems to come down to different ways of looking at the same problem. For example, is SQL injection strictly an input validation vulnerability, or output sanitization/validation/encoding or whatever you want to call it? In a database, the name "O'Reilly" may pass your input validation step, but you need to properly quote it before sending messages to the database. And the actual database platform itself has no domain model to "validate" whether the incoming query is consistent with business logic. My personal thinking, which seems reflected by many web application people, is that many injection issues are related to encoding at their core, and the role of input validation is more defense-in-depth (WITH RESPECT TO INJECTION ONLY). Yet smart people insist that it's still input validation, even when presented with the example I gave. So So what's the perspective difference that's causing the disconnect? 2) Countless mitigations were suggested by contributors on top of some of the ones already in the CWE entries (admittedly some of them weak). Fortunately, we had time (for some definition of "time" that sometimes excluded a personal life) to update many of the core CWE entries. Many mitigations had limitations, either in terms of impacts on usability, whether they could be applied at all in some circumstances, or if they were sufficiently effective. The variety of customized mitigations is staggering, which to me suggests that more framework/methodology definition is needed. 3) Contributors advocated selecting list items based on how often the weakness appears in software (prevalence) and how severe the consequences are when the weakness leads to a vulnerable condition (severity). Many people advocated using real-world data to make the determination for prevalence. Problem: there's no real-world data available! CVE vulnerability data is insufficient - they concentrate on the vulnerability side ("XSS") instead of the weakness side ("e.g. use of the wrong encoding at the wrong time"). If people have real-world, weakness-focused data, then they aren't telling. 4) Some questions with respect to the assignment of severity scores led me to attempt to build a threat model and to try to more formally define other supporting fields like ease of detection, in light of the "skilled, determined attacker." I don't think this model was sufficiently vetted, and I'm sure people will have concerns with how it's been defined (including "your threat model is really just talking about a threat agent.") HOWEVER, I don't know of any other prioritized list that has tried to define some threat model to help with prioritization. I would love to see this kind of investigation continue in other efforts. (An acronym called CWSS comes to mind...) 5) The threat model, as roughly implied by how most people were "voting" for which items to include on the Top 25, treated availability as slightly less important than integrity and confidentiality. Thus CWE-400 (Resource Consumption) had the dubious distinction of being number 26. (CWE-400 and other also-rans are in the "On the Cusp" section of MITRE's Top 25 site). Clearly, availability may be more important in some environments e.g. critical infrastructure or e-commerce. The unsurprising implications are that no single threat model will work for different types of organizations when composing a general list like the Top 25. Thus it has a sort of fudge factor that helps make it generally applicable to organizations with varying threat environments, within some degree of tolerance. It seems like a fundamental problem with a list of that sort. 6) Many expert reviewers will notice the varying abstraction levels and overlapping concepts in the list. There are various explanations for this as summarized in the change logs and FAQ. My main point in bringing this up was, a lot of people want things to be at a fixed level of abstraction and mutually exclusive, but I don't think it's feasible given our current understanding of security issues. The CWE Research View (CWE-1000) tries to achieve mutual exclusivity, but are still working on it. One of the things I think the CWE project has brought to the table overall is the semi-formal modeling of how weaknesses are inter-related, in the form of chains and composites [1]. In short, it's difficult to have a mutually exclusive list because every kind of bug can trigger every other kind of bug! (example: buffer overflow could overwrite an adjacent numeric variable that leads to a calculation error, or vice versa). 7) Today, Chris Wysopal announced that Veracode had 61% coverage for the Top 25, and maybe they could boost that up to 75%. The gap is probably related to design-level issues. Chris noted that human analysis will always be required for the things that tools can't find, so software development teams need to find other approaches to fill the gap. I think this has interesting implications for future research and prioritization. 8) Terminology and definitions varied widely, which is probably a surprise to noone. This posed some problems in writing the document, and it's a general challenge for us in CWE and, I believe, a general challenge for the software community. 9) The OWASP Top Ten became a tool for PCI to easily establish minimum requirements for web applications, without PCI having to get into gory technical details. This was contrary to OWASP's original intentions for the Top Ten as an educational tool. SANS and MITRE both definitely have "technical clarification of expectations for assurance" as a goal, but the implications are a little unclear if this turns into "compliance." Nonetheless, I see it as an important tool for non-software-security experts to ask for security in terms that are amenable to some form of measurement. I grant that many of these issues may not be surprises to SC-L, but I thought they were interesting. - Steve [1] http://cwe.mitre.org/data/reports/chains_and_composites.html _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________
