I think the pie chart by vulnerability category is too simplistic a view. One also needs to look at the prevalence of each of the categories inspected for. If a tool was able to find the top 25 most prevalent categories of risk (as reported in CVE for 2006) it would be finding 70% of reported vulnerabilities while having a category coverage of less than 5%. A tool that could inspect for the 95% least prevalent vulnerability categories would only be finding 30% of the reported vulnerabilities. Which tool would you like to run on your code?
-Chris -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steven M. Christey Sent: Monday, December 17, 2007 6:07 PM To: McGovern, James F (HTSC, IT) Cc: Secure Coding Subject: Re: [SC-L] Interesting Blog Entry on Tools Coverage All, The original blog entry stems from a CWE pie chart that won't die until we replace it with a more well-grounded pie chart. We posted a followup here: http://www.matasano.com/log/912/finger-79tcp-christeymartin-evolution-of -the-cwe-pie-chart/ In short, CWE contains several types of nodes at multiple levels of abstraction, including general categories ("input validation problems") and arbitrary groupings ("problems related to memory management"). The original pie chart mixed these node types with 'real' weaknesses, and we included it in a CWE briefing as a demonstrative example of the utility of CWE in comparing code analysis tools. While that pie chart is still partially usable for showing a relative lack of overlap between tools (modulo the abstraction problem), the "only 45% of weakness types are found by tools" figure is probably low, since CWE currently has many nodes that are organizational in nature, so they would be excluded from any comparative analysis. (Although we're also probably relatively shallow with respect to design issues compared to implementation bugs, which might pull the numbers in another direction as CWE continues to fill in the gaps). As vaguely implied in the followup blog entry above, we will be working on a new pie chart with a better selection of CWE nodes, which should generate more credible numbers. We've been doing the ground work, e.g. explicitly identifying the types of nodes that could then be excluded from such analyses, but I can't be sure of when we'll have a new-and-improved pie chart. Rest assured that we are highly motivated to replace the existing chart, however, and I think we've learned our lesson about releasing "demonstrative statistics" in new technology areas that don't have any. - Steve _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________ _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________
