hi jim, We chose organizations that in our opinion are doing a superior job with software security. You are welcome to disagree with our choices.
Microsoft has a shockingly good approach to software security that they are kind enough to share with the world through the SDL books and websites. Google has a much different approach with more attention focused on open source risk and testing (and much less on code review with tools). Adobe has a newly reinvigorated approach under new leadership that is making some much needed progress. The three firms that you cited were all members of the original nine whose data allowed us to construct the model. There are now 30 firms in the BSIMM study, and their BSIMM data vary as much as you might expect...about which more soon. gem company www.cigital.com podcast www.cigital.com/silverbullet blog www.cigital.com/justiceleague book www.swsec.com On 2/4/10 12:50 PM, "Jim Manico" <j...@manico.net> wrote: Why are we holding up the statistics from Google, Adobe and Microsoft ( http://www.bsi-mm.com/participate/ ) in BDSIMM? These companies are examples of recent "epic security failure". Probably the most financially damaging infosec attack, ever. Microsoft let a plain-vanilla 0-day slip through ie6 for years, Google has a pretty basic network segmentation and policy problem, and Adobe continues to be the laughing stock of client side security. Why are we holding up these companies as BDSIMM champions? - Jim > > On Wed, 3 Feb 2010, Gary McGraw wrote: > >> Popularity contests are not the kind of data we should count on. But >> maybe we'll make some progress on that one day. > > That's my hope, too, but I'm comfortable with making baby steps along > the way. > >>> Ultimately, I would love to see the kind of linkage between the >>> collected >>> data ("evidence") and some larger goal ("higher security" whatever THAT >>> means in quantitative terms) but if it's out there, I don't see it >> >> Neither do I, and that is a serious issue with models like the BSIMM >> that measure "second order" effects like activities. Do the >> activities actually do any good? Important question! > > And one we can't answer without more data that comes from the > developers who adopt any particular practice, and without some > independent measure of what success means. For example: I am a big > fan of the attack surface metric originally proposed by Michael Howard > and taken up by Jeanette Wing et al. at CMU (still need to find the > time to read Manadhata's thesis, alas...) It seems like common sense > that if you reduce attack surface, you reduce the number of security > problems, but how do you KNOW!? > >>> The 2010 OWASP Top 10 RC1 is more data-driven than previous >>> versions; same >>> with the 2010 Top 25 (whose release has been delayed to Feb 16, btw). >>> Unlike last year's Top 25 effort, this time I received several >>> sources of >>> raw prevalence data, but unfortunately it wasn't in sufficiently >>> consumable form to combine. >> >> I was with you up until that last part. Combining the prevalence >> data is something you guys should definitely do. BTW, how is the >> 2010 CWE-25 (which doesn't yet exist) more data driven?? > > I guess you could call it a more refined version of the "popularity > contest" that you already referred to (with the associated > limitations, and thus subject to some of the same criticisms as those > pointed at BSIMM): we effectively conducted a survey of a diverse set > of organizations/individuals from various parts of the software > security industry, asking what was most important to them, and what > they saw the most often. This year, I intentionally designed the Top > 25 under the assumption that we would not have hard-core quantitative > data, recognizing that people WANTED hard-core data, and that the few > people who actually had this data, would not want to share it. (After > all, as a software vendor you may know what your own problems are, but > you might not want to share that with anyone else.) > > It was a bit of a surprise when a handful of participants actually had > real data - but, then the problem I'm referring to with respect to > "consumable form" reared its ugly head. One third-party consultant > had statistics for a broad set of about 10 high-level categories > representing hundreds of evaluations; one software vendor gave us a > specific weakness history - representing dozens of different CWE > entries across a broad spectrum of issues, sometimes at very low > levels of detail and even branching into the GUI part of CWE which > almost nobody pays attention to - but "only" for 3 products. Another > vendor rep evaluated the dozen or two publicly-disclosed > vulnerabilities that were most severe according to associated CVSS > scores. Those three data sets, plus the handful of others based on > some form of analysis of hard-core data, are not merge-able. The irony > with CWE (and many of the making-security-measurable efforts) is that > it brings sufficient clarity to recognize when there is no clarity... > the "known unknowns" to quote Donald Rumsfeld. I saw this in 1999 in > the early days of CVE, too, and it's still going on - observers of the > oss-security list see this weekly. > > For data collection at such a specialized level, the situation is not > unlike the breach-data problem faced by the Open Security Foundation > in their Data Loss DB work - sometimes you have details, sometimes you > don't. The Data Loss people might be able to say "well, based on this > 100-page report we examined, we think it MIGHT have been SQL > injection" but that's the kind of data we're dealing with right now. > > Now, a separate exercise in which we compare/contrast the customized > top-n lists of those who have actually progressed to the point of > making them... that smells like opportunity to me. > >>> I for one am pretty satisfied with the rate at which things are >>> progressing and am delighted to see that we're finally getting some raw >>> data, as good (or as bad) as it may be. The data collection process, >>> source data, metrics, and conclusions associated with the 2010 Top >>> 25 will >>> probably be controversial, but at least there's some data to argue >>> about. >> >> Cool! > > To clarify to others who have commented on this part - I'm talking > specifically about the rate in which the software security industry > seems to be maturing, independently of how quickly the threat > landscape is changing. That's a whole different, depressing problem. > > - Steve > _______________________________________________ > Secure Coding mailing list (SC-L) SC-L@securecoding.org > List information, subscriptions, etc - > http://krvw.com/mailman/listinfo/sc-l > List charter available at - http://www.securecoding.org/list/charter.php > SC-L is hosted and moderated by KRvW Associates, LLC > (http://www.KRvW.com) > as a free, non-commercial service to the software security community. > _______________________________________________ _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________
