Why are we holding up the statistics from Google, Adobe and Microsoft (
http://www.bsi-mm.com/participate/ ) in BDSIMM?
These companies are examples of recent "epic security failure". Probably
the most financially damaging infosec attack, ever. Microsoft let a
plain-vanilla 0-day slip through ie6 for years, Google has a pretty
basic network segmentation and policy problem, and Adobe continues to be
the laughing stock of client side security. Why are we holding up these
companies as BDSIMM champions?
- Jim
On Wed, 3 Feb 2010, Gary McGraw wrote:
Popularity contests are not the kind of data we should count on. But
maybe we'll make some progress on that one day.
That's my hope, too, but I'm comfortable with making baby steps along
the way.
Ultimately, I would love to see the kind of linkage between the
collected
data ("evidence") and some larger goal ("higher security" whatever THAT
means in quantitative terms) but if it's out there, I don't see it
Neither do I, and that is a serious issue with models like the BSIMM
that measure "second order" effects like activities. Do the
activities actually do any good? Important question!
And one we can't answer without more data that comes from the
developers who adopt any particular practice, and without some
independent measure of what success means. For example: I am a big
fan of the attack surface metric originally proposed by Michael Howard
and taken up by Jeanette Wing et al. at CMU (still need to find the
time to read Manadhata's thesis, alas...) It seems like common sense
that if you reduce attack surface, you reduce the number of security
problems, but how do you KNOW!?
The 2010 OWASP Top 10 RC1 is more data-driven than previous
versions; same
with the 2010 Top 25 (whose release has been delayed to Feb 16, btw).
Unlike last year's Top 25 effort, this time I received several
sources of
raw prevalence data, but unfortunately it wasn't in sufficiently
consumable form to combine.
I was with you up until that last part. Combining the prevalence
data is something you guys should definitely do. BTW, how is the
2010 CWE-25 (which doesn't yet exist) more data driven??
I guess you could call it a more refined version of the "popularity
contest" that you already referred to (with the associated
limitations, and thus subject to some of the same criticisms as those
pointed at BSIMM): we effectively conducted a survey of a diverse set
of organizations/individuals from various parts of the software
security industry, asking what was most important to them, and what
they saw the most often. This year, I intentionally designed the Top
25 under the assumption that we would not have hard-core quantitative
data, recognizing that people WANTED hard-core data, and that the few
people who actually had this data, would not want to share it. (After
all, as a software vendor you may know what your own problems are, but
you might not want to share that with anyone else.)
It was a bit of a surprise when a handful of participants actually had
real data - but, then the problem I'm referring to with respect to
"consumable form" reared its ugly head. One third-party consultant
had statistics for a broad set of about 10 high-level categories
representing hundreds of evaluations; one software vendor gave us a
specific weakness history - representing dozens of different CWE
entries across a broad spectrum of issues, sometimes at very low
levels of detail and even branching into the GUI part of CWE which
almost nobody pays attention to - but "only" for 3 products. Another
vendor rep evaluated the dozen or two publicly-disclosed
vulnerabilities that were most severe according to associated CVSS
scores. Those three data sets, plus the handful of others based on
some form of analysis of hard-core data, are not merge-able. The irony
with CWE (and many of the making-security-measurable efforts) is that
it brings sufficient clarity to recognize when there is no clarity...
the "known unknowns" to quote Donald Rumsfeld. I saw this in 1999 in
the early days of CVE, too, and it's still going on - observers of the
oss-security list see this weekly.
For data collection at such a specialized level, the situation is not
unlike the breach-data problem faced by the Open Security Foundation
in their Data Loss DB work - sometimes you have details, sometimes you
don't. The Data Loss people might be able to say "well, based on this
100-page report we examined, we think it MIGHT have been SQL
injection" but that's the kind of data we're dealing with right now.
Now, a separate exercise in which we compare/contrast the customized
top-n lists of those who have actually progressed to the point of
making them... that smells like opportunity to me.
I for one am pretty satisfied with the rate at which things are
progressing and am delighted to see that we're finally getting some raw
data, as good (or as bad) as it may be. The data collection process,
source data, metrics, and conclusions associated with the 2010 Top
25 will
probably be controversial, but at least there's some data to argue
about.
Cool!
To clarify to others who have commented on this part - I'm talking
specifically about the rate in which the software security industry
seems to be maturing, independently of how quickly the threat
landscape is changing. That's a whole different, depressing problem.
- Steve
_______________________________________________
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc -
http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC
(http://www.KRvW.com)
as a free, non-commercial service to the software security community.
_______________________________________________
_______________________________________________
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
_______________________________________________
