On Wed, 3 Feb 2010, Gary McGraw wrote:
Popularity contests are not the kind of data we should count on. But
maybe we'll make some progress on that one day.
That's my hope, too, but I'm comfortable with making baby steps along the
way.
Ultimately, I would love to see the kind of linkage between the collected
data ("evidence") and some larger goal ("higher security" whatever THAT
means in quantitative terms) but if it's out there, I don't see it
Neither do I, and that is a serious issue with models like the BSIMM
that measure "second order" effects like activities. Do the activities
actually do any good? Important question!
And one we can't answer without more data that comes from the developers
who adopt any particular practice, and without some independent measure of
what success means. For example: I am a big fan of the attack surface
metric originally proposed by Michael Howard and taken up by Jeanette Wing
et al. at CMU (still need to find the time to read Manadhata's thesis,
alas...) It seems like common sense that if you reduce attack surface,
you reduce the number of security problems, but how do you KNOW!?
The 2010 OWASP Top 10 RC1 is more data-driven than previous versions; same
with the 2010 Top 25 (whose release has been delayed to Feb 16, btw).
Unlike last year's Top 25 effort, this time I received several sources of
raw prevalence data, but unfortunately it wasn't in sufficiently
consumable form to combine.
I was with you up until that last part. Combining the prevalence data
is something you guys should definitely do. BTW, how is the 2010 CWE-25
(which doesn't yet exist) more data driven??
I guess you could call it a more refined version of the "popularity
contest" that you already referred to (with the associated limitations,
and thus subject to some of the same criticisms as those pointed at
BSIMM): we effectively conducted a survey of a diverse set of
organizations/individuals from various parts of the software security
industry, asking what was most important to them, and what they saw the
most often. This year, I intentionally designed the Top 25 under the
assumption that we would not have hard-core quantitative data, recognizing
that people WANTED hard-core data, and that the few people who actually
had this data, would not want to share it. (After all, as a software
vendor you may know what your own problems are, but you might not want to
share that with anyone else.)
It was a bit of a surprise when a handful of participants actually had
real data - but, then the problem I'm referring to with respect to
"consumable form" reared its ugly head. One third-party consultant had
statistics for a broad set of about 10 high-level categories representing
hundreds of evaluations; one software vendor gave us a specific weakness
history - representing dozens of different CWE entries across a broad
spectrum of issues, sometimes at very low levels of detail and even
branching into the GUI part of CWE which almost nobody pays attention to -
but "only" for 3 products. Another vendor rep evaluated the dozen or two
publicly-disclosed vulnerabilities that were most severe according to
associated CVSS scores. Those three data sets, plus the handful of others
based on some form of analysis of hard-core data, are not merge-able.
The irony with CWE (and many of the making-security-measurable efforts) is
that it brings sufficient clarity to recognize when there is no clarity...
the "known unknowns" to quote Donald Rumsfeld. I saw this in 1999 in the
early days of CVE, too, and it's still going on - observers of the
oss-security list see this weekly.
For data collection at such a specialized level, the situation is not
unlike the breach-data problem faced by the Open Security Foundation in
their Data Loss DB work - sometimes you have details, sometimes you don't.
The Data Loss people might be able to say "well, based on this 100-page
report we examined, we think it MIGHT have been SQL injection" but that's
the kind of data we're dealing with right now.
Now, a separate exercise in which we compare/contrast the customized top-n
lists of those who have actually progressed to the point of making them...
that smells like opportunity to me.
I for one am pretty satisfied with the rate at which things are
progressing and am delighted to see that we're finally getting some raw
data, as good (or as bad) as it may be. The data collection process,
source data, metrics, and conclusions associated with the 2010 Top 25 will
probably be controversial, but at least there's some data to argue about.
Cool!
To clarify to others who have commented on this part - I'm talking
specifically about the rate in which the software security industry seems
to be maturing, independently of how quickly the threat landscape is
changing. That's a whole different, depressing problem.
- Steve
_______________________________________________
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
_______________________________________________
