Hey Brian. The method you describe below is only scientifically valid if you're dealing with a system in a steady state. Think back to 10 years ago. If the same approach you describe below was used then, would have declared a false victory ages ago and moved on to scratching our heads as to why this wonderfully empirical model hadn't captured all the methods we needed to solve the problem.
Don't forget the whole reason why the 9 organizations you spoke with were doing *any* of those things was because experts used real experience with making improvements to specify what others should do. That's pretty much the way any system improves, I think. Perhaps I'm being overly critical, but your words seem to indicate contempt for what experts have actually done to improve this space when you use phrases like "people who have nothing but rhetoric and personal reputation" who "theorize" and "make stuff up". Sure, there are people in the space that fit that description at face value, but I suspect the majority that make real impacts actually use knowledge, real-life experience, and critical thinking to improve a system where no prior solutions are growing on the trees. As Einstein says, "We cannot solve our problems with the same thinking we used when we created them." All that being said, I think more data collection is useful in determining the state of the practice today. This is useful for us to set expectations and build a better plan for formulating future recommendations. But first, we just really need to (as Ben suggests) normalize the way in which the data is collected/measured and the sampling method, and all those other good things that experiments require. Thanks. p. On Wed, Mar 11, 2009 at 7:48 AM, Brian Chess <br...@fortify.com> wrote: > Ben! Thank you! When you talk about sample size, it gives me hope that > we’re on the right track. We can either: > > 1) Use ideas that “experts” theorize will work > or > 2) Gather empirical evidence to judge one idea against another. > > We in the security crowd often try to hide behind the need for secrecy, and > that’s pushed us toward relying almost entirely on people who have nothing > but rhetoric and personal reputation to stand on. BSIMM pretty well shows > that, in 2009, we can do better. It’s a big step forward to collect data > and then argue about what it means. I know it’s already made the rounds, > but let’s have some XKCD to celebrate: > http://xkcd.com/552/ > > I think your question about defining success is an important one. We were > loose about it in this first round, and I hope it’s something we can tighten > up in our follow-on work. Here’s my thinking as of today: software security > is not the goal, it’s one of the many things an organization needs to do in > order to meet it’s objectives. We need to look at how a software security > initiative (or lack thereof) effects the organization’s ability to meet it’s > objectives. This is going to be messy, but it’s either that or go back to > making stuff up. > > BTW, I checked the BSIMM web site after I read your message. It worked for > me. Try this? > http://www.downforeveryoneorjustme.com/bsi-mm.com > > Brian > > On 3/11/09 10:48 AM, "Benjamin Tomhave" <list-s...@secureconsulting.net> > wrote: > > I think it's an interesting leap of faith. Statistically speaking, 9 is > a very small sample size. Thus, the proposed model will be viewed > skeptically until it is validated with a much larger and more diverse > sample. Putting it another way, there's no way I can take this to a > small or medium sized org and have them see immediate relevance, because > their first reaction is going to be "those are 9 large orgs with lots of > resources - we don't have that luxury." > > You quoted "we can say with confidence that these activities are > commonly found in highly successful programs" - how do you define a > "highly successful program"? What's the rule or metric? Is this a rule > or metric that can be genericized easily to all development teams? > > My concern is exactly what you speculate about... organizations are > going to look at this and either try to tackle everything (and fail) or > decide there's too much to tackle (and quit). In my experience working > with maturity models as a consultant, very few people actually > understand the concept. Folks are far more tuned-in to a PCI-like > prescriptive method. Ironically, the PCI folks say the same thing you > did - that it's not meant to be prescriptive, that it's supposed to be > based on risk management practices - yet look how it's used. > > Maybe you've addressed this, but it doesn't sound like it. I'd perhaps > be better educated here if the web site wasn't down... ;) > > -ben > > Sammy Migues wrote: >> Hi Pravir, >> >> Thanks for clarifying what you're positing. I'm not sure how we could >> have been more clear in the BSIMM text accompanying the exposition of >> the collective activities about the need to take this information and >> work it into your own culture (i.e., do "risk management"). As a few >> examples: >> >> p. 3: "BSIMM is meant as a guide for building and evolving a software >> security initiative. As you will see when you familiarize yourself >> with the BSIMM activities, instilling software security into an >> organization takes careful planning and always involves broad >> organizational change. By clearly noting objectives and goals and by >> tracking practices with metrics tailored to your own initiative, you >> can methodically build software security in to your organization’s >> software development practices." >> >> p. 47: "Choosing which of the 110 BSIMM activities to adopt and in >> what order can be a challenge. We suggest creating a software >> security initiative strategy and plan by focusing on goals and >> objectives first and letting the activities select themselves. >> Creating a timeline for rollout is often very useful. Of course >> learning from experience is also a good strategy." >> >> p. 47: "Of the 110 possible activities in BSIMM, there are ten >> activities that all of the nine programs we studied carry out. Though >> we can’t directly conclude that these ten activities are necessary >> for all software security initiatives, we can say with confidence >> that these activities are commonly found in highly successful >> programs. This suggests that if you are working on an initiative of >> your own, you should consider these ten activities particularly >> carefully (not to mention the other 100)." >> >> p. 48: "The chart below shows how many of the nine organizations we >> studied have adopted various activities. Though you can use this as a >> rough “weighting” of activities by prevalence, a software security >> initiative plan is best approached through goals and objectives." >> >> Your words (...BSIMM fails...) imply (to me) that you posit >> organizations attempting to use the collected wisdom in BSIMM will, >> inexplicably, look at it and say "Okay, we have to do all 110 of >> these things exactly as written, so let's get started" without regard >> to their local need. This is as opposed to, say, looking at it and >> thinking "Here's what nine companies have spent dozens of >> person-decades and millions of dollars learning about what works; >> let's see what we can glean from that." Uhmmmm, okay. >> >> Yes, previous models exist. Although it may have come up in >> conversation, we did not ask any of the nine something like "What >> model did you start with back in the beginning?" because it simply >> isn't relevant to what we're trying to accomplish (documenting what >> successful organizations are doing), just as "could" and "should" >> aren't relevant. We asked "What *are* you doing now?" and documented >> it so others could learn from it. >> >> --Sammy. >> >> -----Original Message----- From: Pravir Chandra >> [mailto:chan...@list.org] Sent: Wednesday, March 11, 2009 4:00 AM To: >> Sammy Migues; sc-l-boun...@securecoding.org; sc-l@securecoding.org >> Subject: Re: [SC-L] Positive impact of an SSG >> >> Yes, I don't think its exclusive to your BSIMM interviews that you >> found when people put controls into place, they saw improvement. >> That's what I (and I'm sure many other consultanting firms) have been >> doing for years based upon previous models (CLASP, MS SDL, etc.). >> Nothing to do with BSIMM per se (actually, most of what DTCC started >> doing was based on CLASP), just that they added controls 'early into >> software development lifecycle' and saw benefit, which isn't >> surprising. >> >> That being said, the important part we're missing as 'software >> security guys' isn't the specification of all the possible things >> that an organization *could* do, but rather what a given organization >> *should* do based on good business decisions around risk management. >> And that's the crux of what BSIMM fails to do. By basing the current >> maturity model on the collected practices of 9 massive firms that >> spend the most on that problem, anyone (aside from firms in a similar >> situation to your 9) that attempts to apply it to their organization >> effectively throws risk management decisions out the window and >> commits to a much more costly solution than they could have created >> based on the knowledge of their own business needs since all the >> practices are based solely on the behaviors of the select few firms >> you interviewed. I'm not discounting the validity of the empirical >> data, I'm just positting that it isn't scientifically valid for >> solving the problem at hand. >> >> I'm interested to hear what you learn when you get to the small and >> medium sized businesses as well as firms using agile development >> models (something I particularly considered and accounted for with >> SAMM). >> >> Regardless of whether we agree on the percentage of orgs for which a >> dedicated SSG isn't cost effective, I'm sure we can agree that >> affording 'someone in charge of success' doesn't equate to a >> dedicated SSG. There's a myriad of ways that can be accomplished in >> any organizational structure. >> >> Thanks! >> >> p. >> >> ~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~ ~~~~~ ~~~ ~~ ~ Pravir >> Chandra chandra<at>list<dot>org PGP: CE60 >> 0E10 9207 7290 06EB 5107 4032 63FC 338E 16E4 ~ ~~ ~~~ ~~~~~ >> ~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~ >> >> -----Original Message----- From: Sammy Migues <smig...@cigital.com> >> >> Date: Tue, 10 Mar 2009 23:15:39 To: >> sc-l@securecoding.org<sc-l@securecoding.org> Subject: Re: [SC-L] >> Positive impact of an SSG >> >> >> Hi Pravir, >> >> Yes, I agree completely: the data gathered in the BSIMM interviews >> seem to indicate that "the controls over all" led to what the >> interviewees saw as improvements in their capability to produce >> secure software. >> >> In the nine companies interviewed, those controls (BSIMM activities, >> I think) sprang from well established SSGs -- that is, a specific >> person or persons with the responsibility for ensuring lots (110, >> collectively) of activities actually get done. >> >> The BSIMM data to date from specific large organizations indicate >> that a little under 100:1 is the average ratio for dev/QA to SSG >> size. It'll be interesting to see how this changes when we get to >> interviewing smaller organizations and we see if and how they're >> actually getting it done. >> >> Personally, I don't believe I agree with your guess that 95% of >> organizations building code can't afford an SSG. I believe every >> organization that wants to succeed can afford to have someone in >> charge of success, but that's just my opinion and isn't relevant to >> BSIMM. >> >> Cheers, >> >> --Sammy. >> >> >> -----Original Message----- From: Pravir Chandra >> [mailto:chan...@list.org] Sent: Tuesday, March 10, 2009 6:31 PM To: >> Sammy Migues Cc: sc-l@securecoding.org Subject: Re: [SC-L] Positive >> impact of an SSG >> >> Hey Sammy. >> >> How does that pertain to a software security group (SSG) per se? The >> details below seem to indicate that it was the controls over all that >> lead to the positive impact. >> >> My main point is that supporting an SSG isn't cost effective for 95% >> of the organizations out there that are building code. That's why in >> SAMM, we didn't mandate the structure of the organization and instead >> concentrated on the functions fulfilled by security guys (regardless >> of their placement in the org). >> >> p. >> >> On Tue, Mar 10, 2009 at 7:48 AM, Sammy Migues <smig...@cigital.com> >> wrote: >>> Hi all, >>> >>> I've received some private questions about the 110 activities in >>> BSIMM (bsi-mm.com). Since we built the model directly from the data >>> gathered, each activity is actually being done in one of the nine >>> organizations interviewed. The question is whether there's any >>> evidence the activities are actually effective as opposed to simply >>> being done. >>> >>> Since we can't publish any private data, I'd like to point folks at >>> this recent article in Information Security Magazine. Jim Routh, >>> CISO of DTCC (one of the nine organizations interviewed), is quoted >>> as follows relative to the impact of software security group >>> activities: >>> >>> >>> http://searchsecurity.techtarget.com/magazineFeature/0,296894,sid14_gci1346974,00.html >>> >>> >>> "One of Routh's big wins is inserting security controls early into >>> software development lifecycle at the DTCC. Vulnerabilities are >>> weeded out well before they appear in functional code that ends up >>> in production and that has resulted in close to $2 million in >>> productivity gains on a base of $150 million spend for development, >>> Routh says. >>> >>> "Those gains are exclusively the result of having mature and >>> effective controls within our system and software development >>> lifecycle," Routh says. This is a three-year-old initiative that >>> educates and certifies developers in all DTCC environments in >>> security. Developers are also provided with the necessary >>> code-scanning tools and consulting and services help to keep >>> production code close to pristine." >>> >>> --Sammy. >>> >>> Sammy Migues Principal, Technology 703.404.5830 - >>> http://www.cigital.com Software confidence. Achieved. >>> smig...@cigital.com >>> >>> >>> >>> _______________________________________________ Secure Coding >>> mailing list (SC-L) SC-L@securecoding.org List information, >>> subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List >>> charter available at - http://www.securecoding.org/list/charter.php >>> SC-L is hosted and moderated by KRvW Associates, LLC >>> (http://www.KRvW.com) as a free, non-commercial service to the >>> software security community. >>> _______________________________________________ >>> >> >> >> > > -- > Benjamin Tomhave, MS, CISSP > fal...@secureconsulting.net > LI: http://www.linkedin.com/in/btomhave > Blog: http://www.secureconsulting.net/ > Photos: http://photos.secureconsulting.net/ > Web: http://falcon.secureconsulting.net/ > > [ Random Quote: ] > "Don't you wish there were a knob on the TV to turn up the intelligence? > There's one marked 'Brightness,' but it doesn't work." > Gallagher > _______________________________________________ > Secure Coding mailing list (SC-L) SC-L@securecoding.org > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l > List charter available at - http://www.securecoding.org/list/charter.php > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) > as a free, non-commercial service to the software security community. > _______________________________________________ > > > _______________________________________________ > Secure Coding mailing list (SC-L) SC-L@securecoding.org > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l > List charter available at - http://www.securecoding.org/list/charter.php > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) > as a free, non-commercial service to the software security community. > _______________________________________________ > > -- ~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~ ~~~~~ ~~~ ~~ ~ Pravir Chandra chandra<at>list<dot>org PGP: CE60 0E10 9207 7290 06EB 5107 4032 63FC 338E 16E4 ~ ~~ ~~~ ~~~~~ ~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~ _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________
