Hi Jim, Jim, thanks for the comments. It's a fair statement that pen tests don't just happen. There are many organizations who don't pay attention to application security at all - and they don't really fit in this model.
You're bang on about the lack of design activities. There just doesn't seem to be consistency here until people take a more meaningful approach like the SDL. That's not to say there aren't exceptions - like we mentioned in the posting, many organizations *do* have some sort of design or architecture assessment it just doesn't appear to be consistent in our observations. With respect to implementing metrics, I think this is a sign of maturity that means organizations are pulling away from a reactive approach. To keep the model simple, we've left out details about iterating although it's very important. Tool selection would typically be contained within the individual step to which the tool applies (e.g. static analysis within source code review). On Thu, Aug 11, 2011 at 1:58 PM, Jim Bird <jimb...@shaw.ca> wrote: > Hi Rohit, > > I just returned from overseas and read through the original post and this > email thread. If this Organic model is descriptive (based on what you've > observed at companies that you've done work for) then this progression seems > to make sense for companies who are working on a reactive basis, and > starting with outside help. I guess that it starts with consultant-based > work like pen testing and source code review, because the customers that > call in consultants would ask for this. Of course then a prerequisite would > be some kind of business case or risk assessment or other trigger (attack, > CEO reading scary things in the Wall Street Journal, ...) to bring in > consultants in the first place to see just how bad things are. Pen tests > don't just happen. > > As John Steven pointed out, there are other important steps like putting in > metrics and tracking, and implementing/upgrading tools/frameworks, and (in > my experience at least, an important early step) understanding (and later > tracking changes to) the attack surface. And iterating through all of this. > > I can see how Cigital's experience with larger enterprise customers that > drives BSIMM would be different, because these customers themselves would > drive additional requirements and have additional resources at hand, and > because Cigital has its own methods and engagement model and practices and > tools that it would bring into the customer. > > I am surprised to see that this model is so "code heavy/design lite": > there's little emphasis on threat modeling / ARA maybe because many > companies find it so hard to do? > > I like the idea of an end-state where security gets burned in to QA like > other problems in software development, making the team responsible for > security in reviews and testing etc. That's a big step to get to. > > /Jim > > > ----- Original Message ----- > From: Rohit Sethi <rkli...@gmail.com> > Date: Tuesday, July 19, 2011 4:18 pm > Subject: Re: The Organic Secure SDLC > To: John Steven <jste...@cigital.com> > Cc: Secure Code Mailing List <sc-l@securecoding.org>, "jimb...@shaw.ca" < > jimb...@shaw.ca>, Paco Hope <p...@cigital.com> > > > Hi John, > > > > Thanks for the feedback. This is exactly what we were looking > > for. We've > > certainly sought simplicity in this model, even at the expense being > > incomplete. It's not necessarily aimed at the one man shop - > > it's aimed at > > any organization where secure software is just not an explicit > > top-level > > priority. It doesn't address any of the short-comings of any > > previous model > > because it's not an alternative to them. It's simply an > > observation of a > > seemingly natural - organic- progression of steps. I agree > > with you about > > its value. No organization matches this model completely - there > > are often > > additional steps, some that you mentioned, which one > > organization or another > > takes or where the order is slightly different than what we've > > outlined. You > > can think of the steps we've outlined as a line of best fit: the > > steps we've > > seen to be most common. > > > > I'm often surprised to find security practitioners thinking they > > are way > > behind industry because they are are struggling to convince the > > lines of > > business to participate in security activities. One motivation > > for the model > > is to let those practitioners know that they're not alone. > > > > Case studies are a fantastic idea. We will add these to the > > model over time. > > We also want to be able to point to useful resources for people > > at each > > step, so if you (i.e. anyone reading this) has written relevant > > articles or > > whitepapers let me know. > > > > On Tue, Jul 19, 2011 at 4:43 PM, John Steven > > <jste...@cigital.com> wrote: > > > > > Paco, > > > > > > Thank you for cogently clarifying BSIMM. I'm a bit > > disappointed in the > > > community's ignorance regarding the model given it's both > > freely available > > > and Creative Commons licensed. Equally disappointing, to me, > > are positions > > > borne out of a "Just use [MyModel™: BSIMM || SAMM]" > > perspective. Rohit > > > asked: > > > > > > > If you're an actual practitioner who has lived through > > developing a > > > secure SDLC I'd love to hear your thoughts about the model's > > accuracy / > > > relevancy. > > > > > > Responses to this request would provide this mailing list's > > readership more > > > value. As one practitioner responsible for several SDL > > programs, I'll > > > respond ignoring Organic vs. BSIMM. I don't see much value in > > such a > > > comparison. > > > > > > [Is 'Organic' a model?] > > > > > > Yes. Paraphrasing one definition, a model is anything that > > abstracts a > > > system's factors in a way to helps its users quickly gain > > insight into the > > > subject's behavior. > > > > > > Inaccuracy isn't a fatal blow to a model, quoting Paul > > Wilmott's Manifesto > > > [BW1]: > > > > > > • I will remember > > that I didn't make the world and that > > > it > > doesn't satisfy my equations. > > > > > > • Though I will use > > models boldly to estimate value, I > > > will not > > be overly impressed by mathematics. > > > > > > • I will never > > sacrifice reality for elegance without > > > > > explaining why I have done so. Nor will I give the > > > people > > who use my model false comfort about its > > > > > accuracy. Instead, I will make explicit its > > > > > assumptions and oversights. > > > > > > • I understand that > > my work may have enormous effects > > > on > > society and the economy, many of them beyond my > > > comprehension. > > > > > > ...Navigators managed rather well with a "Flat Earth" > > hypothesis for some > > > time--no? So, we don't need over one hundred activities in our > > app. sec. > > > model in order to provide value. > > > > > > [Motivation] > > > Most of you know I respect Rohit a fair amount and so when I > > read his post, > > > you can imagine my thought, "In a world aware of BSIMM what is > > the value of > > > 'Organic'?" with honest curiosity, not disdain. I immediately > > guessed> 'Organic' was meant to address a common complaint > > regarding almost every > > > prior model: > > > > > > "I'm challenged > > applying <this> to smaller shops just > > > beginning > > their Application Security initiatives" > > > > > > Jim Bird has thoughtfully discussed the "one man shop" problem > > extensively> in his blog [JB1]. Rohit's own explanation mentions > > "no top down support" as > > > an indication of model applicability. > > > > > > [Accuracy] > > > 'Organic' ignores a lot of key components that even smaller > > shops already > > > have in place or care about improving. Three essential ones > > include 1) > > > measurement and iterative approach [JB2], 2) security policy > > [PC1], and 3) > > > security toolkits/frameworks [JB2][FM1]. While Rohit's post > > indicates> explicitly that things have been omitted, he focuses > > on having left out > > > architecture and related activities. > > > > > > To me, even if 'Organic' is designed to focus only on development > > > activities, ignoring a potential need for compliance to > > regulatory/security> policy, leveraging toolkits to make > > developers' jobs easier, or failing to > > > set up a measure-and-iterate loop are dire mistakes. I can > > point to small > > > organizations that have taken very different tacks and don't > > fit the model. > > > Some start with training. Others lean on SCR tools or security > > toolkits> before ever institutionalizing pen-testing. > > > > > > Perhaps it's inaccurate. Maybe it doesn't meet our industry > > need for > > > addressing "one man shop". So is it good-for-nothing? No. It's > > useful.> > > > [Value] > > > An immediate value that jumped out at me was the "Climb the > > Wall" element > > > [BS1]. I can again pull from experience organizations that > > don't fit the > > > model but the single most salient 'take home' from 'Organic', > > to me, is: > > > > > > You're going to need > > a champion, and until you "climb > > > the wall" you may > > hit barriers to progress. > > > > > > This concept seems to resonate with almost every security > > professional,> even if the form in which it comes can be contentious. > > > > > > Yet, often as vendors, internal champions, or <whatever> we > > don't always > > > make this requirement for progress explicit. We often > > underestimate its > > > cost, LoE, and impact. As a result things stall or slide backwards. > > > > > > Another key model elements that resonate with me is "Have a security > > > champion": > > > > > > > It's also worth pointing out that while the whole process grows > > > organically, at some stage it still requires a champion within the > > > organization to get the organization to move to the next step. > > That champion > > > usually has application security in their job description, but > > not always. > > > > > > BSIMM respondents handle this in consistent fashion [BS2] but > > Rohit's way > > > of explaining this is perhaps less foreboding to those getting > > their start > > > and anointing their first victim <cough> champion. > > > > > > There's also tremendous value (to the 'Organic' model) in > > admitting what I > > > think we all know implicitly: programs may have to "show need" > > (through> assessment) in order to progress. It's not shocking to > > see penetration > > > testing at the beginning of 'Organic'--" 'sploits always > > create splash" as I > > > say. > > > > > > [Conclusion] > > > To me, the 'Organic' model suffers from key inaccuracies due > > to omission. > > > As such, it doesn't particularly address the principal > > criticism of existing > > > models. Its value stems from simplicity and an potentially > > clear way to > > > drive its users through key tenets (my summary, not Rohit's): > > > > > > A) Anoint a champion > > > B) Show need > > > C) Educate execs > > > D) Drive assessment earlier in lifecycle > > > E) Bake assessment into BAU (QA) > > > > > > The ability to say these things succinctly to an organization > > starting its > > > Application Security journey provides value. I do not, > > however, believe that > > > we're going to see security assessment applied by business > > unit/product team > > > QA folk in a BAU scenario in the next 3-5 years, with the > > notable exception > > > of organizations like Microsoft. > > > > > > If 'Organic' was mine, I would attempt to amplify its > > positives by > > > converting it from a SDL Model to a method accompanied by case > > studies. I'd > > > drive it towards showing, backed by case study, how "climbing > > the wall" can > > > be accomplished most effectively. This is a tough nut and one worth > > > cracking. > > > > > > -jOHN > > > ---- > > > John Steven > > > Senior Director; Advanced Technology Consulting > > > Desk: 703.404.9293 x1204 Cell: 703.727.4034 > > > Key fingerprint = 4772 F7F3 1019 4668 62AD 94B0 AE7F > > EEF4 62D5 F908 > > > > > > twitter: @m1splacedsoul > > > Rss: http://feeds.feedburner.com/M1splacedOnTheWeb > > > web: http://www.cigital.com > > > Software Confidence. Achieved. > > > > > > > > > * [BW1] > > > > > > http://www.businessweek.com/magazine/content/09_02/b4115059823953_page_2.htm> > * [JB1] http://swreflections.blogspot.com/2009/04/opensamm-shows-way.html > > > * [JB2] > > > http://swreflections.blogspot.com/2011/06/safer-software- > > through-secure.html > > > * [PC1] https://www.pcisecuritystandards.org/security_standards/ > > > * [FM1] http://shiro.apache.org/ > > > * [BS1] Analogous to BSIMM SM1.3 "Educate Executives": > > > http://bsimm.com/online/governance/sm/?s=sm1.3#sm1.3 > > > * [BS2] http://www.informit.com/articles/article.aspx?p=1434903 > > > > > > On Jul 19, 2011, at 11:24 AM, Paco Hope wrote: > > > > > > > Jim, > > > > > > > > You're spot on. BSIMM is not a lifecycle for any company. > > Heck, it's not > > > even a set of recommendations. It's simply a way to measure > > what a firm > > > does. It's a model formulated from observations about how some > > firms'> implement software security in their lifecycles. You'll > > never catch us > > > calling the BSIMM a lifecycle. > > > > > > > > As for not translating into the SMB market, I don't > > understand that. > > > Unlike, say prescriptive standards which say "thou shalt do X" > > regardless of > > > how big you are, the BSIMM measures maturity of what a firm > > actually does. > > > There is no reason an SMB could not measure the maturity of > > their effort > > > using the BSIMM. > > > > > > > > Maturity is not a function of size. A team of 10 developers > > might score > > > higher on various criteria than a multi-national bank that has > > a whole team > > > of people dedicated to app sec. Maturity is a function of the > > depth to which > > > one takes a certain activity and their capability within that > > activity.> > > > > > This isn't Pac-Man, either. The goal is not to get the > > highest score and > > > an extra man. :) The goal is to put the right level of effort > > into the right > > > places. A firm can't do that until they know how much effort they're > > > spending on different activities. The BSIMM will illuminate > > the level of > > > effort. It allows a firm to decide to rebalance and spread the > > budget/people> around across the activities that make sense. > > Whether that's a team of 10 > > > developers or a team of 1000 developers, the principle is the > > same. The > > > execution varies. > > > > > > > > Here's another analogy. You can have a GPS and know your exact > > > coordinates, to within 3 meters, but not know how to get to > > the airport by > > > car. The BSIMM will tell you your coordinates at the present > > time. It does > > > not tell you the best way to the airport. It can tell you the > > crow-fly > > > distance to the airport, but it can't tell you that the > > airport is where you > > > want to be. > > > > > > > > Paco > > > > > > > > > > > > Paco, > > > > > > > > By your same logic I would not consider BSIMM a lifecycle > > either. It's > > > > a thermometer to measure an SDLC against what some some of > > the largest > > > > companies are doing. As others have noted, BSIMM does > > not translate > > > > well into the SMB market where most software is written. > > Don't get me > > > > wrong, BSIMM is very interesting data and is useful. But a > > > > comprehensive secure software lifecycle for every company it > > is not. > > > > > > > > - Jim Manico > > > > > > > > > > > -- > > Rohit Sethi > > SD Elements > > http://www.sdelements.com > > twitter: rksethi > > > -- Rohit Sethi SD Elements http://www.sdelements.com twitter: rksethi
_______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates _______________________________________________
