Re: [SC-L] [SAMM] NIST SP 800-37

[McGovern, James F. (eBusiness)]

I like your take. Maybe the SAMM team could provide formal commentary to
NIST in this regard. I suspect that in not providing feedback, it will
be published and those who read it at a later date will get confused as
to the value proposition of each aka more disturbances in the force...



From: samm-boun...@lists.owasp.org [mailto:samm-boun...@lists.owasp.org]
On Behalf Of Bart De Win
Sent: Wednesday, February 03, 2010 2:51 PM
To: Software Assurance Maturity Model (SAMM); SecureCode Mailing List
Subject: Re: [SAMM] NIST SP 800-37



James,

 

I'm not familiar (yet) with the details of SP800-37.

 

However, to add another NIST SP document to this discussion, SP 800-64
(R2 of October 2008) is definitely also worth looking at in the secure
development lifecycle context. Imho, from a bird's eye view, the main
differences between SP 800-64 and SAMM/BSIMM are:

-  The NIST model is a process model, while SAMM and BSIMM are
maturity models. This is a fundamentally different. In that sense, it is
more related to the SDL/CLASP/TouchPoint type of models.

-  In the same line of reasoning, the NIST model is
waterfall-based, while SAMM and BSIMM are actually process agnostic
(they can be applied to waterfall, agile and other types of processes)

-  NIST SP 800-64 focuses much more on deployment, operations
and disposal than any of the other models that I've seen so far.

 

I'd be also interested in hearing any other opinions about this one.

 

Best regards,

Bart.

 


Bart De Win CSSLP
Principal Consultant, CC Leader Application Assurance
Tel.: +32 (0)9 243.10.20, Mob: +32 (0)479 46.79.57

"Ascure, demonstrating excellence in operational risk management"

Looking for world class education? Check-out www.ascureacademy.eu
  and www.bcmacademy.be
 .



This message may be confidential. It is also solely for the use of the
individual or group to whom it is addressed. If you have received it by
mistake, please let us know by e-mail reply. Ascure is not liable for
any direct or indirect damage arising from errors, inaccuracies or any
loss in the message, from unauthorized use, disclosure, copying or
alteration of it. 

For the complete version or other languages of this disclaimer see
http://www.ascure.com/disclaimer.htm
  

 

From: samm-boun...@lists.owasp.org [mailto:samm-boun...@lists.owasp.org]
On Behalf Of McGovern, James F. (eBusiness)
Sent: woensdag 3 februari 2010 19:13
To: Secure Code Mailing List; Software Assurance Maturity Model (SAMM)
Subject: [SAMM] NIST SP 800-37

 

NIST has created a draft document entitled: Guide for applying risk
management framework to federal information systems: a security
lifecycle approach. Curious to know if anyone has identified gaps,
differences in opinion, etc between NIST and how either SAMM or BSIMM
would define the same?


This communication, including attachments, is for the exclusive use of
addressee and may contain proprietary, confidential and/or privileged
information.  If you are not the intended recipient, any use, copying,
disclosure, dissemination or distribution is strictly prohibited.  If
you are not the intended recipient, please notify the sender immediately
by return e-mail, delete this communication and destroy all copies.


This communication, including attachments, is for the exclusive use of 
addressee and may contain proprietary, confidential and/or privileged 
information.  If you are not the intended recipient, any use, copying, 
disclosure, dissemination or distribution is strictly prohibited.  If you are 
not the intended recipient, please notify the sender immediately by return 
e-mail, delete this communication and destroy all copies.

___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___

Re: [SC-L] BSIMM update (informIT)

[Gary McGraw]

hi steve,

It's BSIMM Begin and we are delinquent looking into the data.  Hope to do that 
soon.  We have 75 partial vectors in the set (including some "control" vectors 
from full BSIMM participants).

Anyone who wants to help us "top off" the data...(I was hoping to gather 100 
vectors)...click here:

http://bsi-mm.com/begin

gem

company www.cigital.com
podcast www.cigital.com/silverbullet
blog www.cigital.com/justiceleague
book www.swsec.com



On 2/2/10 7:23 PM, "Steven M. Christey"  wrote:



On Tue, 2 Feb 2010, Arian J. Evans wrote:

> BSIMM is probably useful for government agencies, or some large
> organizations. But the vast majority of clients I work with don't have
> the time or need or ability to take advantage of BSIMM. Nor should
> they. They don't need a software security group.

I'm looking forward to what BSIMM Basic discovers when talking to small
and mid-size developers.  Many of the questions in the survey PDF assume
that the respondent has at least thought of addressing software security,
but not all questions assume the presence of an SSG, and there are even
questions about the use of general top-n lists vs. customized top-n lists
that may be informative.

- Steve
___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___


___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___

Re: [SC-L] BSIMM update (informIT)

[Gary McGraw]

Hi Arian,

Some more particulars regarding your posting.  Sorry for the delay...

On 2/2/10 4:32 PM, "Arian J. Evans"  wrote:
>Strategic folks (VP, CxO) ...Initially ...ask for descriptive information, but 
>once they get
>going they need strategic prescriptions.

Please see my response to Kevin.  I hope it's clear what the BSIMM is for.  
It's for measuring your initiative and comparing it to others.  Given some 
solid BSIMM data, I believe you can do a superior job with strategy...and 
results measurement.  It is a tool for strategic people to use to build an 
initiative that works.

>Tactical folks tend to ask:
>+ What should we fix first? (prescriptive)
>+ What steps can I take to reduce XSS attack surface by 80%?

The BSIMM is not for tactical folks.  But should you base your decision 
regarding "what to fix first" on goat sacrifice?  What should drive that 
decision?  Moon phase?

> Implementation level folks ask:
>+ What do I do about this specific attack/weakness?
>+ How do I make my compensating control (WAF, IPS) block this specific attack?

BSIMM != code review tool, top-n list, book, coding experience, ...

>BSIMM is probably useful for government agencies, or some large
>organizations. But the vast majority of clients I work with don't have
>the time or need or ability to take advantage of BSIMM. Nor should
>they. They don't need a software security group.

Where to start.  All I can say about BSIMM so far is that is appears to be 
useful for 30 large commercial organizations carrying out real software 
security initiatives.  We have studied 0 (count 'em...none) government 
organizations to date.  In my experience, the government is always lagging when 
it comes to software security.  I'm hoping to gather some government data forth 
with, starting with the US Air Force.  We shall see.

But what about SMB (small to medium sized business)?  Arian, who are your 
clients?  How many developers do they have?  Who do you report to as a 
consultant?  How do you help them make business decisions?

Regarding the existence of an SSG, see this article 
.  Are your customers 
too small to have an SSG?  Are YOU the SSG?  Are your customers not mature 
enough for an SSG?  Data would be great.

>I agree and strongly empathize with Gary on many
>premises of his article - including that not many folks have metrics,
>and tend to have more faith and magic.

Sadly I think we're stuck with "second order" metrics like the BSIMM.  Heck, we 
even studied the metrics that real initiatives use in the BSIMM (bugs per 
square inch anyone?), but you know what?  Everyone has different metrics.  
Really.

>But, as should be no surprise, I cateogrically disagree with the
>entire concluding paragraph of the article. Sadly it's just more faith
>and magic from Gary's end. We all can do better than that.

You guys and your personal attacks.  Yeesh.  I am pretty sure you meant the 
"next to last" paragraph, because Feynman wrote the entire last one.  Here is 
the next to last one:

As I have said before, the time has come to put away the bug parade boogeyman 
, the top 25 tea 
leaves , black box web 
app goat sacrifice, and the occult reading of pen testing entrails. It's 
science time.  And the more descriptive and data driven we are, the better.

Can you be more specific about your disagreements please?  Did you read 
articles at the end of the pointers?  Where am I wrong?  Better yet, why?

We'll just ignore the Nader > Feynman stuff.

gem

company www.cigital.com
podcast www.cigital.com/silverbullet
blog www.cigital.com/justiceleague
book www.swsec.com


On Tue, Feb 2, 2010 at 9:30 AM, Wall, Kevin  wrote:
> On Thu, 28 Jan 2010 10:34:30 -0500, Gary McGraw wrote:
>
>> Among other things, David [Rice] and I discussed the difference between
>> descriptive models like BSIMM and prescriptive models which purport to
>> tell you what you should do.  I just wrote an article about that for
>> informIT.  The title is
>>
>> "Cargo Cult Computer Security: Why we need more description and less
>> prescription."
>> http://www.informit.com/articles/article.aspx?p=1562220
>
> First, let me say that I have been the team lead of a small Software
> Security Group (specifically, an Application Security team) at a
> large telecom company for the past 11 years, so I am writing this from
> an SSG practitioner's perspective.
>
> Second, let me say that I appreciate descriptive holistic approaches to
> security such as BSIMM and OWASP's OpenSAMM. I think they are much
> needed, though seldom heeded.
>
> Which brings me to my third point. In my 11 years of experience working
> on this SSG, it is very rare that application development teams are
> looking for a _descriptive_ approach. Almost always, they are
> looking for a _prescriptive_ one. They want specific solutions
> to specific p

Re: [SC-L] BSIMM update (informIT)

[Gary McGraw]

Hi Steve (and sc-l),

I'll invoke my skiing with Eli excuse again on this thread as well...

On Tue, 2 Feb 2010, Wall, Kevin wrote:
> To study something scientifically goes _beyond_ simply gathering
> observable and measurable evidence. Not only does data needs to be
> collected, but it also needs to be tested against a hypotheses that offers
> a tentative *explanation* of the observed phenomena;
> i.e., the hypotheses should offer some predictive value.

On 2/2/10 4:12 PM, "Steven M. Christey"  wrote:
>>I believe that the cross-industry efforts like BSIMM, ESAPI, top-n lists,
>>SAMATE, etc. are largely at the beginning of the data collection phase.

I agree 100%.  It's high time we gathered some data to back up our claims.  I 
would love to see the top-n lists do more with data.

Here's an example.  In the BSIMM,  10 of 30 firms have built top-N bug lists 
based on their own data culled from their own code.  I would love to see how 
those top-n lists compare to the OWASP top ten or the CWE-25.  I would also 
love to see whether the union of these lists is even remotely interesting.  One 
of my (many) worries about top-n lists that are NOT bound to a particular code 
base is that the lists are so generic as to be useless and maybe even unhelpful 
if adopted wholesale without understanding what's actually going on in a 
codebase. [see ].

Note for the record that "asking lots of people what they think should be in 
the top-10" is not quite the same as taking the union of particular top-n lists 
which are tied to particular code bases.  Popularity contests are not the kind 
of data we should count on.  But maybe we'll make some progress on that one day.

>Ultimately, I would love to see the kind of linkage between the collected
>data ("evidence") and some larger goal ("higher security" whatever THAT
>means in quantitative terms) but if it's out there, I don't see it

Neither do I, and that is a serious issue with models like the BSIMM that 
measure "second order" effects like activities.  Do the activities actually do 
any good?  Important question!

>The 2010 OWASP Top 10 RC1 is more data-driven than previous versions; same
>with the 2010 Top 25 (whose release has been delayed to Feb 16, btw).
>Unlike last year's Top 25 effort, this time I received several sources of
>raw prevalence data, but unfortunately it wasn't in sufficiently
>consumable form to combine.

I was with you up until that last part.  Combining the prevalence data is 
something you guys should definitely do.  BTW, how is the 2010 CWE-25 (which 
doesn't yet exist) more data driven??

>I for one am pretty satisfied with the rate at which things are
>progressing and am delighted to see that we're finally getting some raw
>data, as good (or as bad) as it may be.  The data collection process,
>source data, metrics, and conclusions associated with the 2010 Top 25 will
>probably be controversial, but at least there's some data to argue about.

Cool!

>So in that sense, I see Gary's article not so much as a clarion call for
>action to a reluctant and primitive industry, but an early announcement of
>a shift that is already underway.

Well put.

gem

company www.cigital.com
podcast www.cigital.com/~gem
blog www.cigital.com/justiceleague
book www.swsec.com


___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___

Re: [SC-L] BSIMM update (informIT)

[Gary McGraw]

Hi again Mike,

Yadda yadda, delay, and so on...

On 2/2/10 9:30 PM, "Mike Boberski"  wrote:
 But the vast majority of clients I work with don't have 
the time or need or ability to >take advantage of BSIMM

 >>Mike's Top 5 Web Application Security Countermeasures:
>>1. Add a security guy or gal who has a software development background to 
>>your application's software >>development team.

Dang, this would have saved Microsoft lots of money.  With 30,000 developers 
that security gal would have been pretty busy though.

>>3. Build an Enterprise Security API (a.k.a. an ESAPI, e.g. OWASP's several 
>>different ESAPI toolkits) that is >>specific to your solution stack and 
>>minimally provides input validation controls that use whitelists, output 
encoding/escaping controls (optionally use parameterized interfaces for 
>>SQL), and authentication controls. >>Build your ESAPI to target a specific 
>>level of overall security when all of your security controls are viewed as 
a whole (e.g. an OWASP Application Security Verification Standard (ASVS) 
>>level).

Why do you believe that an ESAPI (which is a good idea) is the best place to 
start?  Why not training?  Why not pen testing by Mike?  Etc.  This was not 
"job 1" in any firm I have been involved with.

>>4. Write a programming manual (i.e. a secure coding standard that is specific 
>>to your solution stack that is >>organized by vulnerability type or security 
>>requirement with before and after code snippets, e.g. a >>cookbook that 
>>provides before and after code snippets and links to API documentation) that 
>>contains step->>by-step instructions for using your ESAPI to both proactively 
>>guard against vulnerabilities, and to act as a >>quick reference when the 
>>time comes to make fixes.

Again.  How does this fit into a bigger picture?  The notion of code guidelines 
is a good one.  See [CR2.1] in the BSIMM which 11 of 30 companies we observed 
carry out.  This was not "job 2" in any case I am aware of.  How about tying 
such guidance to code review technology.  We've helped multiple clients do that.

How many customers have followed Mike's Way?  What are their results?  How do 
the Mike's Way customers score with the BSIMM?

gem

company www.cigital.com
podcast www.cigital.com/silverbullet
blog www.cigital.com/justiceleague
book www.swsec.com


On Tue, Feb 2, 2010 at 7:23 PM, Steven M. Christey  
wrote:

On Tue, 2 Feb 2010, Arian J. Evans wrote:

BSIMM is probably useful for government agencies, or some large
organizations. But the vast majority of clients I work with don't have
the time or need or ability to take advantage of BSIMM. Nor should
they. They don't need a software security group.

I'm looking forward to what BSIMM Basic discovers when talking to small and 
mid-size developers.  Many of the questions in the survey PDF assume that the 
respondent has at least thought of addressing software security, but not all 
questions assume the presence of an SSG, and there are even questions about the 
use of general top-n lists vs. customized top-n lists that may be informative.

- Steve

___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___



___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___

Re: [SC-L] BSIMM update (informIT)

[Gary McGraw]

hi mike,

On 2/2/10 9:28 PM, "Mike Boberski"  wrote:
>Fun article. To try to be equally pithy in my response: the article reads to 
>me like a high-tech, application security-specific >form of McCarthyism.

As a die hard liberal, I take offense to the McCarthy comment (hah).  Anyway 
some interleaved thoughts...sorry for the delay...etc and so on.

>The amount of reinvention and discussion about the problems in this space is 
>spectacular.  If one has >something to start from which one can then tailor 
>for one's own purposes, why wouldn't one do this? Does one >need to discover 
>SQL injection on one's own before deciding to do some escaping?

I am with you on this.

>It's crazy in my opinion to think that the majority of the planet has the 
>expertise let alone the bandwidth (Agile, >anyone?) to thoughtfully research 
>and derive anything that results in a net effect of a targeted, measurable, 
>>comparable level of security.

Who is arguing that?  Is this supposed to be some straw man for the BSIMM?  I'm 
lost.  What the heck are you talking about?

gem

company www.cigital.com
podcast www.cigital.com/silverbullet
blog www.cigital.com/justiceleague
book www.swsec.com


On Tue, Feb 2, 2010 at 4:32 PM, Arian J. Evans  
wrote:
100% agree with the first half of your response, Kevin. Here's what
people ask and need:


Strategic folks (VP, CxO) most frequently ask:

+ What do I do next? / What should we focus on next? (prescriptive)

+ How do we tell if we are reducing risk? (prescriptive guidance again)

Initially they ask for descriptive information, but once they get
going they need strategic prescriptions.


Tactical folks tend to ask:

+ What should we fix first? (prescriptive)

+ What steps can I take to reduce XSS attack surface by 80%? (yes, a
prescriptive blacklist can work here)


 Implementation level folks ask:

+ What do I do about this specific attack/weakness?

+ How do I make my compensating control (WAF, IPS) block this specific attack?

etc.

BSIMM is probably useful for government agencies, or some large
organizations. But the vast majority of clients I work with don't have
the time or need or ability to take advantage of BSIMM. Nor should
they. They don't need a software security group.

They need a clear-cut tree of prescriptive guidelines that work in a
measurable fashion. I agree and strongly empathize with Gary on many
premises of his article - including that not many folks have metrics,
and tend to have more faith and magic.

But, as should be no surprise, I cateogrically disagree with the
entire concluding paragraph of the article. Sadly it's just more faith
and magic from Gary's end. We all can do better than that.

There are other ways to gather and measure useful metrics easily
without BSIMM. Black Box and Pen Test metrics, and Top(n) List metrics
are metrics, and highly useful metrics. And definitely better than no
metrics.

Pragmatically, I think Ralph Nader fits better than Feynman for this discussion.

Nader's Top(n) lists and Bug Parades earned us many safer-society
(cars, water, etc.) features over the last five decades.

Feynman didn't change much in terms of business SOP.

Good day then,

---
Arian Evans
capitalist marksman. eats animals.



On Tue, Feb 2, 2010 at 9:30 AM, Wall, Kevin  wrote:
> On Thu, 28 Jan 2010 10:34:30 -0500, Gary McGraw wrote:
>
>> Among other things, David [Rice] and I discussed the difference between
>> descriptive models like BSIMM and prescriptive models which purport to
>> tell you what you should do.  I just wrote an article about that for
>> informIT.  The title is
>>
>> "Cargo Cult Computer Security: Why we need more description and less
>> prescription."
>> http://www.informit.com/articles/article.aspx?p=1562220
>
> First, let me say that I have been the team lead of a small Software
> Security Group (specifically, an Application Security team) at a
> large telecom company for the past 11 years, so I am writing this from
> an SSG practitioner's perspective.
>
> Second, let me say that I appreciate descriptive holistic approaches to
> security such as BSIMM and OWASP's OpenSAMM. I think they are much
> needed, though seldom heeded.
>
> Which brings me to my third point. In my 11 years of experience working
> on this SSG, it is very rare that application development teams are
> looking for a _descriptive_ approach. Almost always, they are
> looking for a _prescriptive_ one. They want specific solutions
> to specific problems, not some general formula to an approach that will
> make them more secure. To those application development teams, something
> like OWASP's ESAPI is much more valuable than something like BSIMM or
> OpenSAMM. In fact, I you confirm that you BSIMM research would indicate that
> many companies' SSGs have developed their own proprietary security APIs
> for use by their application development teams. Therefore, to that end,
> I would not say we need less _prescriptive_ and more _descriptive_
> approaches. Both are useful and i

Re: [SC-L] BSIMM update (informIT)

[Gary McGraw]

hi kevin (and sc-l),

Sorry for the delay responding to this.  I was skiing yesterday with my son Eli 
and just flew across the country for the SANS summit this morning (leaving 
behind 6 inches of new snow in VA).  Anyway, better late than never.

I'll interleave responses below.

On Thu, 28 Jan 2010 10:34:30 -0500, Gary McGraw wrote:
>> "Cargo Cult Computer Security: Why we need more description and less
>> prescription."  http://www.informit.com/articles/article.aspx?p=1562220

  >On 2/2/10 12:30 PM, "Wall, Kevin"  wrote:
   > In my 11 years of experience working
>on this SSG, it is very rare that application development teams are
>looking for a _descriptive_ approach. Almost always, they are
>looking for a _prescriptive_ one. They want specific solutions
>to specific problems, not some general formula to an approach that will
  >make them more secure.

Absolutely.  I think as an SSG lead in a particular company environment you 
must have a prescriptive approach but that the approach you develop will be 
better if informed by data from a descriptive model like BSIMM.  (For the 
record, I see SAMM as a prescriptive model that tells you often in great detail 
what your initiative should be doing without knowing one whit about how your 
organization ticks.)   If you read the article carefully, there are two 
paragraphs that together should make this clear.

Here's the first:
"Prescriptive models purport to tell you what you should do.  Promulgators of 
such models say things more like, "the model is chocked full of value 
judgements [sic] about what organizations SHOULD be doing."   That's just 
dandy, as long as any prescriptive model only became prescriptive over time 
based on sufficient observation and testing."

And here's the second:
"Also worthy of mention in this section is the "one size fits all" problem that 
many prescriptive models suffer from.  The fact is, nobody knows your 
organizational culture like you do. A descriptive comparison allows you to 
gather descriptive data and adapt good ideas from others while taking your 
culture into account."

BSIMM is meant to be a tool for the people running and SSG (and for that 
matter, strategizing about a company's software security initiative).  The 
article is really about the differences between BSIMM and SAMM than anything 
else.  It's not really about the difference between BSIMM and ESAPI.  BSIMM and 
things like ESAPI fit together.

>Both are useful and ideally should go together like hand and glove.

Exactly right.

>I suspect that this apparent dichotomy in our perception of the
>usefulness of the prescriptive vs. descriptive approaches is explained
>in part by the different audiences with whom we associate.

Agreed.  See above.   BSIMM is a tool for executives to help build, measure, 
and maintain a software security initiative.

>If our SSG were to hand them something like
>BSIMM, they would come away telling their management that we didn't help
>them at all.

Please do NOT even think about handing the BSIMM to developers as a solution!  
The BSIMM is a yardstick for an initiative, and it's meant for a guy like you.  
The notion is to measure your own initiative and most importantly of all 
compare your initiative to your peers.

>This brings me to my fourth, and likely most controversial point. Despite
>the interesting historical story about Feynman, I question whether BSIMM
>is really "scientific" as the BSIMM community claims. I would contend
>that we are only fooling ourselves if we claim otherwise.

I think this is a valid criticism.  The only thing that makes BSIMM more 
scientific than other methodologies like the Touchoints, SDL, CLASP, or SAMM, 
is that the BSIMM uses real data and real measurement.  However the measurement 
technique is certainly not foolproof.  (Incidentally, I state that view pretty 
clearly in the article...computer science, and other fields with "science" in 
their name are usually not.)

>While I am certainly not privy to the exact method used to arrive at the
>BSIMM data (I have read through the "BSIMM Begin" survey, but have not
>been involved in a full BSIMM assessment), I would contend that the
>process is not repeatable to the necessary degree required by science.

This criticism holds some water, but you are shooting from the hip and it is 
pretty clear that you have not read the BSIMM itself.   That, and the first 
article we wrote about the BSIMM explain our methods pretty clearly. Please 
read those two things and lets continue this line of questioning.

>I challenge [the BSIMM team] to put forth additional information explaining 
>their data collection
>process and in particular, describing how it avoids unintentional bias. (E.g., 
>Are assessment participants choose >at random? By whom?  How do you know you 
>have a representative sample of
>a company? Etc.)

This is pretty clearly explained in the BSIMM itself.

>In my opinion, comparison of observations from two companie

Re: [SC-L] NIST SP 800-37

[Benjamin Tomhave]

800-37 has been in release for a while, providing the basis for the C&A
process. My understanding is that C&A is evolving (and going the way of
the dinosaur) very soon as NIST works with CNSS/JTF on the next big
thing. I'm blanking on the rest of the details (not my space), but
pinging Mike Smith (@rybolov) or Dan Philpott (@danphilpott) on Twitter
would likely be a good starting point.

On 2/3/10 1:12 PM, McGovern, James F. (eBusiness) wrote:
> NIST has created a draft document entitled: Guide for applying risk 
> management framework to federal information systems: a security 
> lifecycle approach. Curious to know if anyone has identified gaps, 
> differences in opinion, etc between NIST and how either SAMM or
> BSIMM would define the same?
> 
>  This
> communication, including attachments, is for the exclusive use of
> addressee and may contain proprietary, confidential and/or privileged
> information.  If you are not the intended recipient, any use,
> copying, disclosure, dissemination or distribution is strictly
> prohibited.  If you are not the intended recipient, please notify the
> sender immediately by return e-mail, delete this communication and
> destroy all copies. 
> 
> 
> 
> 
> ___ Secure Coding mailing
> list (SC-L) SC-L@securecoding.org List information, subscriptions,
> etc - http://krvw.com/mailman/listinfo/sc-l List charter available at
> - http://www.securecoding.org/list/charter.php SC-L is hosted and
> moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free,
> non-commercial service to the software security community. 
> ___

-- 
Benjamin Tomhave, MS, CISSP
tomh...@secureconsulting.net
Blog: http://www.secureconsulting.net/
Twitter: http://twitter.com/falconsview
LI: http://www.linkedin.com/in/btomhave

[ Random Quote: ]
"Opportunity is missed by most people because it is dressed in overalls
and looks like work."
Thomas A. Edison
___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___

Re: [SC-L] BSIMM update (informIT)

[Kenneth Van Wyk]

On Jan 28, 2010, at 10:34 AM, Gary McGraw wrote:
> Among other things, David and I discussed the difference between descriptive 
> models like BSIMM and prescriptive models which purport to tell you what you 
> should do. 

Thought I'd chime in on this a bit, FWIW...  From my perspective, I welcome 
BSIMM and I welcome SAMM.  I don't see it in the least as a "one or the other" 
debate.

A decade(ish) since the first texts on various aspects of software security 
started appearing, it's great to have a BSIMM that surveys some of the largest 
software groups on the planet to see what they're doing.  What actually works.  
That's fabulously useful.  On the other hand, it is possible that ten thousand 
lemmings can be wrong.  Following the herd isn't always what's best.

SAMM, by contrast, was written by some bright, motivated folks, and provides us 
all with a set of targets to aspire to.  Some will work, and some won't, 
without a doubt.

To me, both models are useful as guide posts to help a software group--an SSG 
if you will--decide what practices will work best in their enterprise.

But as useful as both SAMM and BSIMM are, I think we're all fooling ourselves 
if we consider these to be standards or even maturity models.  Any other 
engineering discipline on the planet would laugh us all out of the room by the 
mere suggestion.  There's value to them, don't get me wrong.  But we're still 
in the larval mode of building an engineering discipline here folks.  After 
all, as a species, we didn't start (successfully) building bridges in a decade.

For now, my suggestion is to read up, try things that seem reasonable, and 
build a set of practices that work for _you_.  

Cheers,

Ken

-
Kenneth R. van Wyk
KRvW Associates, LLC
http://www.KRvW.com



smime.p7s
Description: S/MIME cryptographic signature
___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___

[SC-L] NIST SP 800-37

[McGovern, James F. (eBusiness)]

NIST has created a draft document entitled: Guide for applying risk
management framework to federal information systems: a security
lifecycle approach. Curious to know if anyone has identified gaps,
differences in opinion, etc between NIST and how either SAMM or BSIMM
would define the same?

This communication, including attachments, is for the exclusive use of 
addressee and may contain proprietary, confidential and/or privileged 
information.  If you are not the intended recipient, any use, copying, 
disclosure, dissemination or distribution is strictly prohibited.  If you are 
not the intended recipient, please notify the sender immediately by return 
e-mail, delete this communication and destroy all copies.

___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___

Re: [SC-L] BSIMM update (informIT)

[McGovern, James F. (eBusiness)]

OK, being the insurance enterprisey security guy I think you may be onto
something. One of the many reasons why actuarial science can work in
insurance is the fact that there is a lot more public data than in IT
security. If you smash your car into a wall, your chosen carrier doesn't
just pay the claim. This information is shared in what we refer to as
the CLUE database. Other carriers should you decide to switch carriers
will also know the characteristics of your loss. 

CLUE works because folks have figured out that sharing of negative
information can benefit the business. Likewise, CLUE did enough homework
to figure out the right taxonomy and metadata in order to make it
happen. Have security professionals ever figured out how to turn
something bad into something good for the same organization? Have
security professionals ever figured out even how to describe a security
"event" in a consistent enough way such that acturial type calculations
could occur...

FYI. Clue is successful and isn't done for regulatory reasons. It is
done for sound business practice. The same model we should operate
within...

-Original Message-
From: Benjamin Tomhave [mailto:list-s...@secureconsulting.net] 
Sent: Wednesday, February 03, 2010 11:07 AM
To: McGovern, James F. (P+C Technology)
Cc: Secure Code Mailing List
Subject: Re: [SC-L] BSIMM update (informIT)

I challenge the validity of any risk assessment/rating approach in use
today in infosec circles, whether it be OWASP or FAIR or IAM/ISAM or
whatever. They are all fundamentally flawed in that they are based on
qualitative values the introduce subjectivity, and they lack the
historical data seen in the actuarial science to make the probability
estimates even remotely reasonable. FAIR tries to compensate for this by
using Bayesian statistics, but the qualitative->quantitative conversion
is still highly problematic.

On prescriptive... the problem is this: businesses will not spend money
unless they're required to do so. Security will never succeed without at
least an initial increased spend. It is exceedingly difficult to make a
well-understood business case for proper security measures and spend. I
think this is something you guys in insurance (you, Chris Hayes, etc.)
perhaps take for granted. The other businesses - especially SMBs - don't
even understand what we're talking about, and they certainly don't have
any interest in dropping a penny on "security" without seeing a direct
benefit.

Do I trust regulators to do things right? Of course not, but that's only
one possible fork. The other possible fork is relying on the courts to
finally catch-up such that case law can develop around defining
"reasonable standard of care" and then evolving it over time. In either
case, you need to set a definitive mark that says "you must do THIS MUCH
or you will be negligent and held accountable." I hate standards like
PCI as much as the next guy because I hate being told how I should be
doing security, but in the short-to-mid-term it's the right approach
because it tells people the expectation for performance. If you never
set expectations for performance, then you shouldn't be disappointed
when people don't achieve them. The bottom line here is that we need to
get far more proactive in the regulatory space so that we can influence
sensible regulations that mandate change rather than relying on
businesses to "do the right thing" without understand the underlying
business value.

Conceptually, I agree with the idealist approach, but in reality I don't
find that it works well at all. I've worked with a half-dozen or more
companies of varying size in the last couple years and NONE of them
understood risk, risk management, current security theory, or how the
implicit AND explicit value of security changes. It's just not intuitive
to most people, not the least of which because bad behaviors are
generally divorced from tangible consequences. Anyway... :)

I can go on forever on this topic... :)

-ben

On 2/3/10 10:06 AM, McGovern, James F. (eBusiness) wrote:
> While Wall Street's definition of risk collapsed, the insurance model 
> of risk stood the test of time :-)
> 
> Should we explore your question of "how are risk levels defined in 
> business terms" more deeply or can we simply say that if you don't 
> have your own industry-specific regulatory way of quantifying, a good 
> starting point may be to leverage the OWASP Risk Rating system?
> 
> I also would like to challenge and say NO to prescriptive. Security 
> people are not Vice Presidents of the NO department. Instead we need 
> to figure out how to align with other value systems (Think Agile 
> Manifesto). We can be secure without being prescriptive. One example 
> is to do business exercises such as Protection Poker.
> 
> Finally, we shouldn't say yes to regulatory mandates as most of them 
> are misses on the real risk at hand. The challenge here is that they 
> always mandate process but never competency. If a regulation

Re: [SC-L] BSIMM update (informIT)

[Benjamin Tomhave]

I challenge the validity of any risk assessment/rating approach in use
today in infosec circles, whether it be OWASP or FAIR or IAM/ISAM or
whatever. They are all fundamentally flawed in that they are based on
qualitative values the introduce subjectivity, and they lack the
historical data seen in the actuarial science to make the probability
estimates even remotely reasonable. FAIR tries to compensate for this by
using Bayesian statistics, but the qualitative->quantitative conversion
is still highly problematic.

On prescriptive... the problem is this: businesses will not spend money
unless they're required to do so. Security will never succeed without at
least an initial increased spend. It is exceedingly difficult to make a
well-understood business case for proper security measures and spend. I
think this is something you guys in insurance (you, Chris Hayes, etc.)
perhaps take for granted. The other businesses - especially SMBs - don't
even understand what we're talking about, and they certainly don't have
any interest in dropping a penny on "security" without seeing a direct
benefit.

Do I trust regulators to do things right? Of course not, but that's only
one possible fork. The other possible fork is relying on the courts to
finally catch-up such that case law can develop around defining
"reasonable standard of care" and then evolving it over time. In either
case, you need to set a definitive mark that says "you must do THIS MUCH
or you will be negligent and held accountable." I hate standards like
PCI as much as the next guy because I hate being told how I should be
doing security, but in the short-to-mid-term it's the right approach
because it tells people the expectation for performance. If you never
set expectations for performance, then you shouldn't be disappointed
when people don't achieve them. The bottom line here is that we need to
get far more proactive in the regulatory space so that we can influence
sensible regulations that mandate change rather than relying on
businesses to "do the right thing" without understand the underlying
business value.

Conceptually, I agree with the idealist approach, but in reality I don't
find that it works well at all. I've worked with a half-dozen or more
companies of varying size in the last couple years and NONE of them
understood risk, risk management, current security theory, or how the
implicit AND explicit value of security changes. It's just not intuitive
to most people, not the least of which because bad behaviors are
generally divorced from tangible consequences. Anyway... :)

I can go on forever on this topic... :)

-ben

On 2/3/10 10:06 AM, McGovern, James F. (eBusiness) wrote:
> While Wall Street's definition of risk collapsed, the insurance model of
> risk stood the test of time :-)
> 
> Should we explore your question of "how are risk levels defined in
> business terms" more deeply or can we simply say that if you don't have
> your own industry-specific regulatory way of quantifying, a good
> starting point may be to leverage the OWASP Risk Rating system?
> 
> I also would like to challenge and say NO to prescriptive. Security
> people are not Vice Presidents of the NO department. Instead we need to
> figure out how to align with other value systems (Think Agile
> Manifesto). We can be secure without being prescriptive. One example is
> to do business exercises such as Protection Poker.
> 
> Finally, we shouldn't say yes to regulatory mandates as most of them are
> misses on the real risk at hand. The challenge here is that they always
> mandate process but never competency. If a regulation said that I should
> have someone with a fancy title overseeing a program, the business world
> would immediately fill the slot with some non-technical resource who is
> really good at PowerPoint but nothing else. In other words a figurehead.
> Likewise, while regulations cause people to do things that they should
> be doing independently, it has a negative side effect on our economy by
> causing folks to spend money in non-strategic ways.
> 
> -Original Message-
> From: sc-l-boun...@securecoding.org
> [mailto:sc-l-boun...@securecoding.org] On Behalf Of Benjamin Tomhave
> Sent: Tuesday, February 02, 2010 10:19 PM
> To: Arian J. Evans
> Cc: Secure Code Mailing List
> Subject: Re: [SC-L] BSIMM update (informIT)
> 
> While I can't disagree with this based on modern reality, I'm
> increasingly hesitant to allow the conversation to bring in risk, since
> it's almost complete garbage these days. Nobody really understands it,
> nobody really does it very well (especially if we redact out financial
> services and insurance - and even then, look what happened to Wall
> Street risk models!), and more importantly, it's implemented so shoddily
> that there's no real, reasonable way to actually demonstrate risk
> remediation/reduction because talking about it means bringing in a whole
> other range of discussions ("what is most important to the business?"
> and 

Re: [SC-L] BSIMM update (informIT)

[McGovern, James F. (eBusiness)]

While Wall Street's definition of risk collapsed, the insurance model of
risk stood the test of time :-)

Should we explore your question of "how are risk levels defined in
business terms" more deeply or can we simply say that if you don't have
your own industry-specific regulatory way of quantifying, a good
starting point may be to leverage the OWASP Risk Rating system?

I also would like to challenge and say NO to prescriptive. Security
people are not Vice Presidents of the NO department. Instead we need to
figure out how to align with other value systems (Think Agile
Manifesto). We can be secure without being prescriptive. One example is
to do business exercises such as Protection Poker.

Finally, we shouldn't say yes to regulatory mandates as most of them are
misses on the real risk at hand. The challenge here is that they always
mandate process but never competency. If a regulation said that I should
have someone with a fancy title overseeing a program, the business world
would immediately fill the slot with some non-technical resource who is
really good at PowerPoint but nothing else. In other words a figurehead.
Likewise, while regulations cause people to do things that they should
be doing independently, it has a negative side effect on our economy by
causing folks to spend money in non-strategic ways.

-Original Message-
From: sc-l-boun...@securecoding.org
[mailto:sc-l-boun...@securecoding.org] On Behalf Of Benjamin Tomhave
Sent: Tuesday, February 02, 2010 10:19 PM
To: Arian J. Evans
Cc: Secure Code Mailing List
Subject: Re: [SC-L] BSIMM update (informIT)

While I can't disagree with this based on modern reality, I'm
increasingly hesitant to allow the conversation to bring in risk, since
it's almost complete garbage these days. Nobody really understands it,
nobody really does it very well (especially if we redact out financial
services and insurance - and even then, look what happened to Wall
Street risk models!), and more importantly, it's implemented so shoddily
that there's no real, reasonable way to actually demonstrate risk
remediation/reduction because talking about it means bringing in a whole
other range of discussions ("what is most important to the business?"
and "how are risk levels defined in business terms?" and "what role do
data and systems play in the business strategy?" and "how does data flow
into and out of the environment?" and so on). Anyway... the long-n-short
is this: let's stop fooling ourselves by pretending that risk has
anything to do with these conversations.

I think:
 - yes to prescriptive!
 - yes to legal/regulatory mandates!
 - caution: we need some sort of evolving maturity framework to which
the previous two points can be pegged!

cheers,

-ben

This communication, including attachments, is for the exclusive use of 
addressee and may contain proprietary, confidential and/or privileged 
information.  If you are not the intended recipient, any use, copying, 
disclosure, dissemination or distribution is strictly prohibited.  If you are 
not the intended recipient, please notify the sender immediately by return 
e-mail, delete this communication and destroy all copies.



___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___

Re: [SC-L] BSIMM update (informIT)

[Mike Boberski]

> But the vast majority of clients I work with don't have the time or need
or ability to take advantage of BSIMM

Mike's Top 5 Web Application Security Countermeasures:

1. Add a security guy or gal who has a software development background to
your application's software development team.

2. Turn SSL/TLS on for all connections (including both external and backend
connections) that are authenticated or that involve sensitive data or
functions.

3. Build an Enterprise Security API (a.k.a. an ESAPI, e.g. OWASP's several
different ESAPI toolkits) that is specific to your solution stack and
minimally provides input validation controls that use whitelists, output
encoding/escaping controls (optionally use parameterized interfaces for
SQL), and authentication controls. Build your ESAPI to target a specific
level of overall security when all of your security controls are viewed as a
whole (e.g. an OWASP Application Security Verification Standard (ASVS)
level).

4. Write a programming manual (i.e. a secure coding standard that is
specific to your solution stack that is organized by vulnerability type or
security requirement with before and after code snippets, e.g. a cookbook
that provides before and after code snippets and links to API documentation)
that contains step-by-step instructions for using your ESAPI to both
proactively guard against vulnerabilities, and to act as a quick reference
when the time comes to make fixes.

5. Gate releases of your ESAPI library (e.g. if it is being packaged in a
wrapper for subsequent use by other developers throughout the application)
with security functional tests that include sufficient negative test cases
to demonstrate the security controls are working using data that is specific
to your application. Gate releases of your application (ideally gate source
control checkins) with security-focused code reviews of all new or updated
application code produced during the release (looking out for where new or
updated security controls/security control configuration updates are
needed).

Mike


On Tue, Feb 2, 2010 at 7:23 PM, Steven M. Christey wrote:

>
> On Tue, 2 Feb 2010, Arian J. Evans wrote:
>
>  BSIMM is probably useful for government agencies, or some large
>> organizations. But the vast majority of clients I work with don't have
>> the time or need or ability to take advantage of BSIMM. Nor should
>> they. They don't need a software security group.
>>
>
> I'm looking forward to what BSIMM Basic discovers when talking to small and
> mid-size developers.  Many of the questions in the survey PDF assume that
> the respondent has at least thought of addressing software security, but not
> all questions assume the presence of an SSG, and there are even questions
> about the use of general top-n lists vs. customized top-n lists that may be
> informative.
>
> - Steve
>
> ___
> Secure Coding mailing list (SC-L) SC-L@securecoding.org
> List information, subscriptions, etc -
> http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> ___
>
___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___

Re: [SC-L] BSIMM update (informIT)

[Mike Boberski]

Fun article. To try to be equally pithy in my response: the article reads to
me like a high-tech, application security-specific form of McCarthyism.

To explain...

The amount of reinvention and discussion about the problems in this space is
spectacular.

If one has something to start from which one can then tailor for one's own
purposes, why wouldn't one do this? Does one need to discover SQL injection
on one's own before deciding to do some escaping?

It's crazy in my opinion to think that the majority of the planet has the
expertise let alone the bandwidth (Agile, anyone?) to thoughtfully research
and derive anything that results in a net effect of a targeted, measurable,
comparable level of security.

To all the good folks out there, here is some advice for free: don't start
from scratch, whether it's at the program level, the project level, or the
toolkit level. Use the top x lists to make sure whatever you're doing is up
to date with the latest best practices and technologies. On the subject of
tools and products specifically since the article veers there very
specifically: if you're looking to build or buy a product that provides
security functions, go look into CC. If you're looking at a cryptomodule, go
look into FIPS 140. If you're looking at an enterprise app, go look into
ASVS. If you need a toolkit that validates form input data strings in PHP
using a whitelist because you're trying to provide a first layer of defense
against XSS and SQLi, use BSIMM. Just kidding. Yes, use ESAPI in those
cases.

FWIW,

Best,

Mike


On Tue, Feb 2, 2010 at 4:32 PM, Arian J. Evans
wrote:

> 100% agree with the first half of your response, Kevin. Here's what
> people ask and need:
>
>
> Strategic folks (VP, CxO) most frequently ask:
>
> + What do I do next? / What should we focus on next? (prescriptive)
>
> + How do we tell if we are reducing risk? (prescriptive guidance again)
>
> Initially they ask for descriptive information, but once they get
> going they need strategic prescriptions.
>
>
> Tactical folks tend to ask:
>
> + What should we fix first? (prescriptive)
>
> + What steps can I take to reduce XSS attack surface by 80%? (yes, a
> prescriptive blacklist can work here)
>
>
>  Implementation level folks ask:
>
> + What do I do about this specific attack/weakness?
>
> + How do I make my compensating control (WAF, IPS) block this specific
> attack?
>
> etc.
>
> BSIMM is probably useful for government agencies, or some large
> organizations. But the vast majority of clients I work with don't have
> the time or need or ability to take advantage of BSIMM. Nor should
> they. They don't need a software security group.
>
> They need a clear-cut tree of prescriptive guidelines that work in a
> measurable fashion. I agree and strongly empathize with Gary on many
> premises of his article - including that not many folks have metrics,
> and tend to have more faith and magic.
>
> But, as should be no surprise, I cateogrically disagree with the
> entire concluding paragraph of the article. Sadly it's just more faith
> and magic from Gary's end. We all can do better than that.
>
> There are other ways to gather and measure useful metrics easily
> without BSIMM. Black Box and Pen Test metrics, and Top(n) List metrics
> are metrics, and highly useful metrics. And definitely better than no
> metrics.
>
> Pragmatically, I think Ralph Nader fits better than Feynman for this
> discussion.
>
> Nader's Top(n) lists and Bug Parades earned us many safer-society
> (cars, water, etc.) features over the last five decades.
>
> Feynman didn't change much in terms of business SOP.
>
> Good day then,
>
> ---
> Arian Evans
> capitalist marksman. eats animals.
>
>
>
> On Tue, Feb 2, 2010 at 9:30 AM, Wall, Kevin  wrote:
> > On Thu, 28 Jan 2010 10:34:30 -0500, Gary McGraw wrote:
> >
> >> Among other things, David [Rice] and I discussed the difference between
> >> descriptive models like BSIMM and prescriptive models which purport to
> >> tell you what you should do.  I just wrote an article about that for
> >> informIT.  The title is
> >>
> >> "Cargo Cult Computer Security: Why we need more description and less
> >> prescription."
> >> http://www.informit.com/articles/article.aspx?p=1562220
> >
> > First, let me say that I have been the team lead of a small Software
> > Security Group (specifically, an Application Security team) at a
> > large telecom company for the past 11 years, so I am writing this from
> > an SSG practitioner's perspective.
> >
> > Second, let me say that I appreciate descriptive holistic approaches to
> > security such as BSIMM and OWASP's OpenSAMM. I think they are much
> > needed, though seldom heeded.
> >
> > Which brings me to my third point. In my 11 years of experience working
> > on this SSG, it is very rare that application development teams are
> > looking for a _descriptive_ approach. Almost always, they are
> > looking for a _prescriptive_ one. They want specific solutions
> > to specific problems, not som

Re: [SC-L] BSIMM update (informIT)

[Benjamin Tomhave]

While I can't disagree with this based on modern reality, I'm
increasingly hesitant to allow the conversation to bring in risk, since
it's almost complete garbage these days. Nobody really understands it,
nobody really does it very well (especially if we redact out financial
services and insurance - and even then, look what happened to Wall
Street risk models!), and more importantly, it's implemented so shoddily
that there's no real, reasonable way to actually demonstrate risk
remediation/reduction because talking about it means bringing in a whole
other range of discussions ("what is most important to the business?"
and "how are risk levels defined in business terms?" and "what role do
data and systems play in the business strategy?" and "how does data flow
into and out of the environment?" and so on). Anyway... the long-n-short
is this: let's stop fooling ourselves by pretending that risk has
anything to do with these conversations.

I think:
 - yes to prescriptive!
 - yes to legal/regulatory mandates!
 - caution: we need some sort of evolving maturity framework to which
the previous two points can be pegged!

cheers,

-ben

On 2/2/10 4:32 PM, Arian J. Evans wrote:
> 100% agree with the first half of your response, Kevin. Here's what
> people ask and need:
> 
> 
> Strategic folks (VP, CxO) most frequently ask:
> 
> + What do I do next? / What should we focus on next? (prescriptive)
> 
> + How do we tell if we are reducing risk? (prescriptive guidance again)
> 
> Initially they ask for descriptive information, but once they get
> going they need strategic prescriptions.
> 
> 
> Tactical folks tend to ask:
> 
> + What should we fix first? (prescriptive)
> 
> + What steps can I take to reduce XSS attack surface by 80%? (yes, a
> prescriptive blacklist can work here)
> 
> 
>  Implementation level folks ask:
> 
> + What do I do about this specific attack/weakness?
> 
> + How do I make my compensating control (WAF, IPS) block this specific attack?
> 
> etc.
> 
> BSIMM is probably useful for government agencies, or some large
> organizations. But the vast majority of clients I work with don't have
> the time or need or ability to take advantage of BSIMM. Nor should
> they. They don't need a software security group.
> 
> They need a clear-cut tree of prescriptive guidelines that work in a
> measurable fashion. I agree and strongly empathize with Gary on many
> premises of his article - including that not many folks have metrics,
> and tend to have more faith and magic.
> 
> But, as should be no surprise, I cateogrically disagree with the
> entire concluding paragraph of the article. Sadly it's just more faith
> and magic from Gary's end. We all can do better than that.
> 
> There are other ways to gather and measure useful metrics easily
> without BSIMM. Black Box and Pen Test metrics, and Top(n) List metrics
> are metrics, and highly useful metrics. And definitely better than no
> metrics.
> 
> Pragmatically, I think Ralph Nader fits better than Feynman for this 
> discussion.
> 
> Nader's Top(n) lists and Bug Parades earned us many safer-society
> (cars, water, etc.) features over the last five decades.
> 
> Feynman didn't change much in terms of business SOP.
> 
> Good day then,
> 
> ---
> Arian Evans
> capitalist marksman. eats animals.
> 
> 
> 
> On Tue, Feb 2, 2010 at 9:30 AM, Wall, Kevin  wrote:
>> On Thu, 28 Jan 2010 10:34:30 -0500, Gary McGraw wrote:
>>
>>> Among other things, David [Rice] and I discussed the difference between
>>> descriptive models like BSIMM and prescriptive models which purport to
>>> tell you what you should do.  I just wrote an article about that for
>>> informIT.  The title is
>>>
>>> "Cargo Cult Computer Security: Why we need more description and less
>>> prescription."
>>> http://www.informit.com/articles/article.aspx?p=1562220
>>
>> First, let me say that I have been the team lead of a small Software
>> Security Group (specifically, an Application Security team) at a
>> large telecom company for the past 11 years, so I am writing this from
>> an SSG practitioner's perspective.
>>
>> Second, let me say that I appreciate descriptive holistic approaches to
>> security such as BSIMM and OWASP's OpenSAMM. I think they are much
>> needed, though seldom heeded.
>>
>> Which brings me to my third point. In my 11 years of experience working
>> on this SSG, it is very rare that application development teams are
>> looking for a _descriptive_ approach. Almost always, they are
>> looking for a _prescriptive_ one. They want specific solutions
>> to specific problems, not some general formula to an approach that will
>> make them more secure. To those application development teams, something
>> like OWASP's ESAPI is much more valuable than something like BSIMM or
>> OpenSAMM. In fact, I you confirm that you BSIMM research would indicate that
>> many companies' SSGs have developed their own proprietary security APIs
>> for use by their application development teams. Therefore, to that end,
>>