Re: [SC-L] Really dumb questions?

2007-08-30 Thread Bret Watson
At 10:51 PM 29/08/2007, McGovern, James F (HTSC, IT) wrote:


- So when a vendor says that they are focused on quality and not
security, and vice versa what exactly does this mean? I don't have a
great mental model of something that is a security concern that isn't a
predictor of quality. Likewise, in terms of quality, other than
producing metrics on things such as depth of inheritance, cyclomatic
complexity, etc wouldn't bad numbers here at least be a predictor of a
bad design and therefore warrant deeper inspection from a security
perspective?


My opinion is that security and quality are inherently related. Poor 
security indicates poor design, poor testing etc  poor quality 
management tends to result in poor application security..


In fact two jobs ago I used this argument to implement security at a 
company who was extremely strong in quality (5% of the workforce 
belonged to the quality dept). I've also found that using quality 
tools such as FMECA etc for security assessments works very easily.

Cheers

Bret 
___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___


Re: [SC-L] Really dumb questions?

2007-08-30 Thread John Steven
James,

Not dumb questions: an unfortunate situation. I do tool bakeoffs for clients a 
fair amount. I'm responsible for the rules Cigital initially sold to Fortify. I 
also attempt to work closely with companies like Coverity and understand deeply 
the underpinnings of that tool's engine. I've a fair amount of experience with 
Klocwork, unfortunately less with Ounce.

I understand the situation like this: technical guys at each of these companies 
are all great guys, smart, and understand their tool's capabilities and focus. 
They accurately describe what their tool does and don't misrepresent it.

On the other hand, I've experienced competition bashing in the sales process as 
I've helped companies with tool selections and bake offs. I see NO value in 
this. As I said in a previous post to this list: the tools differ both 
macroscopically in terms of approach and microscopically in terms rule 
implementation. Please see my previous post about bake-offs and such if you'd 
like more information on how to disambiguate tool capabilities objectively.

No blanket statement about quality or security fits any vendors' tool; ANY 
vendor. Ignore this level of commentary by the vendors.*(1)

No boolean answer exists to your question, let me give you some of my 
experiences:


 *   Fortify's SCA possesses far-and-away the largest rule set, covering both 
topics people consider purely security and those that may-or-may-not create 
opportunity for exploit (often when combined with other factors) which one may 
call quality rules. My impression is that SCA can be effectively used by 
Security Folk, QA Folk, or developers with a mind to improve the quality or 
security of their code. Recent inclusion of Findbugs bolsters SCA's 
capabilities to give code quality commentary.


 *   Coverity's Prevent often gets pigeon-holed as a quality tool, but does 
an exceptional job of tracking down memory issues in C, C++. Skilled security 
consultants will tell you that failing to fix Prevent's results in your code 
will result in various memory-based command injection opportunities (BO, format 
string, write-anywhere's, etc.). It also effectively targets time-and-state 
issues, as well as other classes of bug. Prevent can effectively be used by 
Security Folk and Developers (or your rare hardcore QA person) to improve code 
quality and squelch opportunity for exploit.


 *   Klocwork's tool targets rule spaces similar to Fortify, but possesses 
less. Often pegged as a quality tool (as well), do find its UI (more than its 
engine) possess helpful features that only a QA professional would enjoy. This 
includes its defect density calculation, reverse engineering capabilities, 
and its reporting/time-series style. Klocwork can be effectively used by a 
Security guy to find security bugs, but I believe Fortify and Ounce have 
widened the rules gap in recent years.

Tackling your other questions in rapid succession:

There is no difference, technically, between the ability to scan for quality or 
security. However, each engine focuses on parsing and keeping track of only 
that state which provides meaningful data to their rules. You can imagine that 
Fortify carries a fair amount of information about where data came from and 
what functions may be dangerous and can therefore support new security rules 
easily. They don't carry around information to aggregate defect density readily 
like K7 can. Does this make one intrinsically better than the other for quality 
or security? Perhaps having worked on static analysis tools I'm cranky but I 
say, No. If the market clearly mandated something specifically, all the 
vendors would augment their engine to support it. Some would be in a better 
position to offer it than others.

When I talk to vendors about COBOL and similar support they shudder. I think 
this space represents a huge opportunity for the first vendor to support it, 
but as a commercial organization, I wouldn't hold your breath on near-term 
support.

I could answer how these tools support new languages, but that doesn't seem 
like public domain knowledge. I'll let the vendors tackle that 'un.


John Steven
Technical Director; Principal, Software Security Group
Direct: (703) 404-5726 Cell: (703) 727-4034
Key fingerprint = 4772 F7F3 1019 4668 62AD  94B0 AE7F EEF4 62D5 F908

Blog: http://www.cigital.com/justiceleague
Papers: http://www.cigital.com/papers/jsteven

http://www.cigital.com
Software Confidence. Achieved.


*(1) I'm also explicitly dodging the quality vs. security debate here. Having 
read/posted to this list for the last 7 years, that semi-annual topic has been 
flogged more than your average dead equine.



From: McGovern, James F (HTSC, IT) [EMAIL PROTECTED]

Most recently, we have met with a variety of vendors including but not
limited to: Coverity, Ounce Labs, Fortify, Klocwork, HP and so on. In
the conversation they all used interesting phrases to describe they
classify their 

Re: [SC-L] Really dumb questions?

2007-08-30 Thread Robert C. Seacord
James, Bret-

I agree with Bret that security and quality are inherently related (as
well as many other system attributes).

I think vendors (particularly sales guys) tend to reflect back to
customers what they are hearing from other customers.  So I think many
customers go to these vendors asking for securitysolutions or looking
for just general QA tools.  Of course, there are also subsets of
coding defects that are more high correlated with security
vulnerabilities which is what a vendor often means by a security focus.

rCs

 At 10:51 PM 29/08/2007, McGovern, James F (HTSC, IT) wrote:


   
 - So when a vendor says that they are focused on quality and not
 security, and vice versa what exactly does this mean? I don't have a
 great mental model of something that is a security concern that isn't a
 predictor of quality. Likewise, in terms of quality, other than
 producing metrics on things such as depth of inheritance, cyclomatic
 complexity, etc wouldn't bad numbers here at least be a predictor of a
 bad design and therefore warrant deeper inspection from a security
 perspective?
 


 My opinion is that security and quality are inherently related. Poor 
 security indicates poor design, poor testing etc  poor quality 
 management tends to result in poor application security..


 In fact two jobs ago I used this argument to implement security at a 
 company who was extremely strong in quality (5% of the workforce 
 belonged to the quality dept). I've also found that using quality 
 tools such as FMECA etc for security assessments works very easily.

 Cheers

 Bret 
 ___
 Secure Coding mailing list (SC-L) SC-L@securecoding.org
 List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
 List charter available at - http://www.securecoding.org/list/charter.php
 SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
 as a free, non-commercial service to the software security community.
 ___
   


-- 
Robert C. Seacord
Senior Vulnerability Analyst
CERT/CC 

Work: 412-268-7608
FAX: 412-268-6989

___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___


Re: [SC-L] Really dumb questions?

2007-08-30 Thread Leichter, Jerry
| Most recently, we have met with a variety of vendors including but not
| limited to: Coverity, Ounce Labs, Fortify, Klocwork, HP and so on. In
| the conversation they all used interesting phrases to describe they
| classify their competitors value proposition. At some level, this has
| managed to confuse me and I would love if someone could provide a way to
| think about this in a more boolean way.
| 
| - So when a vendor says that they are focused on quality and not
| security, and vice versa what exactly does this mean? I don't have a
| great mental model of something that is a security concern that isn't a
| predictor of quality. Likewise, in terms of quality, other than
| producing metrics on things such as depth of inheritance, cyclomatic
| complexity, etc wouldn't bad numbers here at least be a predictor of a
| bad design and therefore warrant deeper inspection from a security
| perspective?
What you're seeing here is an attempt to control the set of measure-
ments that will be applied to your product.  If you say you have a
product that tests security, on the one hand, people will ask you
whether out-of-the-box you detect some collection of known security
bugs.  That collection might be good, or it might contain all kinds
of random junk gathered from 20 years of Internet reports.  As a
vendor, I don't (quite reasonably) want to be measured on some
unpredictable, uncontrollable set of hard (easily quantifiable)
standards.

Beyond that, saying you provide security gets you right in to the
maelstrom of security standards, certification, government and
accounting requirements, etc.  Should there be a security problem
with customer code that your product accepted, if you said you
provided security testing, expect to be pulled in to any bad
publicity, lawsuits, etc.

It's so much safer to say you help ensure quality.  That's virtually
impossible to quantify or test, and you clearly can only be one
part of a much larger process.  If things go wrong, it's very hard
for the blame to fall on your product (assuming it's any good at
all).  After all, if you pointed out 100 issues, the fixing of which
clearly improved quality, well, someone else must have screwed up in
letting one other issue through that had a quality impact - quality
is a result of the whole process, not any one element of it.  You
can't test quality in.  (This sounds like excuses, but it happens
also to be mainly true!)
 
| - Even if the rationale is more about people focus rather than tool
| capability, is there anything else that would prevent one tool from
| serving both purposes?
Well, I can speak to two products whose presentations I saw about a
year ago.  Fortify concentrated on security.  They made the point
that they had hundreds of pre-programmed tests specific to known
security vulnerabilities.  The weaknesses in their more general
analysis - e.g, they had almost no ability to do value flow
analysis - they could dismiss as not relevant to their specific
mission of finding security flaws.

Coverity, on the other hand, did very deep general analysis, but
for the most part found security flaws as a side-effect of general
fault-finding mechanisms.  Fortify defenders - including some on
this list - pointed out this limitation, saying that the greatest
value of Fortify came from the extensive work that had gone into
analyzing actual security holes in real code and making sure they
would be caught.

In practice, while there was an overlap between what the two
products found, neither subsumed the other.  If you can afford
it, I'd use both.  (In theory, you could produce a security hole
library for Coverity that copied the one in Fortify.  But no one
is doing that.)

| - Is it reasonable to expect that all of the vendors in this space will
| have the ability to support COBOL, Ruby and Smalltalk sometime next year
| so that customers don't have to specifically request it?
Well, the effort depends on the nature of the product.  Products that
mainly scan for particular patterns are probably easier to adjust to
work with other languages than products that do deeper analysis.  On
the other hand, you have to build up a library of patterns to scan
for, which to some degree will be language-specific.

Products that depend on deep analysis of data flow and such usually
don't care much about surface syntax - a new front end isn't a
big deal - but will run into fundamental problems if they rely on
static properties of programs, like static types.  Languages like
Smalltalk and even more Ruby are likely to be very problematic for
this kind of analysis, since so much is subject to change at run
time.

| - Do the underlying compilers need to do something different since
| languages such as COBOL aren't object-oriented which would make analysis
| a bit different?
I doubt object orientation matters all that much.  In general, anything
that provides (a) static information (b) encapsulation makes analysis
easier.  COBOL is old enough to have elements that 

Re: [SC-L] Really dumb questions?

2007-08-30 Thread Brian Chess
 - So when a vendor says that they are focused on quality and not
 security, and vice versa what exactly does this mean?

We spend most of Chapter 2 of Secure Programming with Static Analysis
describing the different problems that static analysis tools try to solve,
and we show where we think all of the companies you mention (plus a lot of
others) fit in.  The relative importance of false positives vs false
negatives is one important difference, but so is extensibility, rule set (as
John mentioned), ability of the tool to prioritize its findings, and the
interface the tool presents for exploring the results.  From my experience,
the vendors do different things in all of these areas, and these differences
aren't just a result of dumb luck.  They stem from different philosophies
about what the tools are supposed to do.  Quality vs. Security may be an
oversimplification, but the differences between the tools are much more than
cosmetic.


 - Is it reasonable to expect that all of the vendors in this space will
 have the ability to support COBOL, Ruby and Smalltalk sometime next year
 so that customers don't have to specifically request it?

I don't think so.  The way a tool is designed can make it easier or harder
to add support for a new language, but unless you're doing a really
superficial analysis, adding a new language is always a big deal. Supporting
a language requires more than just being able to parse it.  The tools often
have to do special work to make sure that the meaning of common idioms
carries over correctly in the analysis, then there's the small matter of
developing a rule set.

Someone mentioned that Ruby makes life hard because it lacks static types.
While that's true, it compensates in other ways.  For example, because of a
lack of static types, there are often more bugs to find.  There's some
really good academic work going on right now around security analysis of
scripting languages (mostly PHP).  Here's my pick of the week:

Sound and Precise Analysis of Web Applications for Injection Vulnerabilities
by Gary Wassermann and Zhendong Su
http://wwwcsif.cs.ucdavis.edu/~wassermg/research/pldi07.pdf


Regards,
Brian





___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___