Re: [SC-L] InformIT: comparing static analysis tools

[Jim Manico]

Hello Chris,

Thanks for replying!

I think the reaction from "my boss" was not so much knee-jerk, but a
reasonable concern. The risk of persisting intellectual property on a
cloud service is real. And that risk differs depending on your business
(as well as many other factors). I'm eager to see vendors like Veracode
publish more assurance evidence, especially around how they write
software (I'm a lot less worried about the infrastructure in play, that
is pretty much a solved issue. Building secure software is not).

I published an OWASP Podcast with ChrisW recently
http://www.owasp.org/download/jmanico/owasp_podcast_80.mp3 and frankly,
I was impressed. The only issue that I thought was NOT answered in depth
was regarding software centric assurance evidence - especially since
that is your core business.

> (automated scanning plus manual penetration tests, multi-factor
authentication, extremely granular roles and access controls,
per-application backend encryption of results, flexible retention
policies, etc.).

Now this is a great start. I'd like to hear more. How do you do data
contextual access control? How you do key management for backend
encryption?  Are you encrypting db backups? How do you do input
validation and contextual encoding? How do you ensure that all queries
are parametrized/bound? Etc..etc... Perhaps we can get one of you on the
show to discuss how YOU write secure software, and how you prove that to
your clients? Assessment is interesting, but lessons in "building
security in" is much more important to our industry right now, IMO.

> First, the customer needs to understand that they are NOT, in fact,
uploading their code.They are uploading binaries -- compiled code, or
bytecode -- not their source.

Please note, it's trivial to convert bytecode to source code in both the
.NET and Java ecosystems. This distinction feels more sales centric, but
is not technically correct, IMO.

Regards,
Jim



> I'm not the Chris you posed the question to but I'll answer anyway.  :)
> 
> Usually the type of response you described is a knee-jerk reaction.  It's a 
> different model than people are used to, and sometimes people are averse to 
> change, whether that's warranted or not.  It's important to get past the 
> initial reaction and actually have a substantive conversation.
> 
> Naturally, we try to understand each customer's specific hang-ups, but 
> generally speaking there are a couple of things we always cover.  


> 
> Viewing this with a wider lens, there are a lot of factors involved in 
> selecting a tool/service vendor.  One factor that comes into play for us is 
> simply that our solution scales, and many others do not.  We can address the 
> application supply chain problem in ways that others can't.  
> 
> -chris
> 
> 
> Chris Eng
> Senior Director, Research
> Veracode, Inc.
> Office: 781.418.3828
> Mobile: 617.501.3280
> c...@veracode.com 

___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
___

Re: [SC-L] InformIT: comparing static analysis tools

[Arian J. Evans]

That is a great question. According to Gartner, HA has the stench of
inevitability. And in general, I agree.

There are cases where dynamic and static each have clear strengths.
Pragmatic combination of of the two has promise is solving a broad
spectrum of test-cases. Additionally -HA can help improve each other
by improving context, but developing the underlying technology to make
that happen is non-trivial.

This is my guess as to how things will unfold:

Current HA attempts are at the vuln-mashup phase. Let's call this "correlation".

FP reduction: the next step that folks are working on in HA is
"suppression therapy". e.g- using correlation to filter and suppress
false-positives, increase signal-to-noise in output from both analysis
types.

FN reduction: HA has the promise of heatmapping coverage of both
static and dynamic testing. This would more fully allow the expert
running the solution to see what is and isn't getting covered. This
provides a better notion of False Negatives, and allow targeted tuning
and optimization. Or decide where best to focus expert human review
efforts.

Contextualization: The holy grail of HA would be to automatically have
both types of automation feed and tune each other. Black box would be
significantly enhanced by being feed framework config files, and
getting access to things like function names/parameters and objects
that are not directly exposed. This would really help dynamic on MVC
testing. Likewise, I expect dynamic testing could provide some notion
of design or control-flow back to the static engine to enhance static
authentication and authorization analysis. This would also help solve
for mobile: static could extract calls and functions from mobile
binaries, and dynamic could test the back-end web services they talk
to more effectively with that static context.

Context enhancement via HA, however, is kind of the holy grail of HA.
While it sounds great in theory, the complexity bar is high enough it
may be a long time coming.

As development shifts to more modular code on top of "platforms"
(iphone, xbox, rails, etc.) this is also driving interest in
lightweight solutions that can scan modular bits of code. Given that,
I think there is room for a very simplified, streamlined type of HA to
provide simple SAST that can feed a DAST unit-test type capability.
This is probably more realistic to build than the Ultimate Context
Integration Engine idea mentioned above. The more the world moves
towards coding in this manner, the more a solution like this make
sense. You would miss a lot, but it should be lightweight and actually
work.

For now though - the HA options boil down to mashups, and whether or
not suppression therapy is right for you.

We will see where it goes next...

---
Arian Evans
Software Security Scanning Snob


On Fri, Feb 4, 2011 at 2:21 PM, Prasad N Shenoy  wrote:
> Yeah, clear the "cloud" of confusion before talking about the cloud so to
> speak. Not all SaaS offerings available today qualify to be cloud based.
> Well, this thread got morphed into a cloudy discussion. Attempting to get
> back on track, I would say IMHO, it's subjective whether the static analysis
> or dynamic analysis (pen testing/bb testing) technologies have hit the wall
> - depends on who you ask. There is some element of saturation there I
> believe else the industry (term very generously used here)won't be focusing
> on things like Hybrid Analysis. Having said that, what's the future of HA?
> Sent from my iPhone
> On Feb 4, 2011, at 12:27 PM, Ben Laurie  wrote:
>
>
>
> On 4 February 2011 09:22, Chris Wysopal  wrote:
>>
>>
>>
>> “Breaking news.  Google says not to use the cloud.  Improving on-premise
>> tools is the future.”
>
> My view is personal. However, in general, whether the cloud is a good place
> for your data depends on your data and the relationship you have with the
> cloud provider. If your boss says "no, you can't push this stuff outside our
> network" then clearly the cloud is not the right answer (or your boss
> doesn't understand the problem).
>
>>
>>
>>
>> Sorry, I couldn’t help myself. J
>>
>>
>>
>> -Chris
>>
>>
>>
>> From: Ben Laurie [mailto:b...@google.com]
>> Sent: Friday, February 04, 2011 11:34 AM
>> To: Jim Manico
>> Cc: Chris Wysopal; Secure Code Mailing List
>> Subject: Re: [SC-L] InformIT: comparing static analysis tools
>>
>>
>>
>>
>>
>> On 3 February 2011 16:02, Jim Manico  wrote:
>>
>> Chris,
>>
>> I've tried to leverage Veracode in recent engagements. Here is how the
>> conversation went:
>>
>> Jim:
>> "Boss, can I upload all of your code to this cool SaaS service for
>> analysis?"
>>
>> Client:
>> "Uh no, and next time you ask, I'm having you committed".
>>
>> I'm sure you have faced these objections before. How do you work around
>> them?
>>
>>
>>
>> Don't use SaaS, obviously.
>>
>>
>>
>> I'd rather see LLVM's static analysis tools get improved (the framework,
>> btw, is really nice to work with).
>>
>>
>>
>> -Jim Manico
>> http://manico.net
>>
>

Re: [SC-L] InformIT: comparing static analysis tools

[Prasad N Shenoy]

Yeah, clear the "cloud" of confusion before talking about the cloud so to 
speak. Not all SaaS offerings available today qualify to be cloud based.

Well, this thread got morphed into a cloudy discussion. Attempting to get back 
on track, I would say IMHO, it's subjective whether the static analysis or 
dynamic analysis (pen testing/bb testing) technologies have hit the wall - 
depends on who you ask. There is some element of saturation there I believe 
else the industry (term very generously used here)won't be focusing on things 
like Hybrid Analysis. Having said that, what's the future of HA?

Sent from my iPhone

On Feb 4, 2011, at 12:27 PM, Ben Laurie  wrote:

> 
> 
> On 4 February 2011 09:22, Chris Wysopal  wrote:
>  
> 
> “Breaking news.  Google says not to use the cloud.  Improving on-premise 
> tools is the future.”
> 
> 
> My view is personal. However, in general, whether the cloud is a good place 
> for your data depends on your data and the relationship you have with the 
> cloud provider. If your boss says "no, you can't push this stuff outside our 
> network" then clearly the cloud is not the right answer (or your boss doesn't 
> understand the problem).
>  
>  
> 
> Sorry, I couldn’t help myself. J
> 
>  
> 
> -Chris
> 
>  
> 
> From: Ben Laurie [mailto:b...@google.com] 
> Sent: Friday, February 04, 2011 11:34 AM
> To: Jim Manico
> Cc: Chris Wysopal; Secure Code Mailing List
> Subject: Re: [SC-L] InformIT: comparing static analysis tools
> 
>  
> 
>  
> 
> On 3 February 2011 16:02, Jim Manico  wrote:
> 
> Chris,
> 
> I've tried to leverage Veracode in recent engagements. Here is how the 
> conversation went:
> 
> Jim:
> "Boss, can I upload all of your code to this cool SaaS service for analysis?"
> 
> Client:
> "Uh no, and next time you ask, I'm having you committed".
> 
> I'm sure you have faced these objections before. How do you work around them?
> 
>  
> 
> Don't use SaaS, obviously.
> 
>  
> 
> I'd rather see LLVM's static analysis tools get improved (the framework, btw, 
> is really nice to work with).
> 
>  
> 
> 
> -Jim Manico
> http://manico.net
> 
> 
> On Feb 3, 2011, at 1:54 PM, Chris Wysopal  wrote:
> 
> >
> > Nice article.  In the 5 years Veracode has been selling static analysis 
> > services we have seen the market mature.  In the beginning, organizations 
> > were down in the weeds. "What false positive rate or false negative rate 
> > does the tool/service have over a test suite such as SAMATE."  Then we saw 
> > a move up to looking at the trees.  "Did the tool/service support the Java 
> > frameworks I am using?"  Now we are seeing organizations look at the 
> > forest. "Can I scale static analysis effectively over all my development 
> > sites, my outsourcers, and vendors?"  This is a good sign of a maturing 
> > market.
> >
> > It is my firm belief that software security has a consumption problem.  We 
> > know what the defects are.  We know how to fix them.  We even have 
> > automation for detecting a lot of them.  The problem is getting the 
> > information and technology to the right person at the right time 
> > effectively and managing an organization-wide program.  This is the next 
> > challenge for static analysis. I think SaaS based software is 
> > more easily consumed and this isn't any different for software 
> > security
> >
> > -Chris
> >
> > -Original Message-
> > From: sc-l-boun...@securecoding.org [mailto:sc-l-boun...@securecoding.org] 
> > On Behalf Of Gary McGraw
> > Sent: Wednesday, February 02, 2011 9:49 AM
> > To: Secure Code Mailing List
> > Subject: [SC-L] InformIT: comparing static analysis tools
> >
> > hi sc-l,
> >
> > John Steven and I recently collaborated on an article for informIT.  The 
> > article is called "Software [In]security: Comparing Apples, Oranges, and 
> > Aardvarks (or, All Static Analysis Tools Are Not Created Equal)" and is 
> > available here:
> > http://www.informit.com/articles/article.aspx?p=1680863
> >
> > Now that static analysis tools like Fortify and Ounce are hitting the 
> > mainstream there are many potential customers who want to compare them and 
> > pick the best one.  We explain why that's more difficult than it sounds at 
> > first and what to watch out for as you begin to compare tools.  We did this 
> > in order to get out in front of "test suites" that purport to work for tool 
> > comparison.  If you wonder why such suites may not work as advertised, read 
> > the article.
> >
> > Your feedback is welcome.
> >
> > gem
> >
> > company www.cigital.com
> > podcast www.cigital.com/silverbullet
> > blog www.cigital.com/justiceleague
> > book www.swsec.com
> >
> > ___
> > Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, 
> > subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> > List charter available at - http://www.securecoding.org/list/charter.php
> > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) 

Re: [SC-L] InformIT: comparing static analysis tools

[Jeremiah Grossman]

Hi Gary,

No offense taken. :) Securing Web software is a plenty big enough challenge for 
me. 270+ million websites accessible to 2 billion people. And let's not even go 
into the hundreds of thousands of mobile apps, which are basically all mini 
webapps. After I'm done solving that problem I'll think about moving on to 
mainframes, operating systems, and databases. uhh, maybe not.

One thing in your note deserves further WebAppSec clarity:

"You can't push dynamic testing very far back in the SDLC, because your code 
has to run before you can test it dynamically.  For me, way back in the SDLC 
means architectural risk analysis or even security requirements analysis."

Very true, no one appreciates this better than us.

As you probably know the Agile approach to software development is winning out 
in the Web application world. Release early. Release fast. Release often. Push 
or die on Tuesday, that's the motto. This means the time distance between 
architectural risk analysis / security requirements analysis and production 
deployment is roughly 2 - 4 weeks (6 max). Whatever analysis necessary to 
perform must also be repeated on each iteration in case a change impacts the 
entire system. Maybe you do not agree?

Then QA testing windows are shortened to  1 - 3 days regardless of the 
preferred method of software security testing (SAST or DAST). Collective then 
this all begs the question, what forms of software security checks does the 
enterprise have the time and resources to deploy.


Regards,

Jeremiah-


On Feb 4, 2011, at 2:47 AM, Gary McGraw wrote:

> hi arian,
> 
> Glad you liked the article.
> 
> I guess my brush was a bit too wide when it comes to dynamic testing.  I
> was really only referring to the Web application testing tools which in my
> mind "hit the wall" for two reasons.  Reason one is that they only work
> over port80 and are designed to take advantage of the fact that HTTP is a
> stateless protocol (with a few small caveats).  IMPORTANT NOTE: lots of
> software is not web software (sorry Jeremiah).  Reason two has to do with
> the canned nature of the tests.  The generic tests in the black box Web
> app testing tools are, well, generic.  If your software falls prey to
> those tests, it sucks.  IMPORTANT NOTE: Lots of software does, in fact,
> suck.
> 
> As you probably know I call those tools "badness-ometers" and also
> ***suggest that everyone buy and use one***. See this ancient post (and
> associated informIT article) from 2007:
> 
> http://www.cigital.com/justiceleague/2007/03/19/badness-ometers-are-good-do
> -you-own-one/
> 
> Now, there are many other kinds of dynamic testing tools (think of any
> kind of fault injection tool).  I wrote a software engineering tome about
> that way back in 1998 called Software Fault Injection
> .  And you are right that dynamic testing has
> a place.  However, short of fuzzing tools generally tied to a
> grammar-based protocol and capture-replay tools there are not very many
> dynamic testing tools that work for non-Web software.  Why not?  Because
> genericizing is too hard, making the potential market for a particular
> tool too small.
> 
> Security testing plays a key role in the Touchpoints (my own and Cigital's
> approach to SDLC integration) which are described in Software Security
> .  Hoglund and I also describe some dynamic tools that
> we screwed around with when writing Exploiting Software in 2004
> .  I am in complete agreement that
> dynamic testing is important for software security.
> 
> One quibble with your question.  You can't push dynamic testing very far
> back in the SDLC, because your code has to run before you can test it
> dynamically.  For me, way back in the SDLC means architectural risk
> analysis or even security requirements analysis.
> 
> Sorry for the multiple invocations of the way back machine!  I must be
> getting old.
> 
> gem
> 
> company www.cigital.com
> podcast www.cigital.com/silverbullet
> blog www.cigital.com/justiceleague
> book www.swsec.com
> 
> 
> 
> On 2/3/11 7:26 PM, "Arian J. Evans"  wrote:
> 
>> Great article, Gary. Many of your comments about static technology
>> challenges I have seen and verified first-hand, including
>> multi-million dollar cost overruns. After some great dialogue with
>> John Stevens, I suspect we have had similar experiences.
>> 
>> I was just about to write a similar article at a higher level - about
>> how the vast majority of enterprise customers I work with are actively
>> moving security into the SDLC. The time has come, the event has
>> tipped, and SDLC security is indeed mainstream. This is an exciting
>> time to be in the industry.
>> 
>> However - I was curious about your comments about dynamic tools
>> "reaching their limit" or something like that, as customers move
>> security efforts deeper into the SDLC. What does that mean?
>> 
>> I see customers making extensive use of dynam

Re: [SC-L] InformIT: comparing static analysis tools

[Chris Eng]

I'm not the Chris you posed the question to but I'll answer anyway.  :)

Usually the type of response you described is a knee-jerk reaction.  It's a 
different model than people are used to, and sometimes people are averse to 
change, whether that's warranted or not.  It's important to get past the 
initial reaction and actually have a substantive conversation.

Naturally, we try to understand each customer's specific hang-ups, but 
generally speaking there are a couple of things we always cover.  First, the 
customer needs to understand that they are NOT, in fact, uploading their code.  
If they are used to using on-premise tools that require source code, they'll 
often make this mistake.  They are uploading binaries -- compiled code, or 
bytecode -- not their source.  Second, we have many layers of safeguards in 
place ranging covering process (Systrust), infrastructure (SAS-70 Type II), and 
of course application security itself (automated scanning plus manual 
penetration tests, multi-factor authentication, extremely granular roles and 
access controls, per-application backend encryption of results, flexible 
retention policies, etc.).  

Viewing this with a wider lens, there are a lot of factors involved in 
selecting a tool/service vendor.  One factor that comes into play for us is 
simply that our solution scales, and many others do not.  We can address the 
application supply chain problem in ways that others can't.  

-chris





Chris Eng
Senior Director, Research
Veracode, Inc.
Office: 781.418.3828
Mobile: 617.501.3280
c...@veracode.com 


-Original Message-
From: sc-l-boun...@securecoding.org [mailto:sc-l-boun...@securecoding.org] On 
Behalf Of Jim Manico
Sent: Thursday, February 03, 2011 7:02 PM
To: Chris Wysopal
Cc: Secure Code Mailing List
Subject: Re: [SC-L] InformIT: comparing static analysis tools

Chris,

I've tried to leverage Veracode in recent engagements. Here is how the 
conversation went:

Jim:
"Boss, can I upload all of your code to this cool SaaS service for analysis?"

Client:
"Uh no, and next time you ask, I'm having you committed".

I'm sure you have faced these objections before. How do you work around them?

-Jim Manico
http://manico.net

On Feb 3, 2011, at 1:54 PM, Chris Wysopal  wrote:

> 
> Nice article.  In the 5 years Veracode has been selling static analysis 
> services we have seen the market mature.  In the beginning, organizations 
> were down in the weeds. "What false positive rate or false negative rate does 
> the tool/service have over a test suite such as SAMATE."  Then we saw a move 
> up to looking at the trees.  "Did the tool/service support the Java 
> frameworks I am using?"  Now we are seeing organizations look at the forest. 
> "Can I scale static analysis effectively over all my development sites, my 
> outsourcers, and vendors?"  This is a good sign of a maturing market.
> 
> It is my firm belief that software security has a consumption problem.  We 
> know what the defects are.  We know how to fix them.  We even have automation 
> for detecting a lot of them.  The problem is getting the information and 
> technology to the right person at the right time effectively and managing an 
> organization-wide program.  This is the next challenge for static analysis. 
> I think SaaS based software is more easily consumed and this 
> isn't any different for software security
> 
> -Chris
> 
> -Original Message-
> From: sc-l-boun...@securecoding.org [mailto:sc-l-boun...@securecoding.org] On 
> Behalf Of Gary McGraw
> Sent: Wednesday, February 02, 2011 9:49 AM
> To: Secure Code Mailing List
> Subject: [SC-L] InformIT: comparing static analysis tools
> 
> hi sc-l,
> 
> John Steven and I recently collaborated on an article for informIT.  The 
> article is called "Software [In]security: Comparing Apples, Oranges, and 
> Aardvarks (or, All Static Analysis Tools Are Not Created Equal)" and is 
> available here:
> http://www.informit.com/articles/article.aspx?p=1680863
> 
> Now that static analysis tools like Fortify and Ounce are hitting the 
> mainstream there are many potential customers who want to compare them and 
> pick the best one.  We explain why that's more difficult than it sounds at 
> first and what to watch out for as you begin to compare tools.  We did this 
> in order to get out in front of "test suites" that purport to work for tool 
> comparison.  If you wonder why such suites may not work as advertised, read 
> the article.
> 
> Your feedback is welcome.
> 
> gem
> 
> company www.cigital.com
> podcast www.cigital.com/silverbullet
> blog www.cigital.com/justiceleague
> book www.swsec.com
> 
> ___
> Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, 
> subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as 
> a fre

Re: [SC-L] InformIT: comparing static analysis tools

[Ben Laurie]

On 4 February 2011 09:22, Chris Wysopal  wrote:

>
>
> “Breaking news.  Google says not to use the cloud.  Improving on-premise
> tools is the future.”
>

My view is personal. However, in general, whether the cloud is a good place
for your data depends on your data and the relationship you have with the
cloud provider. If your boss says "no, you can't push this stuff outside our
network" then clearly the cloud is not the right answer (or your boss
doesn't understand the problem).


>
>
> Sorry, I couldn’t help myself. J
>
>
>
> -Chris
>
>
>
> *From:* Ben Laurie [mailto:b...@google.com]
> *Sent:* Friday, February 04, 2011 11:34 AM
> *To:* Jim Manico
> *Cc:* Chris Wysopal; Secure Code Mailing List
> *Subject:* Re: [SC-L] InformIT: comparing static analysis tools
>
>
>
>
>
> On 3 February 2011 16:02, Jim Manico  wrote:
>
> Chris,
>
> I've tried to leverage Veracode in recent engagements. Here is how the
> conversation went:
>
> Jim:
> "Boss, can I upload all of your code to this cool SaaS service for
> analysis?"
>
> Client:
> "Uh no, and next time you ask, I'm having you committed".
>
> I'm sure you have faced these objections before. How do you work around
> them?
>
>
>
> Don't use SaaS, obviously.
>
>
>
> I'd rather see LLVM's static analysis tools get improved (the framework,
> btw, is really nice to work with).
>
>
>
>
> -Jim Manico
> http://manico.net
>
>
> On Feb 3, 2011, at 1:54 PM, Chris Wysopal  wrote:
>
> >
> > Nice article.  In the 5 years Veracode has been selling static analysis
> services we have seen the market mature.  In the beginning, organizations
> were down in the weeds. "What false positive rate or false negative rate
> does the tool/service have over a test suite such as SAMATE."  Then we saw a
> move up to looking at the trees.  "Did the tool/service support the Java
> frameworks I am using?"  Now we are seeing organizations look at the forest.
> "Can I scale static analysis effectively over all my development sites, my
> outsourcers, and vendors?"  This is a good sign of a maturing market.
> >
> > It is my firm belief that software security has a consumption problem.
>  We know what the defects are.  We know how to fix them.  We even have
> automation for detecting a lot of them.  The problem is getting the
> information and technology to the right person at the right time effectively
> and managing an organization-wide program.  This is the next challenge for
> static analysis. I think SaaS based software is more easily
> consumed and this isn't any different for software security
> >
> > -Chris
> >
> > -Original Message-
> > From: sc-l-boun...@securecoding.org [mailto:
> sc-l-boun...@securecoding.org] On Behalf Of Gary McGraw
> > Sent: Wednesday, February 02, 2011 9:49 AM
> > To: Secure Code Mailing List
> > Subject: [SC-L] InformIT: comparing static analysis tools
> >
> > hi sc-l,
> >
> > John Steven and I recently collaborated on an article for informIT.  The
> article is called "Software [In]security: Comparing Apples, Oranges, and
> Aardvarks (or, All Static Analysis Tools Are Not Created Equal)" and is
> available here:
> > http://www.informit.com/articles/article.aspx?p=1680863
> >
> > Now that static analysis tools like Fortify and Ounce are hitting the
> mainstream there are many potential customers who want to compare them and
> pick the best one.  We explain why that's more difficult than it sounds at
> first and what to watch out for as you begin to compare tools.  We did this
> in order to get out in front of "test suites" that purport to work for tool
> comparison.  If you wonder why such suites may not work as advertised, read
> the article.
> >
> > Your feedback is welcome.
> >
> > gem
> >
> > company www.cigital.com
> > podcast www.cigital.com/silverbullet
> > blog www.cigital.com/justiceleague
> > book www.swsec.com
> >
> > ___
> > Secure Coding mailing list (SC-L) SC-L@securecoding.org List
> information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> > List charter available at - http://www.securecoding.org/list/charter.php
> > SC-L is hosted and moderated by KRvW Associates, LLC (
> http://www.KRvW.com) as a free, non-commercial service to the software
> security community.
> > Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> > ___
> >
> > ___
> > Secure Coding mailing list (SC-L) SC-L@securecoding.org
> > List information, subscriptions, etc -
> http://krvw.com/mailman/listinfo/sc-l
> > List charter available at - http://www.securecoding.org/list/charter.php
> > SC-L is hosted and moderated by KRvW Associates, LLC (
> http://www.KRvW.com)
> > as a free, non-commercial service to the software security community.
> > Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> > ___
>
> ___

Re: [SC-L] InformIT: comparing static analysis tools

[Chris Wysopal]

"Breaking news.  Google says not to use the cloud.  Improving on-premise tools 
is the future."

Sorry, I couldn't help myself. J

-Chris

From: Ben Laurie [mailto:b...@google.com]
Sent: Friday, February 04, 2011 11:34 AM
To: Jim Manico
Cc: Chris Wysopal; Secure Code Mailing List
Subject: Re: [SC-L] InformIT: comparing static analysis tools


On 3 February 2011 16:02, Jim Manico 
mailto:jim.man...@owasp.org>> wrote:
Chris,

I've tried to leverage Veracode in recent engagements. Here is how the 
conversation went:

Jim:
"Boss, can I upload all of your code to this cool SaaS service for analysis?"

Client:
"Uh no, and next time you ask, I'm having you committed".

I'm sure you have faced these objections before. How do you work around them?

Don't use SaaS, obviously.

I'd rather see LLVM's static analysis tools get improved (the framework, btw, 
is really nice to work with).


-Jim Manico
http://manico.net

On Feb 3, 2011, at 1:54 PM, Chris Wysopal 
mailto:cwyso...@veracode.com>> wrote:

>
> Nice article.  In the 5 years Veracode has been selling static analysis 
> services we have seen the market mature.  In the beginning, organizations 
> were down in the weeds. "What false positive rate or false negative rate does 
> the tool/service have over a test suite such as SAMATE."  Then we saw a move 
> up to looking at the trees.  "Did the tool/service support the Java 
> frameworks I am using?"  Now we are seeing organizations look at the forest. 
> "Can I scale static analysis effectively over all my development sites, my 
> outsourcers, and vendors?"  This is a good sign of a maturing market.
>
> It is my firm belief that software security has a consumption problem.  We 
> know what the defects are.  We know how to fix them.  We even have automation 
> for detecting a lot of them.  The problem is getting the information and 
> technology to the right person at the right time effectively and managing an 
> organization-wide program.  This is the next challenge for static analysis. 
> I think SaaS based software is more easily consumed and this 
> isn't any different for software security
>
> -Chris
>
> -Original Message-
> From: sc-l-boun...@securecoding.org 
> [mailto:sc-l-boun...@securecoding.org] 
> On Behalf Of Gary McGraw
> Sent: Wednesday, February 02, 2011 9:49 AM
> To: Secure Code Mailing List
> Subject: [SC-L] InformIT: comparing static analysis tools
>
> hi sc-l,
>
> John Steven and I recently collaborated on an article for informIT.  The 
> article is called "Software [In]security: Comparing Apples, Oranges, and 
> Aardvarks (or, All Static Analysis Tools Are Not Created Equal)" and is 
> available here:
> http://www.informit.com/articles/article.aspx?p=1680863
>
> Now that static analysis tools like Fortify and Ounce are hitting the 
> mainstream there are many potential customers who want to compare them and 
> pick the best one.  We explain why that's more difficult than it sounds at 
> first and what to watch out for as you begin to compare tools.  We did this 
> in order to get out in front of "test suites" that purport to work for tool 
> comparison.  If you wonder why such suites may not work as advertised, read 
> the article.
>
> Your feedback is welcome.
>
> gem
>
> company www.cigital.com
> podcast www.cigital.com/silverbullet
> blog www.cigital.com/justiceleague
> book www.swsec.com
>
> ___
> Secure Coding mailing list (SC-L) 
> SC-L@securecoding.org List information, 
> subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as 
> a free, non-commercial service to the software security community.
> Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> ___
>
> ___
> Secure Coding mailing list (SC-L) 
> SC-L@securecoding.org
> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> ___

___
Secure Coding mailing list (SC-L) 
SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php

Re: [SC-L] InformIT: comparing static analysis tools

[Chris Wysopal]

Many of  traditional benefits of SaaS: no software to install, scaling from 
group to enterprise, and ease of central management, make it easier to roll out 
and manage software security programs enterprise wide.  The bigger and more 
diverse an organization is the more these “consumption” benefits kick in.

-Chris

From: Prasad N Shenoy [mailto:prasad.she...@gmail.com]
Sent: Thursday, February 03, 2011 9:02 PM
To: Chris Wysopal
Cc: Gary McGraw; Secure Code Mailing List
Subject: Re: [SC-L] InformIT: comparing static analysis tools

Very well said Chris. Can you explain what you mean by ". I think 
SaaS based software is more easily consumed and this isn't any different for 
software security"

Sent from my iPhone

On Feb 3, 2011, at 2:54 PM, Chris Wysopal 
mailto:cwyso...@veracode.com>> wrote:
. I think SaaS based software is more easily consumed and this 
isn't any different for software security
___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
___

Re: [SC-L] InformIT: comparing static analysis tools

[Chris Wysopal]

Uploading code isn't an issue with software vendors because we are analyzing 
the artifact that they ship to their customer anyway; the executable version of 
their software, not source code.  Unless of course the executable is source 
code which is the case for JSP or PHP, and other scripting languages but they 
are shipping that to their customer so why not send it to us.

If it is an enterprise app that never leaves the four walls of the business 
then the business has to look at our independent Systrust certification from 
E&Y, our independent penetration test results, our employee background checks 
and our NDAs and decide whether it is worth the risk.  For 11 of the top 25 
banks in the world we have passed this test.  We have had due diligence teams 
from 3 letter agencies and Fortune 50 companies come and kick our tires and we 
have never failed to pass this test. Our environment is designed so our 
customers IP, their executables, is only decrypted on an engine analysis 
machine for the duration of the analysis.

Veracode was founded by security people.  We are a security company.  I think 
this shows through in everything we do.

-Chris 

-Original Message-
From: Jim Manico [mailto:jim.man...@owasp.org] 
Sent: Thursday, February 03, 2011 7:02 PM
To: Chris Wysopal
Cc: Gary McGraw; Secure Code Mailing List
Subject: Re: [SC-L] InformIT: comparing static analysis tools

Chris,

I've tried to leverage Veracode in recent engagements. Here is how the 
conversation went:

Jim:
"Boss, can I upload all of your code to this cool SaaS service for analysis?"

Client:
"Uh no, and next time you ask, I'm having you committed".

I'm sure you have faced these objections before. How do you work around them?

-Jim Manico
http://manico.net

On Feb 3, 2011, at 1:54 PM, Chris Wysopal  wrote:

> 
> Nice article.  In the 5 years Veracode has been selling static analysis 
> services we have seen the market mature.  In the beginning, organizations 
> were down in the weeds. "What false positive rate or false negative rate does 
> the tool/service have over a test suite such as SAMATE."  Then we saw a move 
> up to looking at the trees.  "Did the tool/service support the Java 
> frameworks I am using?"  Now we are seeing organizations look at the forest. 
> "Can I scale static analysis effectively over all my development sites, my 
> outsourcers, and vendors?"  This is a good sign of a maturing market.
> 
> It is my firm belief that software security has a consumption problem.  
> We know what the defects are.  We know how to fix them.  We even have 
> automation for detecting a lot of them.  The problem is getting the 
> information and technology to the right person at the right time 
> effectively and managing an organization-wide program.  This is the 
> next challenge for static analysis. I think SaaS based 
> software is more easily consumed and this isn't any different for 
> software security
> 
> -Chris
> 
> -Original Message-
> From: sc-l-boun...@securecoding.org 
> [mailto:sc-l-boun...@securecoding.org] On Behalf Of Gary McGraw
> Sent: Wednesday, February 02, 2011 9:49 AM
> To: Secure Code Mailing List
> Subject: [SC-L] InformIT: comparing static analysis tools
> 
> hi sc-l,
> 
> John Steven and I recently collaborated on an article for informIT.  The 
> article is called "Software [In]security: Comparing Apples, Oranges, and 
> Aardvarks (or, All Static Analysis Tools Are Not Created Equal)" and is 
> available here:
> http://www.informit.com/articles/article.aspx?p=1680863
> 
> Now that static analysis tools like Fortify and Ounce are hitting the 
> mainstream there are many potential customers who want to compare them and 
> pick the best one.  We explain why that's more difficult than it sounds at 
> first and what to watch out for as you begin to compare tools.  We did this 
> in order to get out in front of "test suites" that purport to work for tool 
> comparison.  If you wonder why such suites may not work as advertised, read 
> the article.
> 
> Your feedback is welcome.
> 
> gem
> 
> company www.cigital.com
> podcast www.cigital.com/silverbullet
> blog www.cigital.com/justiceleague
> book www.swsec.com
> 
> ___
> Secure Coding mailing list (SC-L) SC-L@securecoding.org List 
> information, subscriptions, etc - 
> http://krvw.com/mailman/listinfo/sc-l
> List charter available at - 
> http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as 
> a free, non-commercial service to the software security community.
> Follow KRvW Associates on Twitter at: 
> http://twitter.com/KRvW_Associates
> ___
> 
> ___
> Secure Coding mailing list (SC-L) SC-L@securecoding.org List 
> information, subscriptions, etc - 
> http://krvw.com/mailman/listinfo/sc-l
> List charter available at - 
> http://www.securecodin

Re: [SC-L] InformIT: comparing static analysis tools

[Ben Laurie]

On 3 February 2011 16:02, Jim Manico  wrote:

> Chris,
>
> I've tried to leverage Veracode in recent engagements. Here is how the
> conversation went:
>
> Jim:
> "Boss, can I upload all of your code to this cool SaaS service for
> analysis?"
>
> Client:
> "Uh no, and next time you ask, I'm having you committed".
>
> I'm sure you have faced these objections before. How do you work around
> them?
>

Don't use SaaS, obviously.

I'd rather see LLVM's static analysis tools get improved (the framework,
btw, is really nice to work with).


>
> -Jim Manico
> http://manico.net
>
> On Feb 3, 2011, at 1:54 PM, Chris Wysopal  wrote:
>
> >
> > Nice article.  In the 5 years Veracode has been selling static analysis
> services we have seen the market mature.  In the beginning, organizations
> were down in the weeds. "What false positive rate or false negative rate
> does the tool/service have over a test suite such as SAMATE."  Then we saw a
> move up to looking at the trees.  "Did the tool/service support the Java
> frameworks I am using?"  Now we are seeing organizations look at the forest.
> "Can I scale static analysis effectively over all my development sites, my
> outsourcers, and vendors?"  This is a good sign of a maturing market.
> >
> > It is my firm belief that software security has a consumption problem.
>  We know what the defects are.  We know how to fix them.  We even have
> automation for detecting a lot of them.  The problem is getting the
> information and technology to the right person at the right time effectively
> and managing an organization-wide program.  This is the next challenge for
> static analysis. I think SaaS based software is more easily
> consumed and this isn't any different for software security
> >
> > -Chris
> >
> > -Original Message-
> > From: sc-l-boun...@securecoding.org [mailto:
> sc-l-boun...@securecoding.org] On Behalf Of Gary McGraw
> > Sent: Wednesday, February 02, 2011 9:49 AM
> > To: Secure Code Mailing List
> > Subject: [SC-L] InformIT: comparing static analysis tools
> >
> > hi sc-l,
> >
> > John Steven and I recently collaborated on an article for informIT.  The
> article is called "Software [In]security: Comparing Apples, Oranges, and
> Aardvarks (or, All Static Analysis Tools Are Not Created Equal)" and is
> available here:
> > http://www.informit.com/articles/article.aspx?p=1680863
> >
> > Now that static analysis tools like Fortify and Ounce are hitting the
> mainstream there are many potential customers who want to compare them and
> pick the best one.  We explain why that's more difficult than it sounds at
> first and what to watch out for as you begin to compare tools.  We did this
> in order to get out in front of "test suites" that purport to work for tool
> comparison.  If you wonder why such suites may not work as advertised, read
> the article.
> >
> > Your feedback is welcome.
> >
> > gem
> >
> > company www.cigital.com
> > podcast www.cigital.com/silverbullet
> > blog www.cigital.com/justiceleague
> > book www.swsec.com
> >
> > ___
> > Secure Coding mailing list (SC-L) SC-L@securecoding.org List
> information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> > List charter available at - http://www.securecoding.org/list/charter.php
> > SC-L is hosted and moderated by KRvW Associates, LLC (
> http://www.KRvW.com) as a free, non-commercial service to the software
> security community.
> > Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> > ___
> >
> > ___
> > Secure Coding mailing list (SC-L) SC-L@securecoding.org
> > List information, subscriptions, etc -
> http://krvw.com/mailman/listinfo/sc-l
> > List charter available at - http://www.securecoding.org/list/charter.php
> > SC-L is hosted and moderated by KRvW Associates, LLC (
> http://www.KRvW.com)
> > as a free, non-commercial service to the software security community.
> > Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> > ___
>
> ___
> Secure Coding mailing list (SC-L) SC-L@securecoding.org
> List information, subscriptions, etc -
> http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> ___
>
___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (h

Re: [SC-L] InformIT: comparing static analysis tools

[Steven M. Christey]

Jim,

Maybe you would have had more success if you explicitly said "in the 
cloud" ;-)


- Steve


On Thu, 3 Feb 2011, Jim Manico wrote:


Chris,

I've tried to leverage Veracode in recent engagements. Here is how the 
conversation went:


Jim: "Boss, can I upload all of your code to this cool SaaS service for 
analysis?"

___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
___

Re: [SC-L] InformIT: comparing static analysis tools

[Gary McGraw]

hi arian,

Glad you liked the article.

I guess my brush was a bit too wide when it comes to dynamic testing.  I
was really only referring to the Web application testing tools which in my
mind "hit the wall" for two reasons.  Reason one is that they only work
over port80 and are designed to take advantage of the fact that HTTP is a
stateless protocol (with a few small caveats).  IMPORTANT NOTE: lots of
software is not web software (sorry Jeremiah).  Reason two has to do with
the canned nature of the tests.  The generic tests in the black box Web
app testing tools are, well, generic.  If your software falls prey to
those tests, it sucks.  IMPORTANT NOTE: Lots of software does, in fact,
suck.

As you probably know I call those tools "badness-ometers" and also
***suggest that everyone buy and use one***. See this ancient post (and
associated informIT article) from 2007:

http://www.cigital.com/justiceleague/2007/03/19/badness-ometers-are-good-do
-you-own-one/

Now, there are many other kinds of dynamic testing tools (think of any
kind of fault injection tool).  I wrote a software engineering tome about
that way back in 1998 called Software Fault Injection
.  And you are right that dynamic testing has
a place.  However, short of fuzzing tools generally tied to a
grammar-based protocol and capture-replay tools there are not very many
dynamic testing tools that work for non-Web software.  Why not?  Because
genericizing is too hard, making the potential market for a particular
tool too small.

Security testing plays a key role in the Touchpoints (my own and Cigital's
approach to SDLC integration) which are described in Software Security
.  Hoglund and I also describe some dynamic tools that
we screwed around with when writing Exploiting Software in 2004
.  I am in complete agreement that
dynamic testing is important for software security.

One quibble with your question.  You can't push dynamic testing very far
back in the SDLC, because your code has to run before you can test it
dynamically.  For me, way back in the SDLC means architectural risk
analysis or even security requirements analysis.

Sorry for the multiple invocations of the way back machine!  I must be
getting old.

gem

company www.cigital.com
podcast www.cigital.com/silverbullet
blog www.cigital.com/justiceleague
book www.swsec.com



On 2/3/11 7:26 PM, "Arian J. Evans"  wrote:

>Great article, Gary. Many of your comments about static technology
>challenges I have seen and verified first-hand, including
>multi-million dollar cost overruns. After some great dialogue with
>John Stevens, I suspect we have had similar experiences.
>
>I was just about to write a similar article at a higher level - about
>how the vast majority of enterprise customers I work with are actively
>moving security into the SDLC. The time has come, the event has
>tipped, and SDLC security is indeed mainstream. This is an exciting
>time to be in the industry.
>
>However - I was curious about your comments about dynamic tools
>"reaching their limit" or something like that, as customers move
>security efforts deeper into the SDLC. What does that mean?
>
>I see customers making extensive use of dynamic testing, and
>leveraging it deeper and deeper into the SDLC. Enterprises are
>aggressively rolling out and expanding dynamic testing earlier in the
>SDLC. Newer dynamic testing technologies help solve/reduce some of the
>key pain points that static technologies alone are causing them, as
>you so well illustrated..
>.
>I am very interested in why you sound dismissive of these successful
>technologies? Your article makes it sound like they are hitting some
>invisible limit, when in fact hundreds of enterprises are expanding
>dynamic testing in the SDLC. And these are serious projects that run
>into the 7-figures.
>
>Any insight you can share would be appreciated!
>
>Great work identifying the general shift SDLC security is moving
>mainstream,
>
>---
>Arian Evans
>Software Security Referee
>
>
>
>On Wed, Feb 2, 2011 at 6:48 AM, Gary McGraw  wrote:
>> hi sc-l,
>>
>> John Steven and I recently collaborated on an article for informIT.
>>The article is called "Software [In]security: Comparing Apples, Oranges,
>>and Aardvarks (or, All Static Analysis Tools Are Not Created Equal)" and
>>is available here:
>> http://www.informit.com/articles/article.aspx?p=1680863
>>
>> Now that static analysis tools like Fortify and Ounce are hitting the
>>mainstream there are many potential customers who want to compare them
>>and pick the best one.  We explain why that's more difficult than it
>>sounds at first and what to watch out for as you begin to compare tools.
>> We did this in order to get out in front of "test suites" that purport
>>to work for tool comparison.  If you wonder why such suites may not work
>>as advertised, read the article.
>>
>> Your feedback is welcome.
>>
>> gem
>>
>> company www.cigital.com
>> podca

[SC-L] free and open online secure coding in C course module

[Robert Seacord]

CERT has completed the development of an integer module for our "Secure Coding 
in C" course. A demo course set up at http://oli.web.cmu.edu Enter the course 
key: seccode

The course is open and free. If you want to use the course at your University, 
College, Corporation, or other organization you can register as an instructor 
and we can set up a course for you.

As always, I'm interested in your review and comments.

Thanks,
rCs

___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
___

Re: [SC-L] InformIT: comparing static analysis tools

[Prasad N Shenoy]

Very well said Chris. Can you explain what you mean by ". I think 
SaaS based software is more easily consumed and this isn't any different for 
software security"

Sent from my iPhone

On Feb 3, 2011, at 2:54 PM, Chris Wysopal  wrote:

> . I think SaaS based software is more easily consumed and this 
> isn't any different for software security
___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
___