Mattyson -- I almost complete agree with you. I will say - during ongoing "deep dive" assessments, we commonly find that applications that have one or more authC/Z issues at launch, will reintroduce them over time, if they write and push a lot of code. Anecdotally I would say we see at least one major Auth issue (or similar critical issue) per year in volatile code, over time, during ongoing assessments. (I'll do some hard math later on this.)
The recurring issues are usually singular in nature, and not systemic. I'll give you an example: so you find that an app has a broken auth system; let's say it is a J2EE app, pre-MVC. Fundamental auth design-issue. To remediate they write an auth.Request.Routing servlet, and all servlets are only supposed to take callers that come through auth.Request.Routing. Design issue solved. So about once a year someone pushes a servlet (or form) with sensitive/critical functions, that does not properly call auth.Request.Routing. So now you have an implementation (or process, code review, peer review, SDL) problem. You get the idea. Insert all your points. So regular, or ongoing deep dives have a lot of value for high-risk applications to figure out which of the rest of the "system" broke. :) I agree "scanner jockey" work definitely has a place, and so do ongoing deep dives. The two can be combined with the right platform. Next up is integrating all these different types of analysis, and leveraging them all to provide better business context. There are cost-effective ways to enable people to do deeper blackbox and whitebox, and get better (and more contextual) results, I want to promote that. (Contextual Black Box vs. Blind Black Box). But I will save that for another chapter in this discussion. nota bene: In case I sounded too critical I should add -- one of the best things I hear in the field about Veracode is the strength of their customer-first/customer-centric focus and support, which is near and dear to my own heart. That should be a model for us all, and I wouldn't want to partner/work with with anyone who did things differently. I also hear Veracode is very good about customizing/tuning results of static analysis for clients. I think customization for individual customers is an essential reality for all types of analysis of custom code and business needs. Once again I think this is an important lesson for us all, and in my biased opinion: a major strength of SaaS appsec offerings. Any time you have to eat your own dogfoot, *and* satisfy clients on a recurring basis: you are going to get better fast. ChrisW could also claim I owe him a large intellectual debt for discussions over the years, but thankfully he's too polite for that. :) btw// I used to joke about how the BUFD (big up front design) crowd almost always addresses security with a BFPT (big final pen test) before shipping. So, possibly, we had that discussion over beers. Dusseldorf? ;) Anyway, great discussion. If you look back @SC-L three years ago, discussions like this show how much the industry and customer needs are evolving. Good stuff. Cheers, -- Arian Evans On Wed, Aug 5, 2009 at 7:14 PM, Matt Fisher<m...@piscis-security.com> wrote: >>I think anyone who has experience with deep dynamic testing knows they >>need automation tools with custom configuration ability, the ability to >>record workflow, a framework to create custom tests, etc. > > Absolutely. But Arian there are differing deployment models. You don't just > touch an application once in it's life and leave it, right ? You're doing > architecture reviews, reviewing the functional requirement and RBACs, > reviewing code, doing integrated security testing, doing a final validation > (or as a friend once put it over drinks " the big giant pen-test"). For any > of those activities, you need real live, experienced skilled testers. > > Once it goes live, however, you may very well have a SOC, NOC, or even > "security" team who is tasked with the continual scanning and "monitoring" of > their space who's goal is to touch everything - however lightly - at least > once very x days. For this type of scenario where bulk scalability counts > over quality - AND A QUALITY ASSESSMENT AND VALIDATION WAS ALREADY PERFORMED- > I would suggest a scanner monkey may be appropriate. Of course you would > NEVER want that to be your ONLY assessment or validation. > > Chris, SPI had a product called DevInspect that performed static and dynamic > analysis as a single product, and was definitely around before Aug '07. Not > saying it was red-hot, just saying it was there. > > I'd like to see NTO. Given the slower dev times of the larger companies and > begrudgingly slow addition of core capabilities to them, I'm really hoping > that some of the "smaller guys" end up growing and filling niches. For > instance, I've heard that one smaller player crawls every bit as well as a > major player, and *much* better than the other major player, but while > costing considerably less than either. NTO reps, feel free to spam me (me, > not the list). > > I will say this: Chris I'm completely with you in that I'm convinced that the > majority of the market buying scanners is not doing so based on any objective > empirical testing, but rather on "who found what" or what they "like". I'm > even saddened to say that I recently saw a presentation by an organization > tasked and paid to perform objective empirical analysis of scanners, that > literally ranked them based on what they found, with absolutely no testing > ground truth. > > I'm even more strongly convinced that the majority of those running these > tools completely underestimate the expertise required to properly operate > them and realize full potential from them. Given the complexity of testing > software these days you still really need to know what you're doing to eak > out of them what little value they hold. Even with realizing their full > potential, however, there's still a lot of work to be done beyond a scan to > perform anything resembling a complete assessment. Of course, a human > assisted SaaS model has the potential to fill the gap, but from what I'm the > majority of organizations using scanners like WI and AS in-house don't. Heck, > even some really big name firms selling rather expensive fancily marketed > assessments don't. > > Shame, really. > > -Matt. > > > -----Original Message----- > From: Chris Wysopal [mailto:cwyso...@veracode.com] > Sent: Tuesday, August 04, 2009 8:54 PM > To: Arian J. Evans; Matt Fisher > Cc: Kenneth Van Wyk; Secure Coding > Subject: RE: [SC-L] IBM Acquires Ounce Labs, Inc. > > > I wouldn't say that NTO Spider is a "sort of" dynamic web scanner. It is a > top tier scanner that can battle head to head on false negative rate with the > big conglomerates' scanners: IBM AppScan and HP WebInspect. Larry Suto > published an analysis a year ago, that certainly had some flaws (and was > rightly criticized), but genuinely showed all three to be in the same league. > I haven't seen a better head-to-head analysis conducted by anyone. A little > bird whispered to me that we may see a new analysis by someone soon. > > As a group of security practitioners it is amazing to me that we don't have > more quantifiable testing and tools/services are just dismissed with > anecdotal data. I am glad NIST SATE '09 will soon be underway and, at least > for static analysis tools, we will have unbiased independent testing. I am > hoping for a big improvement over last year. I especially like the category > they are using for some flaws found as "valid but insignificant". Clearly > they are improving based on feedback from SATE '08. > > Veracode was the first company to offer static and dynamic (web) analysis, > and we have been for 2 years (announced Aug 8, 2007). We deliver it as a > service. If you have a .NET or Java web app, you would cannot find a > comparable solution form a single vendor today. > > -Chris > > -----Original Message----- > From: sc-l-boun...@securecoding.org [mailto:sc-l-boun...@securecoding.org] On > Behalf Of Arian J. Evans > Sent: Tuesday, July 28, 2009 1:41 PM > To: Matt Fisher > Cc: Kenneth Van Wyk; Secure Coding > Subject: Re: [SC-L] IBM Acquires Ounce Labs, Inc. > > Right now, officially, I think that is about it. IBM, Veracode, and > AoD (in Germany) claims they have this too. > > As Mattyson mentioned, Veracode only does static binary analysis (no > source analysis). They offer "dynamic scanning" but I believe it is > using NTO Spider IIRC which is a simplified scanner that targets > unskilled users last I saw it. > > At one point I believe Veracode was in discussions with SPI to use WI, > but since the Veracoders haunt this list I'll let them clarify what > they use if they want. > > So IBM: soon. > > Veracode: sort-of. > > AoD: on paper > > And more to come in short order no doubt. I think we all knew this was > coming sooner or later. Just a matter of "when". > > The big guys have a lot of bucks to throw at this problem if they want > to, and pull off some really nice integrations. Be interesting to see > what they do, and how useful the integrations really are to > organizations. > > -- > Arian Evans > > > > > > On Tue, Jul 28, 2009 at 9:29 AM, Matt Fisher<m...@piscis-security.com> wrote: >> Pretty much. Hp /spi has integrations as well but I don't recall devinspect >> ever being a big hit. Veracode does both as well as static binary but as >> asaas model. Watchfire had a RAD integration as well iirc but it clearly >> must not haved had the share ounce does. >> >> -----Original Message----- >> From: Prasad Shenoy <prasad.she...@gmail.com> >> Sent: July 28, 2009 12:22 PM >> To: Kenneth Van Wyk <k...@krvw.com> >> Cc: Secure Coding <SC-L@securecoding.org> >> Subject: Re: [SC-L] IBM Acquires Ounce Labs, Inc. >> >> >> Wow indeed. Does that makes IBM the only vendor to offer both Static >> and Dynamic software security testing/analysis capabilities? >> >> Thanks & Regards, >> Prasad N. Shenoy >> >> On Tue, Jul 28, 2009 at 10:19 AM, Kenneth Van Wyk<k...@krvw.com> wrote: >>> Wow, big acquisition news in the static code analysis space announced today: >>> >>> http://news.prnewswire.com/DisplayReleaseContent.aspx?ACCT=104&STORY=/www/story/07-28-2009/0005067166&EDATE= >>> >>> >>> Cheers, >>> >>> Ken >>> >>> ----- >>> Kenneth R. van Wyk >>> KRvW Associates, LLC >>> http://www.KRvW.com >>> >>> (This email is digitally signed with a free x.509 certificate from CAcert. >>> If you're unable to verify the signature, try getting their root CA >>> certificate at http://www.cacert.org -- for free.) >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Secure Coding mailing list (SC-L) SC-L@securecoding.org >>> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l >>> List charter available at - http://www.securecoding.org/list/charter.php >>> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) >>> as a free, non-commercial service to the software security community. >>> _______________________________________________ >>> >>> >> _______________________________________________ >> Secure Coding mailing list (SC-L) SC-L@securecoding.org >> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l >> List charter available at - http://www.securecoding.org/list/charter.php >> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) >> as a free, non-commercial service to the software security community. >> _______________________________________________ >> >> _______________________________________________ >> Secure Coding mailing list (SC-L) SC-L@securecoding.org >> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l >> List charter available at - http://www.securecoding.org/list/charter.php >> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) >> as a free, non-commercial service to the software security community. >> _______________________________________________ >> > > _______________________________________________ > Secure Coding mailing list (SC-L) SC-L@securecoding.org > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l > List charter available at - http://www.securecoding.org/list/charter.php > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) > as a free, non-commercial service to the software security community. > _______________________________________________ > _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________
