Chris -- Good point with Larry's paper. NTO Spider is, by design, a simplified scanner for unskilled users, and I do not think it was designed to be an effective tool for deep dynamic analysis of a web application. It is, however, probably the best scanner on the market for people who don't have the time or skill to configure dynamic testing tools for their applications!
Larry Suto's paper reinforces this use-case desktop webapp scanners: "Each scanner was run in default mode and not tuned in any capacity to the application. The importance of this lies in how effective the default mode so that scalability of scanning is not limited by manual intervention and setup procedures which can be very time consuming. Second, in most cases it is simply unrealistic to spend much time with many applications." While I definitely look forward to more objective data on this subject, I do not think Suto's report is a good example of how consultants or SaaS providers deliver "expert security analysis" when they use dynamic testing automation IMO. (I know this is not how I ever did things.) I think anyone who has experience with deep dynamic testing knows they need automation tools with custom configuration ability, the ability to record workflow, a framework to create custom tests, etc. I do not believe NTO spider offers any of these essential features. I believe it was explicitly designed for unskilled users' use-cases. (A valid and important market to be sure.) Admittedly I could be very wrong here. The NTO guys are sharp folks and I haven't seen Spider in a while. > As a group of security practitioners it is amazing to me that we don't have > more quantifiable testing and tools/services are just dismissed with > anecdotal data. Completely agreed. I prefer to back up my statements about dynamic tools with hard, quantifiable data. Deprived of that, I tend to rely on historical experience, if it is a subject I have enough experience on within that problem domain. If you recall I used to do extensive testing and benchmarking of dynamic testing tools across custom widgets I wrote, and production enterprise applications, and publish them @OWASP and NIST conferences, and "HE: Webapps 2nd Ed". Back then the quality of the tools was very volatile, and changed significantly every release, and from application to application you would test, so by the time you vetted all your data, it was almost obsolete. Again the importance of customizable, manually guided tools, when dealing with new and bespoke applications. I tried to tackle static analysis but became overwhelmed by the challenge of setting up effective labs, and the huge array of static analysis tools that were available. Given that I now work on a dynamic testing platform: it would be completely fair to accuse me of being "non-objective" when discussing various vendors dynamic testing tools -- and I would have to agree with you. It won't make my statements any less valid, but I have to throw that out there to be fair. Ultimately you hit the need for objective data spot-on. I would be lying if I didn't say that I would LOVE to see more head-on benchmarking between static analysis technology vendors like Veracode, Fortify, Ounce, Coverity, Klockwork, etc. etc. The problem I had in the past with benchmarks was the huge degree of customization in each application I would test. While patterns emerge that are almost always automatable to some degree, the technologies almost always require hand care-and-feeding to get them to an effective place. I think this notion of combining the tools with qualified users is the true potential power of the SaaS solutions that are coming to market. I look forward to seeing the release of more objective analysis by smarter minds than I, and am very impressed with how far things have come since the simple tests I tried to run over the years. $0.02. Cheers, -- Arian Evans On Tue, Aug 4, 2009 at 5:54 PM, Chris Wysopal<cwyso...@veracode.com> wrote: > > I wouldn't say that NTO Spider is a "sort of" dynamic web scanner. It is a > top tier scanner that can battle head to head on false negative rate with the > big conglomerates' scanners: IBM AppScan and HP WebInspect. Larry Suto > published an analysis a year ago, that certainly had some flaws (and was > rightly criticized), but genuinely showed all three to be in the same league. > I haven't seen a better head-to-head analysis conducted by anyone. A little > bird whispered to me that we may see a new analysis by someone soon. > > As a group of security practitioners it is amazing to me that we don't have > more quantifiable testing and tools/services are just dismissed with > anecdotal data. I am glad NIST SATE '09 will soon be underway and, at least > for static analysis tools, we will have unbiased independent testing. I am > hoping for a big improvement over last year. I especially like the category > they are using for some flaws found as "valid but insignificant". Clearly > they are improving based on feedback from SATE '08. > > Veracode was the first company to offer static and dynamic (web) analysis, > and we have been for 2 years (announced Aug 8, 2007). We deliver it as a > service. If you have a .NET or Java web app, you would cannot find a > comparable solution form a single vendor today. > > -Chris > > -----Original Message----- > From: sc-l-boun...@securecoding.org [mailto:sc-l-boun...@securecoding.org] On > Behalf Of Arian J. Evans > Sent: Tuesday, July 28, 2009 1:41 PM > To: Matt Fisher > Cc: Kenneth Van Wyk; Secure Coding > Subject: Re: [SC-L] IBM Acquires Ounce Labs, Inc. > > Right now, officially, I think that is about it. IBM, Veracode, and > AoD (in Germany) claims they have this too. > > As Mattyson mentioned, Veracode only does static binary analysis (no > source analysis). They offer "dynamic scanning" but I believe it is > using NTO Spider IIRC which is a simplified scanner that targets > unskilled users last I saw it. > > At one point I believe Veracode was in discussions with SPI to use WI, > but since the Veracoders haunt this list I'll let them clarify what > they use if they want. > > So IBM: soon. > > Veracode: sort-of. > > AoD: on paper > > And more to come in short order no doubt. I think we all knew this was > coming sooner or later. Just a matter of "when". > > The big guys have a lot of bucks to throw at this problem if they want > to, and pull off some really nice integrations. Be interesting to see > what they do, and how useful the integrations really are to > organizations. > > -- > Arian Evans > > > > > > On Tue, Jul 28, 2009 at 9:29 AM, Matt Fisher<m...@piscis-security.com> wrote: >> Pretty much. Hp /spi has integrations as well but I don't recall devinspect >> ever being a big hit. Veracode does both as well as static binary but as >> asaas model. Watchfire had a RAD integration as well iirc but it clearly >> must not haved had the share ounce does. >> >> -----Original Message----- >> From: Prasad Shenoy <prasad.she...@gmail.com> >> Sent: July 28, 2009 12:22 PM >> To: Kenneth Van Wyk <k...@krvw.com> >> Cc: Secure Coding <SC-L@securecoding.org> >> Subject: Re: [SC-L] IBM Acquires Ounce Labs, Inc. >> >> >> Wow indeed. Does that makes IBM the only vendor to offer both Static >> and Dynamic software security testing/analysis capabilities? >> >> Thanks & Regards, >> Prasad N. Shenoy >> >> On Tue, Jul 28, 2009 at 10:19 AM, Kenneth Van Wyk<k...@krvw.com> wrote: >>> Wow, big acquisition news in the static code analysis space announced today: >>> >>> http://news.prnewswire.com/DisplayReleaseContent.aspx?ACCT=104&STORY=/www/story/07-28-2009/0005067166&EDATE= >>> >>> >>> Cheers, >>> >>> Ken >>> >>> ----- >>> Kenneth R. van Wyk >>> KRvW Associates, LLC >>> http://www.KRvW.com >>> >>> (This email is digitally signed with a free x.509 certificate from CAcert. >>> If you're unable to verify the signature, try getting their root CA >>> certificate at http://www.cacert.org -- for free.) >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Secure Coding mailing list (SC-L) SC-L@securecoding.org >>> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l >>> List charter available at - http://www.securecoding.org/list/charter.php >>> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) >>> as a free, non-commercial service to the software security community. >>> _______________________________________________ >>> >>> >> _______________________________________________ >> Secure Coding mailing list (SC-L) SC-L@securecoding.org >> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l >> List charter available at - http://www.securecoding.org/list/charter.php >> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) >> as a free, non-commercial service to the software security community. >> _______________________________________________ >> >> _______________________________________________ >> Secure Coding mailing list (SC-L) SC-L@securecoding.org >> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l >> List charter available at - http://www.securecoding.org/list/charter.php >> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) >> as a free, non-commercial service to the software security community. >> _______________________________________________ >> > > _______________________________________________ > Secure Coding mailing list (SC-L) SC-L@securecoding.org > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l > List charter available at - http://www.securecoding.org/list/charter.php > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) > as a free, non-commercial service to the software security community. > _______________________________________________ > _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________
