(Apologies if I already sent this to the group; I don't think I did.) There's an interesting presentation at http://www.iarpa.gov/stonesoup_Merced_DHSAWGbrief.pdf about a study done by the US NSA (National Security Agency) of C and Java source code analysis tools. They developed a synthetic test suite, and then ran six tools against the Java version and five tools against the C version (the specific tools and versions used are identified in the presentation). None of the tools found all of the problems, and 40% of the problems weren't found by any of the tools. Even where the problems were found, there was a surprising level of inconsistency among the tools.
Unfortunately, there's not much detail in the presentation. There's a report that's been written, but so far not approved for release (or so I'm told). I don't know whether the issue is classification (they don't want the bad guys to know what sort of things can sneak past their detectors), or proprietary information, or just bureaucracy. It would be interesting to hear comments from vendors on the list as to the limitations on such a test (certainly using synthetic programs isn't realistic), as well as whether they've adapted the tools to find more of these types of problems. --Jeremy P.S. The report is undated, but I believe it's fairly recent. _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________