| Most recently, we have met with a variety of vendors including but not | limited to: Coverity, Ounce Labs, Fortify, Klocwork, HP and so on. In | the conversation they all used interesting phrases to describe they | classify their competitors value proposition. At some level, this has | managed to confuse me and I would love if someone could provide a way to | think about this in a more boolean way. | | - So when a vendor says that they are focused on quality and not | security, and vice versa what exactly does this mean? I don't have a | great mental model of something that is a security concern that isn't a | predictor of quality. Likewise, in terms of quality, other than | producing metrics on things such as depth of inheritance, cyclomatic | complexity, etc wouldn't bad numbers here at least be a predictor of a | bad design and therefore warrant deeper inspection from a security | perspective? What you're seeing here is an attempt to control the set of measure- ments that will be applied to your product. If you say you have a product that tests "security", on the one hand, people will ask you whether out-of-the-box you detect some collection of known security bugs. That collection might be good, or it might contain all kinds of random junk gathered from 20 years of Internet reports. As a vendor, I don't (quite reasonably) want to be measured on some unpredictable, uncontrollable set of "hard" (easily quantifiable) standards.
Beyond that, saying you provide "security" gets you right in to the maelstrom of security standards, certification, government and accounting requirements, etc. Should there be a security problem with customer code that your product accepted, if you said you provided security testing, expect to be pulled in to any bad publicity, lawsuits, etc. It's so much safer to say you help ensure quality. That's virtually impossible to quantify or test, and you clearly can only be one part of a much larger process. If things go wrong, it's very hard for the blame to fall on your product (assuming it's any good at all). After all, if you pointed out 100 issues, the fixing of which clearly improved quality, well, someone else must have screwed up in letting one other issue through that had a quality impact - quality is a result of the whole process, not any one element of it. "You can't test quality in." (This sounds like excuses, but it happens also to be mainly true!) | - Even if the rationale is more about people focus rather than tool | capability, is there anything else that would prevent one tool from | serving both purposes? Well, I can speak to two products whose presentations I saw about a year ago. Fortify concentrated on security. They made the point that they had hundreds of pre-programmed tests specific to known security vulnerabilities. The weaknesses in their more general analysis - e.g, they had almost no ability to do value flow analysis - they could dismiss as not relevant to their specific mission of finding security flaws. Coverity, on the other hand, did very deep general analysis, but for the most part found security flaws as a side-effect of general fault-finding mechanisms. Fortify defenders - including some on this list - pointed out this limitation, saying that the greatest value of Fortify came from the extensive work that had gone into analyzing actual security holes in real code and making sure they would be caught. In practice, while there was an overlap between what the two products found, neither subsumed the other. If you can afford it, I'd use both. (In theory, you could produce a security hole library for Coverity that copied the one in Fortify. But no one is doing that.) | - Is it reasonable to expect that all of the vendors in this space will | have the ability to support COBOL, Ruby and Smalltalk sometime next year | so that customers don't have to specifically request it? Well, the effort depends on the nature of the product. Products that mainly scan for particular patterns are probably easier to adjust to work with other languages than products that do deeper analysis. On the other hand, you have to build up a library of patterns to scan for, which to some degree will be language-specific. Products that depend on deep analysis of data flow and such usually don't care much about surface syntax - a new front end isn't a big deal - but will run into fundamental problems if they rely on static properties of programs, like static types. Languages like Smalltalk and even more Ruby are likely to be very problematic for this kind of analysis, since so much is subject to change at run time. | - Do the underlying compilers need to do something different since | languages such as COBOL aren't object-oriented which would make analysis | a bit different? I doubt object orientation matters all that much. In general, anything that provides (a) static information (b) encapsulation makes analysis easier. COBOL is old enough to have elements that amount to self- modifying code, for which good analysis methods don't exist. But few COBOL programs have used the more dangerous features in many years, so in practice this may be a non-issue. On the other hand, languages like Ruby have resurrected - in a much more controlled fashion - these same methods, so would be difficult to work with. I haven't really kept up with the most recent work, but I doubt the techniques to do any deep analysis of even well-behaved Ruby programs - or, hell, even Java programs that use introspection and dynamic loading and such - exist yet. The fully-enforced dynamic type systems in such languages avoid many significant classes of errors common in other kinds of languages, so the very questions you need to ask are different, and I'm not even sure we know what the right questions are! -- Jerry | | ************************************************************************* | This communication, including attachments, is | for the exclusive use of addressee and may contain proprietary, | confidential and/or privileged information. If you are not the intended | recipient, any use, copying, disclosure, dissemination or distribution is | strictly prohibited. If you are not the intended recipient, please notify | the sender immediately by return e-mail, delete this communication and | destroy all copies. | ************************************************************************* | | | _______________________________________________ | Secure Coding mailing list (SC-L) SC-L@securecoding.org | List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l | List charter available at - http://www.securecoding.org/list/charter.php | SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) | as a free, non-commercial service to the software security community. | _______________________________________________ | | _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________