Re: [SC-L] Microsoft Pushes Secure, Quality Code
I am in full agreement that we need metrics. The challenge is that syntactic metric are easy to compute and not very useful from a management perspective and that business-relevant metrics are much fuzzier and difficult to compute given a glob of code. That said, we should keep trying! I believe one answer is to take advantage of relative metrics over time. gem company www.cigital.com --Original Message-- From: Steven M. Christey To: Gary McGraw Cc: Steven M. Christey Cc: Secure Coding Mailing List Sent: Oct 8, 2007 4:07 PM Subject: RE: [SC-L] Microsoft Pushes Secure, Quality Code On Mon, 8 Oct 2007, Gary McGraw wrote: Not surprising. Last time I looked, attack surface is subjective. McCabe is not. BTW, McCabe's Cyclomatic complexity boils down to 85% lines of code and 15% data flow if you do a principal component analysis on it. Hopefully the SEI people are monitoring this list and can provide their feedback. They've done some concrete work in making attack surface as objective as possible, enough to the point where they compared 2 FTP servers about a year ago. One of their papers comments that they wanted to use the code scanners to make the calculations for them, but for some reason they couldn't. I was under the impression from Mike Howard's comments over the years, that MS had some concrete (perhaps subjective) comparisons between different MS variants, and this was part of the argument for Vista's security over past MS operating systems. Just throw the code in the box and turn the crank. Then discard the results and you're done! While I understand the sentiment, it seems to me that you can't get very far without metrics of some sort. Perhaps more importantly, the real decision-makers need them because it's not their job (and probably not their expertise) to pore through endless details. - Steve ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___
Re: [SC-L] Microsoft Pushes Secure, Quality Code
That said, we should keep trying! I believe one answer is to take advantage of relative metrics over time. I agree that this can be a practical starting point for organizations. I had a client starting down the path with static analysis, they have thousands of developers and many applications. They have a small software security team and they obviously cannot scan every single app. Worse, if they find something they don't necessarily have the governance in place to make sure that a lot of what they find gets addressed. So what we did was to get the CIO to give them one silver bullet a month. They scanned 8-10 apps per month, and whichever one came up worst based on the metrics in the group had to remediate. This approach has some incremental benefits - 1) it gets security out of the its perfect or its broken business 2) at least one project per month makes measurable improvements 3) the projects are not being compared to an ivory tower but rather to their peers who have to deliver under the same constraints, making the suggested remediations more palatable to the developers. There is no way to relativity, relativity is the way. -gp ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___
Re: [SC-L] Microsoft Pushes Secure, Quality Code
Hi Steven, I'm (with Vadim Okun) currently doing some research and prototype development in that direction. We are actually counting the number of diffused inputs (diffuse in a sense of affectation to other variables, even with filter application, etc.) going through sinks. We are working on PHP code only for now since we have to work pretty much from scratch (using yaxx in order to generate the AST), but we started to do evaluation of real code (wordpress, mediawiki, dotclear, joomla etc.). We also plan to try different combination of possible metrics, and see the correlation between them. But well, the main problem with such a metric is that's it's strongly related to how the programmer is working: - Is it better to have lots of different variables that are a variation of a single input? I thought not... - Is it better to have localized inputs in the source code? I think yes... - Shall we count the number of classes, the Object orientation of the code, the number of functions... also? These are some questions that we are currently working one. If you guys have some ideas about that or comments, I would really appreciate :) Romain http://rgaucher.info Steven M. Christey wrote: Interesting that attack surface isn't included, given that Microsoft was one of the earliest advocates of attack surface, a metric that is likely strongly associated with the number of input-related vulnerabilities. It's probably hard to do perfectly, though, especially if any third-party APIs are involved. Are there any tools out there that try to measure attack surface? Has anybody had any experience in trying to apply it? - Steve ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___ ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___
Re: [SC-L] Microsoft Pushes Secure, Quality Code
Interesting that attack surface isn't included, given that Microsoft was one of the earliest advocates of attack surface, a metric that is likely strongly associated with the number of input-related vulnerabilities. It's probably hard to do perfectly, though, especially if any third-party APIs are involved. Are there any tools out there that try to measure attack surface? Has anybody had any experience in trying to apply it? - Steve ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___
Re: [SC-L] Microsoft Pushes Secure, Quality Code
Not surprising. Last time I looked, attack surface is subjective. McCabe is not. BTW, McCabe's Cyclomatic complexity boils down to 85% lines of code and 15% data flow if you do a principal component analysis on it. Just throw the code in the box and turn the crank. Then discard the results and you're done! gem company www.cigital.com podcast www.cigital.com/silverbullet blog www.cigital.clom/justiceleague book www.swsec.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steven M. Christey Sent: Monday, October 08, 2007 1:15 PM To: Secure Coding Subject: Re: [SC-L] Microsoft Pushes Secure, Quality Code Interesting that attack surface isn't included, given that Microsoft was one of the earliest advocates of attack surface, a metric that is likely strongly associated with the number of input-related vulnerabilities. It's probably hard to do perfectly, though, especially if any third-party APIs are involved. Are there any tools out there that try to measure attack surface? Has anybody had any experience in trying to apply it? - Steve ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___ ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___
Re: [SC-L] Microsoft Pushes Secure, Quality Code
On Mon, 8 Oct 2007, Gary McGraw wrote: Not surprising. Last time I looked, attack surface is subjective. McCabe is not. BTW, McCabe's Cyclomatic complexity boils down to 85% lines of code and 15% data flow if you do a principal component analysis on it. Hopefully the SEI people are monitoring this list and can provide their feedback. They've done some concrete work in making attack surface as objective as possible, enough to the point where they compared 2 FTP servers about a year ago. One of their papers comments that they wanted to use the code scanners to make the calculations for them, but for some reason they couldn't. I was under the impression from Mike Howard's comments over the years, that MS had some concrete (perhaps subjective) comparisons between different MS variants, and this was part of the argument for Vista's security over past MS operating systems. Just throw the code in the box and turn the crank. Then discard the results and you're done! While I understand the sentiment, it seems to me that you can't get very far without metrics of some sort. Perhaps more importantly, the real decision-makers need them because it's not their job (and probably not their expertise) to pore through endless details. - Steve ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___
Re: [SC-L] Microsoft Pushes Secure, Quality Code
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hey Steve, Are there any tools out there that try to measure attack surface? Has anybody had any experience in trying to apply it? SecurityInnovation's HoloDeck has an attack surface module, but unfortunately it is just a fancy wrapper for a Win32 strace() :) I am currently working on a research paper for my GCIH Gold that is about measuring code-coverage of an attack surface. For example being able to intelligently measure whether that magic packet you sent into a process _really_ covered 100 basic blocks, or were 20 of them a threading routine completely outside of the main surface (packet parsing) area. It is a tough thing to just measure, however some researchers at Carnegie Mellon have done some interesting work: http://www.cs.cmu.edu/~pratyus/qop.pdf And their mainpage is here: http://www.cs.cmu.edu/~pratyus/as.html Anyways I would be interested to hear some of the gurus respond back on this topic, as its a very relevant metric for QA and infosec professionals. JS -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHCoebKEj7ZJktQNsRArTgAKCWf96Tp8NXF7GuTiQN1BzyVlTEMwCfXpj9 ++VxKBeI8WcXP5tLWUYQdU4= =WoVW -END PGP SIGNATURE- ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. ___
