Gary, I would love a little refinement of the benefits to badnessometers.
Let's say I get a tool to tell me something I already suspect is wrong,
what percentage of the population are better than they expected?
I won't speak for Gary, but working a few doors down I have seen a few of the
same things he has.
Occasionally developers internally run free tools and surrepetitiously fix
problems that the tools find (this happens in some cultures where management is
particularly antagonistic towards security as a first class concern). In those
unusual instances, I could see the results of a badnessometer coming out better
than expected. Management would perceive that such things had never been run,
and would be pleasantly surprised to learn that the sky might not be falling.
Other than that, few people run a tool for the first time and see results
better than they expected. Tools codify all manner of stuff that your
developers almost certainly do not know how to check for (and if they do, they
probably don't have time).
Is it better to do such a badness test by doing a POC with one of the
tool vendors in this space or do I get additional lift by going with
a consulting firm in this regard?
I'm a consultant, take that as implied bias. But, I think you do get lift, and
here's my analogy. Consider yourself a handy guy around the house who is going
to do something moderately complicated, like redo a whole bathroom. You can buy
all the tools and read all the books on how to do it for a lot less money than
hiring a contractor to do the whole thing. There's some pretty specialized
tools in plumbing, though, and they're tools you probably haven't used more
than once or twice. Do you gain some extra insight into the use of those tools
by hiring someone experienced to assist on the complicated parts? I think so.
That someone experienced will come in, help you wield the unfamiliar tool, show
you some things that he has experienced, and get you through the difficult
parts. Then you, being the handy guy you are, are left to finish the bathroom,
doing things you know how to do well.
I think this analogy holds with a lot of the tools in security. You learn a lot
by getting the experience someone brings, assuming you get a good someone. We,
for example, have run a bunch of tools on a lot of different code bases. We
know which rules tend to be alarmist and which ones are really important if
they fire. Tool vendors won't give you that objectivity on their own tool, and
some of the sales engineers don't have the insight into their own tool to know
which warnings are just noise and which warnings are a big deal. A consultant
can help you have a bake-off between tools, whereas a tool vendor typically
lacks that objectivity.
Paco
This electronic message transmission contains information that may be
confidential or privileged. The information contained herein is intended
solely for the recipient and use by any other party is not authorized. If
you are not the intended recipient (or otherwise authorized to receive this
message by the intended recipient), any disclosure, copying, distribution or
use of the contents of the information is prohibited. If you have received
this electronic message transmission in error, please contact the sender by
reply email and delete all copies of this message. Cigital, Inc. accepts no
responsibility for any loss or damage resulting directly or indirectly from
the use of this email or its contents.
Thank You.
___
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
___
