| Date: Mon, 5 Jun 2006 16:50:17 -0400
| From: McGovern, James F (HTSC, IT) [EMAIL PROTECTED]
| To: sc-l@securecoding.org
| Subject: [SC-L] Comparing Scanning Tools
|
| The industry analyst take on tools tends to be slightly different than
| software practitioners at times. Curious if anyone has looked at Fortify
and
| has formed any positive / negative / neutral opinions on this tool and
| others...
We evaluated a couple of static code scanning tools internally. The
following is an extract from an analysis I did. I've deliberately
omitted comparisons - you want to know about Fortify, not how it
compares to other products (which raises a whole bunch of other
issues), and included the text below. Standard disclaimers: This
is not EMC's position, it's my personal take.
Caveats: This analysis is based on a 3-hour vendor presentation. The
presenter may have made mistakes, and I certainly don't claim that my
recall of what he said is error-free. A later discussion with others
familiar with Fortify indicated that the experience we had is typical,
but is not necessarily the right way to evaluate the tool. Effective
use of Fortify requires building a set of rules appropriate to a
particular environment, method of working, constraints, etc., etc.
This takes significant time (6 months to a year) and effort, but
it was claimed that once you've put in the effort, Fortify is a
very good security scanner. I am not in a position to evaluate that
claim myself.
BTW, one thing not called out below is that Fortify can be quite slow.
Our experience in testing was that a Fortify scan took about twice as
long as a C++ compile/link cycle, unless you add data flow analysis -
in which case the time is much, much larger.
The brief summary: In my personal view, Fortify is a worthwhile tool,
but it would not be my first choice. (Given the opportunity to choose
two tools, it would probably be my second.) Others involved in the
evaluation reached the opposite conclusion, and rated Fortify first.
-- Jerry
Fortify
Fortify is aimed as a tool for use in a security audit. It is
deliberately biased in the direction of flagging all potential
security issues. It provides two kinds of analysis - what they call
semantic and data flow. Neither use of terminology is consistent
with industry practice. Their semantic analysis is better described
as a syntactic analysis: It looks at surface features of the
program (use of certain calls, for example). It mainly ignores
context. Fortify's own representative describe this analysis as a
super grep. This analysis is driven by a large database of rules,
which can be extended. (In industry practice, a semantic analysis
would look deeply at the meaning of the program.)
Data flow analysis is better called taint analysis. It traces all
data from external sources to find code that might incorrectly rely on
it.
When run on our code, semantic analysis reports about 3000 problems.
We looked closely at quite a number of them, and with a single
exception (where the code was so complex that no one could be sure),
they were false positives. For a security audit, that's probably OK.
The problem is: What does one do with the false positives? If this is
an isolated audit, the answer is - ignore them. But in practice code
is always changing, so you'll have to audit it again. How do you
avoid dealing with the same false positives every time?
A look at the problems showed that in many if not most of the cases we
looked at, there was no practical way to change the code to stop the
complaints. You can permanently suppress individual complaints, but
doing so appears to be very hazardous: The suppression is based on
the line on which the problem was found. It could well be that this
is a false positive because of the surrounding context - and that
context might change. For example, if a line has an array reference
a[i] that is safe because we can prove based on a test 10 lines
earlier that i is in range - something that Fortify itself cannot do,
since it does no value flow analysis - then we might suppress the
warning. But a later revision to the code could eliminate the test,
and we apparently would not receive a warning.
The data flow analysis gives more useful results. While the vendor
was here, we only had data flow analysis for a very small subset of
the code: Producing this is an extremely long process (days of CPU
time). Again, there were many false positives, though I would
consider almost all of them to be legitimate, in the sense that the
program could not possibly have known certain things about our
environment that would be needed to eliminate the warning. However,
in some cases, the messages were hard to understand: There was
insufficient information in the result to figure out just why the
program had reached the conclusion it had. Based on what the vendor
told us, effective use of the data flow analysis
