Hi all. As many of you have noted, fuzz testing has become a very hot area
of research, with all sorts of new and impressive innovations. While I've
followed this, recent experiences with some of these new fuzz tools had
me wondering if the traditional simple fuzz testing that we created 30
years ago was still useful and applicable. So, last semester, I again
offered fuzz testing as one of the semester project options in my graduate
OS class (CS736).
This paper is the result of that study.
Title: The Relevance of Classic Fuzz Testing: Have We Solved This One?
Authors: Barton P. Miller, Mengxiao Zhang and Elisa R. Heymann
Abstract:
As fuzz testing has passed its 30th anniversary, and in the face of the
incredible progress in fuzz testing techniques and tools, the question
arises if the classic, basic fuzz technique is still useful and
applicable? In that tradition, we have updated the basic fuzz tools and
testing scripts and applied them to a large collection of Unix utilities
on Linux, FreeBSD, and MacOS. As before, our failure criteria was whether
the program crashed or hung. We found that 9 crash or hang out of 74
utilities on Linux, 15 out of 78 utilities on FreeBSD, and 12 out of 76
utilities on MacOS. A total of 24 different utilities failed across the
three platforms. We note that these failure rates are somewhat higher
than our in previous 1995, 2000, and 2006 studies of the reliability of
command line utilities.
In the basic fuzz tradition, we debugged each failed utility and
categorized the causes the failures. Classic categories of failures, such
as pointer and array errors and not checking return codes, were still
broadly present in the current results. In addition, we found a couple of
new categories of failures appearing. We present examples of these
failures to illustrate the programming practices that allowed them to
happen.
As a side note, we tested the limited number of utilities available in a
modern programming language (Rust) and found them to be of no better
reliability than the standard ones.
You can fetch the paper at:
ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz2020.pdf
(A version will be posted on arxiv.org soon.)
Comments and feedback are definitely welcome!
regards,
--bart
_______________________________________________
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
