Hi, I'm running regulary my fuzzer (Fusil) on CPython since summer 2008: I tested Python 2.5, 2.6, 2.7, 3.0, 3.1 and 3.2. I'm only looking for "fatal errors": Python process killed by a signal, or sometimes fuzzer timeouts. I ignore most timeout results because most of them are valid function calls reading from/writing to a file or socket. My goal is to improve Python security: protect it against malicious data injection and denial of service. I prefer fuzzing to static code analyze because it finds few false positives and it directly generate a script reproducing the crash. Fuzzing is just one tool helping to improve the global security.
Bugs found in CPython by Fusil ============================== Modules ------- Fatal errors were only found in modules written in C. Modules: __builtin__ (5), json (4), io (3), bsddb (3), sqlite3 (3), audioop (2), locale (2), cProfile (2), Tkinter (2), dl, struct, binascii, testcapi, cPickle, multibytecodec, ctypes, hotshot, bz2, thread, bisect, weakref, imageop, multiprocessing. __builtin__: Exception, str, unicode, bytearray and long io: BytesIO, StringIO and FileIO It looks like json, bsddb and sqlite3 are young and not enough tested. audioop and imageop bugs are the most critical because they lead to writing to uninitialized memory (which might allow to execute arbitrary code). This module list gives also a first idea of which modules should be blacklisted in a sandbox ;-) Cause ----- The most common causes are insufficient input validation and invalid/missing error handling. "Insufficient input validation" means that the function is vulnerable to malicious data injection. "Invalid error handling" means that the function causes a new error while trying to cleanup data (eg. release memory of an uninitialized variable). "Missing error handling" means that a function result is an error but the caller doesn't check the function result. I don't have a generic solution to detect these problems. Except for "missing error handling": gcc has an extension to the C language to indidate that the result have to be used, __attribute__((warn_unused_result)). The GNU libc uses it the avoid common bugs. Consequence ----------- The most common consequence is to read from/write to uninitialized memory (especially reading from a NULL pointer) which lead sometimes to a segmentation fault (heisenbugs!). The second most common consequence is an unexpected exception during garbage collection: it displays a Fatal Python error and quits Python. I would suggest to log unexpected exception during garbage collection without stopping the whole Python process, as done for exceptions in a destructor. Details ------- Full list of all bugs found by Fusil with links to the bugtracker and to the commits: http://bitbucket.org/haypo/fusil/wiki/Python Interaction with the Python developers ====================================== I open an issue for each bug found in CPython. I describe how to reproduce it and try to write a patch. I have learn to always write an unit test, useful to reproduce the bug, and it makes Python commiters happy :-) The reaction depends on the impacted component, the severity of the bug, the complexity of the code reproducing the bug, and the quality of my bug report :-) The answer was always quick for core components. But some modules are maintained by everyone, which means nobody, like imageop, audioop or cProfile/hotshot. Having a module maitainer, like Guilherme Polo aka gpolo for Tkiner, does really help! It looks like fuzzing bugs are not always appreciated by developers, maybe because they are always "borderline" cases (not "realist"). Sometimes, even if I write a patch, an unit test, explain the problem and the solution, I don't get any comment. It doesn't motivate me to continue fuzzing :-/ Play with Fusil at home ======================= If you would like to fuzz Python with Fusil: download the last version of Fusil and run PYTHON fusil-python as root, where PYTHON is your python interpreter. Use --success=50 to wait for 50 crashs before stopping, --fast to speed up the fuzzing but slow down your computer, and --only-c to test only Python modules written in C. http://bitbucket.org/haypo/fusil/wiki/Home Fusil is running as the user fusil and group fusil to avoid removing arbitrary file or killing an arbitrary process, that's why you need to run it as root. If Fusil found a crash, you can analyze it while Fusil is running. Go into python/<crash directory>/ and read stdout and session.log files. Use "sudo ./replay.py --gdb" command to "replay" the crash in gdb (--valgrind option can also be useful). I'm only working on Linux, but Fusil works on any UNIX/BSD OS. Don't use Fusil on Windows! It might work on Windows but without any protection for your files and processes! I hope that my fuzzing tests helped Python project, and may be someone else would help me to continue these tests ;-) Victor Stinner aka haypo _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
