Graham Dumpleton wrote: > Where are we at with confirming these memory leaks and committing some > changes to fix them?
Close. I spent quite a few hours reading through req_readline yesterday, trying to trace through all the possibilities. There are some... ah... interesting things in there that look a little dodgey. The suspicious bits are likely just corner case that may never happen, but could corrupt memory or the readline output. I'll be soliciting feedback from the list today. If we are going to mess with the code in readline (and obviously we are), I think we may as well do a quick optimization at the same time. Currently a new buffer is malloc'd for each readline(len > 0). There is a comment in the code wondering about reusing this buffer and I think I see a fairly easy way to do it. This could be a nice little optimization for people like Mike L. who are uploading large iso files. For a CD-ROM sized file you would save approx 10000 malloc / free calls by reusing the buffer - that seems worthwhile to me. > Only ask as I have a big patch related to MODPYTHON-63 that am wanting > to commit back into the repository, but don't want to be doing it if it > is going > to make your search harder by introducing some new memory leaks. :-) As long as you don't touch req_readline it shouldn't matter. I'm looking at the memory difference between a simple handler (baseline) and the specific test anyway, so if you introduce a leak elsewhere it will be accounted for. > BTW, what platform does you leak tester require to run? Top that supports the following options: -b > batch mode -u > user (assumes www-data) -n > iterations -d > delay -p > pid It also assumes that the %mem is in column 10 of the top output, but you can obviously tweak the code if your's is different. There is also the ability to dump the memory map with pmap, but this can generate a hugh amount of data and is slow. Also I'm not entirely sure what to do with output, so it's not on by default. If you don't have pmap it's not a problem. (This memory dump also requires "ps -u www-data", to get the pid to dump, but I would expect that output will be the same on any *nix-like system). The biggest limitation is the assumptions made about your Apache setup. I'm running this on a test machine using the system's apache config. There is only one apache instance running on the machine as user www-data, which I'm using to filter the output from top. I'm also starting and stopping apache after each test using: sudo /etc/init.d/apache2 restart There is also an assumption that the leak handlers are at http://localhost/mod_python/leaktests which on my system is a symlink in my document root to the tests. ln -s $HOME/mod_python/debug/leaktest/handlers /var/www/mod_python/leaktests This test harness started out as a quick and dirty hack. Ultimately it should have a configuration similar to our unit tests so it is independent of the main apache configuration. I wanted to tweak the test harness a little more before making it available, but maybe I'll commit it to my sandbox as is. It is usable enough, but just a little to specific to my test setup. > On Mac OS X I am > always seeing slow memory leaks over time even with most basic handler. > This often makes it hard to know if changes I make are introducing new > memory leaks or not. :-( I'm convinced I'm seeing a small leak in the most basic handler as well, but the leaks we've found and fixed so far have been serious enough to overwhelm it so we really haven't noticed. Jim
