I missed one detail in my previous answer:
Lance Hendrix wrote:
> ...
> My current thinking is that the explicit garbage collection (which is
> synchronous with the user request) is what is causing the (at least
> perceived) performance problem
The explicit garbage collection happens after the response is sent, so
the user visible delay shouldn't be impacted by this. The gc in itself
is usually reasonably fast (typically less than 0.1s, but of course this
varies with the system).
Now I wanted to verify that doing this gc.collect after each request
really helped, so I did a few experiments with tracd (--http11), on
Windows using ActivePython 2.5 and python.org Python 2.6:
0.11-stable unmodified (i.e. calling gc.collect after each request):
Python 2.5 (svn 1.6.2) 26-27M
Python 2.6 (no svn) 24-25M
0.11-stable modified (i.e. without calling gc.collect):
Python 2.5 (svn 1.6.2) 34-37M
Python 2.6 (no svn) 40-41M
(SQLite backend in both cases, Genshi advanced-i18n)
The numbers were obtained by repeatedly getting the Timeline page until
the memory growth stabilized (it's never a fixed number, but stays
within a range). Now that was under "normal" operations, one request
after the other.
This shows that using gc.collect helps to keep the memory usage at a low
level.
Then I stressed the server a bit, doing lots of quick refreshes in the
browser, which leads to connection interrupted errors:
...
self._sock.sendall(buffer)
error: (10053, 'Software caused connection abort')
This is expected, though it could be handled more gracefully. But what
was less expected was that the memory usage grew a lot more. I was able
to get up to 70M quite fast using tracd without explicit gc, but also to
60M when using the explicit gc.collect (this needed more efforts, e.g.
many tabs opened in different browsers, doing some "Reload All" before
the requests were actually finished).
Even worse, after the first experiment of that sort, when I tried to
resume to "normal" requests after letting the server recover from the
many interrupted requests, I kept getting back the dreaded
"TimeoutError: Unable to get database connection within 20 seconds" from
an otherwise inactive tracd... It looks like the PooledConnection
instances never got gc'ed, for some reason.
It's really hard to reproduce, though. It even took me quite some time
to get a simple "database is locked" error. To be continued...
The "good" news is that while doing all this heavy load tests, it was
really hard to go above 80M, so there seems to be a limit in the overall
memory usage, which is a good thing. Even after I managed to push it to
85M, it went back to 67M a couple minutes later.
On Windows, Python is quite good at giving memory back to the OS. It
would be interesting to redo the same kind of experiments on Linux (I
remember from the #6614 days that the gc.collect trick was especially
useful on that platform).
-- Christian
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac
Development" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/trac-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
