On Fri, 2010-04-09 at 10:35 +0200, Paul Goins wrote: > There may not be a clear answer to this problem, and the answer may or > may not be Twisted-specific. I'm just trying to fish for ideas here. > > The server I'm working on suffers from extremely high CPU time spent in > the logging functions. I'm spending between 30 and 40% of all CPU time > just in twisted.python.log.msg. > > Describing the environment: > > We use DailyLogFiles for file rotation along with the > PythonLoggingObserver to allow for logLevel support. (Of course, this > gives us significant overhead from Python's complex logging > architecture; maybe we should drop this...) > > On a 1000 request test of this server, we generate 58,274 calls to > t.p.log.msg. There's room for cleanup, but there's lots of pressure to > keep many of the log calls we have for forensic purposes, so we can only > cut this back so far. > > Anyway, does anyone have any ideas for speedups? Any stories from > experience especially? Shots in the dark are welcome. ;-) >
A request/log ratio of 1:60 is extreme. I would call that a log generator ;-) I think you could try to use a custom light weight log function that just queues the log messages (FIFO) and bursts them, say every second, to the log file (e.g. in a thread to use a multicore cpu). _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
