I have a heavily used threaded server program running on one of my Dell Poweredge 1750 
servers. Its a statistical analysis package for websites. Currently it analyses over 
60 million requests a day, which (because of many different reasons) causes it to 
handle around 120 million http requests a day. At peaks around 1500 requests a second.

The system stores most many statistics in memory which is flushed to disk in circles 
by a worker thread.

Another big part is stored in an on-disk database which is mmap()'d into memory. 
Because we do not have enough memory to keep everything in memory at one time the 
mmap() system of course pages data in and out.

When I look in systat -v I see that dirtybuf climbs to about 1700 and then they get 
flushed to disk, causing high disk usage of around 300-400 tps whcih renders the disks 
useless for anything else.

When those flushes occure, my apps state as displayed by top(1) gets into inode state, 
PRI is set to -14 and cpu usage rapidly drops. The program and ALL of its threads are 
stalled at that time. Those inode states take around 2 oe 3 seconds and happen every 
30 seconds or so.

In those 3 seconds we lose around 1500 hits at peak times for processing because the 
app can not handle them fast enough. This results in around 2 million or so hits lost 
over the day for processing.

I am now wondering if anyone can explain to me why ALL threads and not just the 
threads that actually do I/O work get blocked when dirty buffers are flushed and what 
to do to fix this problem.

I would be very happy if someone could reply and point me into the right direction!

Fabian Thylmann
[EMAIL PROTECTED] mailing list
To unsubscribe, send any mail to "[EMAIL PROTECTED]"

Reply via email to