On Tue, 9 Dec 2003, Fabian Thylmann wrote:

> I have a heavily used threaded server program running on one of my Dell
> Poweredge 1750 servers. Its a statistical analysis package for websites.
> Currently it analyses over 60 million requests a day, which (because of
> many different reasons) causes it to handle around 120 million http
> requests a day. At peaks around 1500 requests a second. 
> The system stores most many statistics in memory which is flushed to
> disk in circles by a worker thread. 
> Another big part is stored in an on-disk database which is mmap()'d into
> memory. Because we do not have enough memory to keep everything in
> memory at one time the mmap() system of course pages data in and out. 
> When I look in systat -v I see that dirtybuf climbs to about 1700 and
> then they get flushed to disk, causing high disk usage of around 300-400
> tps whcih renders the disks useless for anything else. 
> When those flushes occure, my apps state as displayed by top(1) gets
> into inode state, PRI is set to -14 and cpu usage rapidly drops. The
> program and ALL of its threads are stalled at that time. Those inode
> states take around 2 oe 3 seconds and happen every 30 seconds or so. 
> In those 3 seconds we lose around 1500 hits at peak times for processing
> because the app can not handle them fast enough. This results in around
> 2 million or so hits lost over the day for processing. 
> I am now wondering if anyone can explain to me why ALL threads and not
> just the threads that actually do I/O work get blocked when dirty
> buffers are flushed and what to do to fix this problem. 
> I would be very happy if someone could reply and point me into the right
> direction! 

You don't mention which version of FreeBSD you're running -- if 4.x, you
probably want to relink your application against the "linuxthreads" port.
This is because libc_r implements threads inside a single process without
the support of the kernel, which means that if the process is blocked in
kernel, all threads will be blocked in kernel.  The linuxthreads package
uses a model similar to Linux's threading implementation (hence the name)
to allow the threads to be scheduled using lightweight versions of
processes (shared file descriptors, etc).  This isn't quite
POSIX-compliant, but it works quite well for disk-bound applications such
as databases.

If you're running on 5.x, especially recent 5.1 or 5.2 prereleases, you
probably want to give libkse a try.  It's the new m:n threading
implementation that will become the default in 5.3, and also permits
parallelism (only in a more POSIX-compliant way, and in theory offering
much greater scalability for large numbers of threads).  I stick the
following lines in my /etc/libmap.conf on 5.x boxes to force all
applications linked against libc_r to use libkse instead:

  libc_r.so.5             libkse.so.1
  libc_r.so               libkse.so

One particularly nice thing about the m:n thread support is that you can
run-time plug the thread library between several options (libc_r, libthr,
libkse) to pick the one that performs best for your application.  Another
benefit of running with a non-libc_r threads package is that if you have
an SMP box, you'll see real parallelism. 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
[EMAIL PROTECTED]      Senior Research Scientist, McAfee Research

[EMAIL PROTECTED] mailing list
To unsubscribe, send any mail to "[EMAIL PROTECTED]"

Reply via email to