I have a large threaded mailing application running about 30 very active
threads. Each maintains an open socket with a client, a database
connection, etc. I also use a very large amount of shared memory, to the
point that most of the data structures used by the app are shared. Among
these structures are pieces of mailing lists 20,000 user records large
divided up into arrays of 200 each (100 shared arrays of 200 records for
each 20,000-record batch, of which there may be 20 loaded into memory at any
given time).
Once these structures are created, several of the threads periodically chew
CPU at an incredible rate. I'm running on dual hyperthreaded Xeon
processors (4 logical CPUs) with 4 gigabytes of memory. Performance should
not be an issue, but I've got 10+ threads each chewing an entire CPU.
An strace of one of the threads during one of these CPU surges shows an
enormous number of these:
futex(0x400c3aa0, FUTEX_WAKE, 1, {0, 0}) = 1
futex(0x400c3aa0, FUTEX_WAIT, 1, {1, 0}) = 1
futex(0x400c3aa0, FUTEX_WAKE, 1, {0, 0}) = 0
In other words, the strace shows the things I'd expect it to be doing,
reading and writing on the socket, reading the system time, logging, etc.,
but every few seconds there is a surge of these futexes and CPU usage
spikes.
My theory is that it's perl's internal shared memory synchronization that's
eating my CPU. Whether or not that's the case, has anyone else seen
anything like this? Is there anything I can do about it? Any help would be
greatly appreciated. I can provide more information if needed.
Thanks,
Jack
