On 7/31/10 3:28 AM, Howard Chu wrote:
Revisiting an old thread...
http://www.openldap.org/lists/openldap-devel/200504/msg00066.html
I'm definitely seeing our current listener running out of steam on
servers with more than 12 or so cores, and some work in this direction
will definitely help. First it would be a good idea to classify all of
the tasks that the current listener manages, before deciding how to
divide them among multiple threads.
The listener is responsible for many events right now:
signal/shutdown processing
idle timeout checks
write timeout checks
runqueue scheduling
listener socket events
threadpool pauses
client socket read events
client socket write events
Splitting the client socket handling across multiple threads will
bring the greatest improvement in scalability. Just need to check our
thinking and make sure the remaining division of labor still makes sense.
There are two cases that annoy me in our current design - why don't we
just dedicate a thread to each listener socket, and let it block on
accept() ? That would eliminate a bit of churn in the current select()
workload.
Likewise, why don't we just let writer threads block in write(),
instead of having them ask the listener to listen for writability on
their socket? Or, if we're using non-blocking sockets, why don't we
let the writer threads block in their own select call, instead of
relying on the central thread to do the select and re-dispatch?
If your writer threads block until the socket is writeable, how many
threads will you need to have if many client never read the data ? You
might quickly not have any remaining writing threads available... The
select() is supposed to tell you when a socket is ready to accept more
write, then it's time to select a writing thread to push data into this
socket. You just need a pool of writing thread, and when a writing
thread has pushed data into the socket, then it becomes available for
the next write.
The first, obvious answer is this: when threads are blocked in system
calls like accept(), we can't simply wake them up again for shutdown
events or other situations. I believe the obvious fix here is to use
select() in each thread, waiting for both the target fd and the
wake_sds fd which is written to whenever a signal is caught. Off the
top of my head I'm not sure, when several threads are selecting on the
same fd, if they all receive a Readable event or if only one of them
will. Anyone know?
Why would you have more than one select() ? Wouldn't it be better to
have one thread processing the select() and dispatching the operation to
a pool of threads ?
All in all, what costs CPU consumption in a server is most certainly the
processing of incoming and outgoing requests, not the processing of the
select() operation, no ?
Im' maybe a bit tainted by Java, but it's really based on the same
mechanisms under the hood...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
