On 8/2/10 5:26 AM, Howard Chu wrote:
<snip/>
Why would you have more than one select() ? Wouldn't it be better to
have one thread processing the select() and dispatching the operation to
a pool of threads ?
That's what we have right now, and as far as I can see it's a
bottleneck that prevents us from utilizing more than 12 cores. (I
could be wrong, and the bottleneck might actually be in the thread
pool manager. I haven't got precise enough measurements yet to know
for sure.)
Here's the situation: suppose you have thousands of clients connected
and active. Even if you have CPUs to spare, the number of connections
you can acknowledge and dispatch is limited by the speed of the single
thread that's processing select(). Even if all it does is walk thru
the list of active descriptors and dispatch a job to the thread pool
for each one, it's only possible to dispatch a fixed number of
ops/second, no matter how many other CPUs there are.
I'm a bit surprised that the select() processing *is* the bottleneck...
All in all, it's just -internally- a matter of processing a bit field to
see which bit is set to 1, and then get back the FD that is associated
with this bit. You must have some other tasks running that create this
bottleneck.
I will have to check OpenLDAP code here...
Right now on a 24 core server I'm seeing 48,000 searches/second and
50% CPU utilization. Adding more clients only seems to increase the
overall latency, but CPU usage and throughput don't increase any further.
Have you tried to do something we did on ADS : remove all the processing
to simply have a mock LDAP server, where only the network part is studied ?
ie, we just send back a mock response when a request has been received.
It helps to focus only on the network layer.
(sadly, as the PDU decoding is a costly operation, you may have to take
that into account).
Also in the test I'm looking at, some of the issues I pointed to above
aren't even happening. E.g., none of the operations are blocking on
write. It's all about handling read events, nothing else. (So again,
maybe it's OK to leave the current write behavior as-is.)
One idea Jean-François Arcand devised was to have two select() = one for
the reads, one for the writes. i'm not sure it's really interesting, but
that may worth to be investigated.
Another idea, and this is what we have implemented in ADS, is to have
more than one select() thread for reads/writes (in fact, we compute the
number of processors, and create as many threads as we have cores, plus
one. Each thread process a select() of course). I'm not convinced that,
in our case, it helps a lot, but it may be helpful in your case if you
don't want/can't modify the way requests are being processed in OpenLDAP.
As I've noted in the past, I can get double the throughput by running
two slapds concurrently, so it's not a question of network or I/O
resources. Just that a single listener thread (and/or a single mutex
controlling the thread pool) will only scale so far, and no further.
All in all, what costs CPU consumption in a server is most certainly the
processing of incoming and outgoing requests, not the processing of the
select() operation, no ?
In relative costs, sure, but eventually even the small costs add up.
Im' maybe a bit tainted by Java, but it's really based on the same
mechanisms under the hood...
Heh. Indeed. Concurrency issues are the same, no matter what language
you use to dress them up.
One other realization I had is that the current design makes it very
easy to build slapd as either threaded or non-threaded. Pushing too
much activity into other threads would break the non-threaded builds.
But at this point, with even cellphones going dual-core, I have to
wonder how important it is to maintain compatibility for non-threaded
slapd builds. ??
Java does not suffer from such limitation, that's for sure, thus we
don't have to care about primitive devices which don't support
threads... As Android - and supposely iOS - is supporting threads, it
should not be such a constraint.
Now, if you have an installed base of users who still depend on a single
threaded server, then may be it's time to split the server in two, and
have the single threaded server having its own branch ? Even if you fix
some bugs in trunk, it's very likely that you will be able to apply the
patch to the single threaded branch, and I'm not sure that it has a long
life expectancy anyway :)
Interesting problems :)
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
