On Mon, 2002-11-25 at 00:20, Manoj Kasichainula wrote: > On Sat, Nov 23, 2002 at 06:40:58PM -0800, Brian Pane wrote: > > Here's an outline of my latest thinking on how to build a > > multiple-connections-per-thread MPM for Apache 2.2. I'm > > eager to hear feedback from others who have been researching > > this topic. > > You prodded me into finally writing up a proposal that's been bouncing > around in my head for a while now. That was in a seperate message, this > will be suggestions for your proposal. > > > 1. Listener thread > > A Listener thread accept(2)s a connection, creates > > a conn_rec for it, and sends it to the Reader thread. > > Some (Most?) protocols have the server initiate the protocol > negotatiation instead of the client, so the listener needs to be able to > pass off to the writer thread as well. > > > * Limiting the Reader and Writer pools to one thread each will > > simplify the design and implementation. But will this impair > > our ability to take advantage of lots of CPUs? > > I was actually wondering why the reader and writer were seperate > threads.
It was a combination of several factors that convinced me to make them separate: * Take advantage of multiple CPUs more easily * Simplify the application logic * Reduce the number of file descriptors that each poll call is handling (important on platforms where we don't have an efficient poll mechanism) > What gets more complex with a thread pool > 1? I know we'd have to add a > mutex around the select+(read|write), but is there something else? If you split the pollset into 'n' sections and have 'n' threads each handling reads or writes on one section of it, it can be hard to balance the load. Some threads will end up with very active connections, while others will have mostly idle connections. The alternative is to have 'n' threads that take turns handling the entire pollset. That doesn't offer as much concurrency, so I'm not sure if it's worth the extra complexity. But it would be easy to test. > > * Can we eliminate the listener thread? It would be faster to just > > have the Reader thread include the listen socket(s) in its pollset. > > But if we did that, we'd need some new way to synchronize the > > accept handling among multiple child processes, because we can't > > have the Reader thread blocking on an accept mutex when it has > > existing connections to watch. > > You could dispense with the listener thread in the single-process case > and just use an intraprocess mutex around select+(accept|read|write) Right, with the accept/read/write all handled by the same thread (or thread pool), the handoff problem goes away. > > * Is there a more efficient way to interrupt a thread that's > > blocked in a poll call? That's a crucial step in the Listener-to- > > Reader and Request Processor-to-Writer handoffs. Writing a byte > > to a pipe requires two extra syscalls (a read and a write) per > > handoff. Sending a signal to the target thread is the only > > other solution I can think of at the moment, but that's bad > > because the target thread might be in the middle of a read > > or write call, rather than a poll, at the moment when we hit > > it with a signal, so the read or write will fail with EINTR. > > For Linux 2.6, file notifications could be done entirely in userland in > the case where no blocking is needed, using "futexes". Thanks! I'll check out futexes. > > But if you want to avoid the extra system calls, you could put a mutex > around maintenence of the pollset and just let the various threads dork > with it directly. > > I do keep mentioning this mutex around the select/poll :). Is there a > performance reason that you're trying to avoid it? In my past skimmings, > I've seen you post a lot of benchmarks and such, so maybe you've studied > this. The real reason I don't like the mutex around the poll is that it would add too much latency if we had to wait for the current poll to complete before adding a new descriptor. When the Listener accepts a new connection, or a Request Processor creates a new response brigade, it needs to get the corresponding socket added to the pollset immediately, which really requires interrupting the current poll. Brian
