Re: Events, Destruction and Locking
On Jul 7, 2009, at 10:17 AM, Graham Leggett wrote: Paul Querna wrote: It breaks the 1:1: connection mapping to thread (or process) model which is critical to low memory footprint, with thousands of connections, maybe I'm just insane, but all of the servers taking market share, like lighttpd, nginx, etc, all use this model. It also prevents all variations of the slowaris stupidity, because its damn hard to overwhelm the actual connection processing if its all async, and doesn't block a worker. But as you've pointed out, it makes our heads bleed, and locks slow us down. At the lowest level, the event loop should be completely async, and be capable of supporting an arbitrary (probably very high) number of concurrent connections. We are looking at a kernel-like scheduler more than anything else...
Re: Events, Destruction and Locking
2009/7/8 Graham Leggett minf...@sharp.fm: Paul Querna wrote: It breaks the 1:1: connection mapping to thread (or process) model which is critical to low memory footprint, with thousands of connections, maybe I'm just insane, but all of the servers taking market share, like lighttpd, nginx, etc, all use this model. It also prevents all variations of the slowaris stupidity, because its damn hard to overwhelm the actual connection processing if its all async, and doesn't block a worker. But as you've pointed out, it makes our heads bleed, and locks slow us down. At the lowest level, the event loop should be completely async, and be capable of supporting an arbitrary (probably very high) number of concurrent connections. If one connection slows or stops (deliberately or otherwise), it won't block any other connections on the same event loop, which will continue as normal. But which for a multiprocess web server screws up if you then have a blocking type model for an application running on top. Specifically, the greedy nature of accepting connections may mean a process accepts more connections which it has high level threads to handle. If the high level threads end up blocking, then any accepted connections for the blocking high level application, for which request headers are still being read, or are pending, will be blocked as well even though another server process may be idle. In the current Apache model a process will only accept connections if it knows it is able to process it at that time. If a process doesn't have the threads available, then a different process would pick it up instead. I have previously commented how this causes problems with nginx for potentially blocking applications running in nginx worker processes. See: http://blog.dscpl.com.au/2009/05/blocking-requests-and-nginx-version-of.html To prevent this you are forced to run event driven system for everything and blocking type applications can't be run in same process. Thus, anything like that has to be shoved out into a separate process. FASTCGI was mentioned for that, but frankly I believed FASTCGI is getting a bit crufty these days. It perhaps really needs to be modernised, with the byte protocol layout simplified to get rid of these varying size length indicator bytes. This may have been warranted when networks were slower and amount of body data being passed around less, but I can't see that that extra complexity is warranted any more. FASTCGI also can't handle things like end to end 100-continue processing and perhaps has other problems as well in respect of handling logging outside of request context etc etc. So, I personally would really love to see a good review of FASTCGI, AJP and any other similar/pertinent protocols done to distill what in these modern times is required and would be a better mechanism. The implementations of FASTCGI could also perhaps be modernised. Of course, even though FASTCGI may not be the most elegant of systems, probably too entrenched to get rid of it. The only way perhaps might be if a improved version formed the basis of any internal communications for a completely restructured internal model for Apache 3.0 based on serf which had segregation between processes handling static files and applications, with user separation etc etc. Graham
Re: Events, Destruction and Locking
On Wed, Jul 8, 2009 at 3:05 AM, Graham Dumpletongraham.dumple...@gmail.com wrote: 2009/7/8 Graham Leggett minf...@sharp.fm: Paul Querna wrote: It breaks the 1:1: connection mapping to thread (or process) model which is critical to low memory footprint, with thousands of connections, maybe I'm just insane, but all of the servers taking market share, like lighttpd, nginx, etc, all use this model. It also prevents all variations of the slowaris stupidity, because its damn hard to overwhelm the actual connection processing if its all async, and doesn't block a worker. But as you've pointed out, it makes our heads bleed, and locks slow us down. At the lowest level, the event loop should be completely async, and be capable of supporting an arbitrary (probably very high) number of concurrent connections. If one connection slows or stops (deliberately or otherwise), it won't block any other connections on the same event loop, which will continue as normal. But which for a multiprocess web server screws up if you then have a blocking type model for an application running on top. Specifically, the greedy nature of accepting connections may mean a process accepts more connections which it has high level threads to handle. If the high level threads end up blocking, then any accepted connections for the blocking high level application, for which request headers are still being read, or are pending, will be blocked as well even though another server process may be idle. In the current Apache model a process will only accept connections if it knows it is able to process it at that time. If a process doesn't have the threads available, then a different process would pick it up instead. I have previously commented how this causes problems with nginx for potentially blocking applications running in nginx worker processes. See: http://blog.dscpl.com.au/2009/05/blocking-requests-and-nginx-version-of.html To prevent this you are forced to run event driven system for everything and blocking type applications can't be run in same process. Thus, anything like that has to be shoved out into a separate process. FASTCGI was mentioned for that, but frankly I believed FASTCGI is getting a bit crufty these days. It perhaps really needs to be modernised, with the byte protocol layout simplified to get rid of these varying size length indicator bytes. This may have been warranted when networks were slower and amount of body data being passed around less, but I can't see that that extra complexity is warranted any more. FASTCGI also can't handle things like end to end 100-continue processing and perhaps has other problems as well in respect of handling logging outside of request context etc etc. So, I personally would really love to see a good review of FASTCGI, AJP and any other similar/pertinent protocols done to distill what in these modern times is required and would be a better mechanism. The implementations of FASTCGI could also perhaps be modernised. Of course, even though FASTCGI may not be the most elegant of systems, probably too entrenched to get rid of it. The only way perhaps might be if a improved version formed the basis of any internal communications for a completely restructured internal model for Apache 3.0 based on serf which had segregation between processes handling static files and applications, with user separation etc etc. TBH, I think the best way to modernize FastCGI or AJP is to just proxy HTTP over a daemon socket, then you solve all the protocol issues... and just treat it like another reverse proxy. The part we really need to write is the backend process manager, to spawn/kill more of these workers.
Re: Events, Destruction and Locking
On 08.07.2009 15:55, Paul Querna wrote: On Wed, Jul 8, 2009 at 3:05 AM, Graham Dumpletongraham.dumple...@gmail.com wrote: 2009/7/8 Graham Leggett minf...@sharp.fm: Paul Querna wrote: It breaks the 1:1: connection mapping to thread (or process) model which is critical to low memory footprint, with thousands of connections, maybe I'm just insane, but all of the servers taking market share, like lighttpd, nginx, etc, all use this model. It also prevents all variations of the slowaris stupidity, because its damn hard to overwhelm the actual connection processing if its all async, and doesn't block a worker. But as you've pointed out, it makes our heads bleed, and locks slow us down. At the lowest level, the event loop should be completely async, and be capable of supporting an arbitrary (probably very high) number of concurrent connections. If one connection slows or stops (deliberately or otherwise), it won't block any other connections on the same event loop, which will continue as normal. But which for a multiprocess web server screws up if you then have a blocking type model for an application running on top. Specifically, the greedy nature of accepting connections may mean a process accepts more connections which it has high level threads to handle. If the high level threads end up blocking, then any accepted connections for the blocking high level application, for which request headers are still being read, or are pending, will be blocked as well even though another server process may be idle. In the current Apache model a process will only accept connections if it knows it is able to process it at that time. If a process doesn't have the threads available, then a different process would pick it up instead. I have previously commented how this causes problems with nginx for potentially blocking applications running in nginx worker processes. See: http://blog.dscpl.com.au/2009/05/blocking-requests-and-nginx-version-of.html To prevent this you are forced to run event driven system for everything and blocking type applications can't be run in same process. Thus, anything like that has to be shoved out into a separate process. FASTCGI was mentioned for that, but frankly I believed FASTCGI is getting a bit crufty these days. It perhaps really needs to be modernised, with the byte protocol layout simplified to get rid of these varying size length indicator bytes. This may have been warranted when networks were slower and amount of body data being passed around less, but I can't see that that extra complexity is warranted any more. FASTCGI also can't handle things like end to end 100-continue processing and perhaps has other problems as well in respect of handling logging outside of request context etc etc. So, I personally would really love to see a good review of FASTCGI, AJP and any other similar/pertinent protocols done to distill what in these modern times is required and would be a better mechanism. The implementations of FASTCGI could also perhaps be modernised. Of course, even though FASTCGI may not be the most elegant of systems, probably too entrenched to get rid of it. The only way perhaps might be if a improved version formed the basis of any internal communications for a completely restructured internal model for Apache 3.0 based on serf which had segregation between processes handling static files and applications, with user separation etc etc. TBH, I think the best way to modernize FastCGI or AJP is to just proxy HTTP over a daemon socket, then you solve all the protocol issues... and just treat it like another reverse proxy. The part we really need to write is the backend process manager, to spawn/kill more of these workers. Though there is one nice feature in the AJP protocol: since it knows it's serving via a reverse proxy, the back end patches some communication data like it were the front end. So if the context on the back end asks for port, protocol, host name etc. it automatically gets the data that looks like the one of the front end. That way cookies, self-referencing links etc. work right. Most of that can be simulated by appropriate configuration with HTTP to (yes, there are a lot of proxy options for this), but in AJP its automatic. Some parts are not configurable right now, like e.g. the client IP. You always have to introduce something that's aware e.g. of the X-Forwarded-For header. Another example would be whether the communication to the reverse proxy was via https. You can transport all that info va custom headers, but the backend usually doesn't know how to handle it. Regards, Rainer
Re: Events, Destruction and Locking
2009/7/9 Rainer Jung rainer.j...@kippdata.de: On 08.07.2009 15:55, Paul Querna wrote: On Wed, Jul 8, 2009 at 3:05 AM, Graham Dumpletongraham.dumple...@gmail.com wrote: 2009/7/8 Graham Leggett minf...@sharp.fm: Paul Querna wrote: It breaks the 1:1: connection mapping to thread (or process) model which is critical to low memory footprint, with thousands of connections, maybe I'm just insane, but all of the servers taking market share, like lighttpd, nginx, etc, all use this model. It also prevents all variations of the slowaris stupidity, because its damn hard to overwhelm the actual connection processing if its all async, and doesn't block a worker. But as you've pointed out, it makes our heads bleed, and locks slow us down. At the lowest level, the event loop should be completely async, and be capable of supporting an arbitrary (probably very high) number of concurrent connections. If one connection slows or stops (deliberately or otherwise), it won't block any other connections on the same event loop, which will continue as normal. But which for a multiprocess web server screws up if you then have a blocking type model for an application running on top. Specifically, the greedy nature of accepting connections may mean a process accepts more connections which it has high level threads to handle. If the high level threads end up blocking, then any accepted connections for the blocking high level application, for which request headers are still being read, or are pending, will be blocked as well even though another server process may be idle. In the current Apache model a process will only accept connections if it knows it is able to process it at that time. If a process doesn't have the threads available, then a different process would pick it up instead. I have previously commented how this causes problems with nginx for potentially blocking applications running in nginx worker processes. See: http://blog.dscpl.com.au/2009/05/blocking-requests-and-nginx-version-of.html To prevent this you are forced to run event driven system for everything and blocking type applications can't be run in same process. Thus, anything like that has to be shoved out into a separate process. FASTCGI was mentioned for that, but frankly I believed FASTCGI is getting a bit crufty these days. It perhaps really needs to be modernised, with the byte protocol layout simplified to get rid of these varying size length indicator bytes. This may have been warranted when networks were slower and amount of body data being passed around less, but I can't see that that extra complexity is warranted any more. FASTCGI also can't handle things like end to end 100-continue processing and perhaps has other problems as well in respect of handling logging outside of request context etc etc. So, I personally would really love to see a good review of FASTCGI, AJP and any other similar/pertinent protocols done to distill what in these modern times is required and would be a better mechanism. The implementations of FASTCGI could also perhaps be modernised. Of course, even though FASTCGI may not be the most elegant of systems, probably too entrenched to get rid of it. The only way perhaps might be if a improved version formed the basis of any internal communications for a completely restructured internal model for Apache 3.0 based on serf which had segregation between processes handling static files and applications, with user separation etc etc. TBH, I think the best way to modernize FastCGI or AJP is to just proxy HTTP over a daemon socket, then you solve all the protocol issues... and just treat it like another reverse proxy. The part we really need to write is the backend process manager, to spawn/kill more of these workers. Though there is one nice feature in the AJP protocol: since it knows it's serving via a reverse proxy, the back end patches some communication data like it were the front end. So if the context on the back end asks for port, protocol, host name etc. it automatically gets the data that looks like the one of the front end. That way cookies, self-referencing links etc. work right. Most of that can be simulated by appropriate configuration with HTTP to (yes, there are a lot of proxy options for this), but in AJP its automatic. Some parts are not configurable right now, like e.g. the client IP. You always have to introduce something that's aware e.g. of the X-Forwarded-For header. Another example would be whether the communication to the reverse proxy was via https. You can transport all that info va custom headers, but the backend usually doesn't know how to handle it. Yes, these are the sort of things which would be nice to be transparent. Paul's comment is valid though in that HTTP itself could be used as the protocol. Right now you couldn't do that over a UNIX socket though for local backend process, and you loose the ability to feed back
Re: Events, Destruction and Locking
On Wed, 2009-07-08 at 11:01 +1000, Bojan Smojver wrote: So, the loop would be: - poll() - try assembling a full request from data read so far - process if successful - go back to poll() if not Too naive? I see that we'd most likely get stuck with the accept mutex (i.e. if another process had it, we would not be poll()-ing already accepted fds any more). We could work around this by using apr_proc_mutex_trylock() if there are any already accepted fds. If this fails, we just poll() already accepted fds (i.e. someone is already poll()-ing to accept()). Otherwise, we poll() the lot. -- Bojan
Re: Events, Destruction and Locking
On Wed, Jul 8, 2009 at 9:11 PM, Bojan Smojverbo...@rexursive.com wrote: On Wed, 2009-07-08 at 11:01 +1000, Bojan Smojver wrote: So, the loop would be: - poll() - try assembling a full request from data read so far - process if successful - go back to poll() if not Too naive? I see that we'd most likely get stuck with the accept mutex (i.e. if another process had it, we would not be poll()-ing already accepted fds any more). But the event mpm doesn't have an accept mutex :D We could work around this by using apr_proc_mutex_trylock() if there are any already accepted fds. If this fails, we just poll() already accepted fds (i.e. someone is already poll()-ing to accept()). Otherwise, we poll() the lot. -- Bojan
Re: Events, Destruction and Locking
On Wed, 2009-07-08 at 22:53 -0400, Paul Querna wrote: But the event mpm doesn't have an accept mutex :D Yeah, I know. I was talking about making prefork async too. -- Bojan
Re: Events, Destruction and Locking
On Mon, Jul 6, 2009 at 10:50 PM, Justin Erenkrantzjus...@erenkrantz.com wrote: On Mon, Jul 6, 2009 at 10:20 PM, Paul Quernap...@querna.org wrote: I am looking for an alternative that doesn't expose all this crazyness of when to free, destruct, or lock things. The best idea I can come up with is for each Connection, it would become 'semi-sticky' to a single thread. Meaning each worker thread would have its own queue of upcoming events to process, and all events for connection X would sit on the same 'queue'. This would prevent two threads waiting for destruction, and other cases of a single connection's mutex locking up all your works, essentially providing basic fault isolation. These queues could be mutable, and you could 'move' a connection between queues, but you would always take all of its events and triggers, and move them together to a different queue. Does the 'connection event queue' idea make sense? I think I see where you're going with this...being so dependent upon mutexes seems...like going into a jungle full of guerillas armed with only a dull kitchen knife. So, a connection gets assigned to a 'thread' - but it has only two states: running or waiting for a network event. The critical part is that the thread *never* blocks on network traffic...all the 'network event' thread does is detect yup, ready to go and throws it back to that 'assigned' thread to process the event. Seems trivial enough to do with a serf-centric system. =) Yes, I think the connection having the two states: running or waiting for a network event is the key to making this work. The thread-stickiness is really just the conceptional model, but basically if a connection is already 'running', all other events that would of fired for it, like a timeout, would just queue up behind the running operation, rather than running directly on another thread. This start solving a multitude of locking and cleanup issues (i think).
Re: Events, Destruction and Locking
On Mon, Jul 6, 2009 at 10:56 PM, Mladen Turkmt...@apache.org wrote: Paul Querna wrote: Can't sleep, so finally writing this email I've been meaning to write for about 7 months now :D Pools don't help, but don't really make it worse, and are good enough for the actual cleanup part -- the difficultly lies in knowing *when* you can cleanup an object. Pool pre cleanup is meant to deal with such issues. You register a pre-cleanup and it will run before any of the pool objects are actually destroyed. In your case pre-cleanup callback could break the wait loop and make sure you don't reference a zombie object. The only issue left is guarding thread access to a singleton pollset interrupt (we even have pollset_interrupt with latest APR) from a pre-cleanup callback (or simply using a queue to serialize the objects that needs to get removed from the pollset) This deals with removing an event from the pollset, but what about an event that had already fired, as I gave in the original example of a timeout event firing the same time a socket close event happened? In that state you have two threads both in a 'run state' for a connection, and I'm not sure how the pre-cleanup to pools solves this in any way? Thanks, Paul
Re: Events, Destruction and Locking
Paul Querna wrote: This deals with removing an event from the pollset, but what about an event that had already fired, as I gave in the original example of a timeout event firing the same time a socket close event happened? In that case I suppose the only solution is to make the operations atomic. Since both operations would lead to the same result (closing a connection) I suppose an atomic state flag should be enough. In that state you have two threads both in a 'run state' for a connection, and I'm not sure how the pre-cleanup to pools solves this in any way? It won't because the cleanup pool API doesn't bother with cleanup callback return values, so there's no way to bail out from the pool cleanup call. I suppose we could modify the pre-cleanup to handle the retval from callback and breaks the entire pool cleanup if one of them returns something other then APR_SUCCESS. Then the callback function can decide weather there is a pending close operation or not. Regards -- ^(TM)
Re: Events, Destruction and Locking
Paul Querna wrote: Can't sleep, so finally writing this email I've been meaning to write for about 7 months now :D One of the challenges in the Simple MPM, and to a smaller degree in the Event MPM, is how to manage memory allocation, destruction, and thread safety. A 'simple' example: - 1) Thread A: Client Connection Created - 2) Thread A: Timer Event Added for 10 seconds in the future to detect IO timeout, - 3) Thread B: Client Socket closes in 9.99 seconds. - 4) Thread C: Timer Event for IO timeout is triggered after 10 seconds The simple answer is placing a Mutex around the connection object. Any operation which two threads are working on the connection, locks this Mutex. As you've said, locks create many problems, the most fatal of which is that locks potentially block the event loop. Ideally if a try_lock fails, the event should reschedule itself to try again at some point in the near future, but that relies on people bothering to write this logic, and I suspect many won't. A pragmatic approach might be to handle a request completely within a single event loop running in a single thread. In this case the timer event for IO timeout is in the same thread as the socket close event. At this point you just need to make sure that your pool cleanups are handled correctly. So if a timeout runs, all the timeout does is apr_destroy_pool(r-pool), and that's it. It is up to the pool cleanup to make sure that all events (such as the event that calls connection close) are cleanly deregistered so that they won't get called in future. We may offer a mechanism (such as a watchdog) that allows a request to kick off code in another thread, but a prerequisite for that is that the pool cleanup will have to be created to make sure that other thread is terminated cleanly, or the request is cleanly removed from that other thread's event loop. I think it is possible to write a complete server that deals with all these intricacies and gets everything just 'right', but as soon as you introduce 3rd party module writers, no matter how 'smart' we are, our castle of event goodness will crumble. You've hit the nail on the head as to why the prefork and worker models are still relevant - they are very forgiving of irresponsible behaviour by modules. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Events, Destruction and Locking
On Tue, Jul 7, 2009 at 7:34 AM, Graham Leggettminf...@sharp.fm wrote: Paul Querna wrote: I think it is possible to write a complete server that deals with all these intricacies and gets everything just 'right', but as soon as you introduce 3rd party module writers, no matter how 'smart' we are, our castle of event goodness will crumble. You've hit the nail on the head as to why the prefork and worker models are still relevant - they are very forgiving of irresponsible behaviour by modules. Nah, 90% of what is done in moduels today should be out of process aka in FastCGI or another method, but out of process. (regardless of MPM)
Re: Events, Destruction and Locking
Paul Querna wrote: Nah, 90% of what is done in moduels today should be out of process aka in FastCGI or another method, but out of process. (regardless of MPM) You're just moving the problem from one server to another, the problem remains unsolved. Whether the code runs within httpd space, or fastcgi space, the code still needs to run, and if it's written badly, the code will still leak/crash, and you still have to cater for it. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Events, Destruction and Locking
On Tue, Jul 7, 2009 at 8:39 AM, Graham Leggettminf...@sharp.fm wrote: Paul Querna wrote: Nah, 90% of what is done in moduels today should be out of process aka in FastCGI or another method, but out of process. (regardless of MPM) You're just moving the problem from one server to another, the problem remains unsolved. Whether the code runs within httpd space, or fastcgi space, the code still needs to run, and if it's written badly, the code will still leak/crash, and you still have to cater for it. Yes, but in a separate process it has fault isolation.. and we can restart it when it fails, neither of which are true for modules using the in-process API directly -- look at the reliability of QMail, or the newer architecture of Google's Chrome, they are both great examples of fault isolation.
Re: Events, Destruction and Locking
On Tue, Jul 7, 2009 at 10:01 AM, Graham Leggettminf...@sharp.fm wrote: Paul Querna wrote: Yes, but in a separate process it has fault isolation.. and we can restart it when it fails, neither of which are true for modules using the in-process API directly -- look at the reliability of QMail, or the newer architecture of Google's Chrome, they are both great examples of fault isolation. As is httpd prefork :) I think the key target for the event model is for low-complexity scenarios like shipping raw files, or being a cache, or a reverse proxy. If we have three separate levels, a process, containing threads, containing an event loop, we could allow the behaviour of prefork (many processes, one thread, one-request-per-event-loop-at-a-time), or the bahaviour of worker (one or many processes, many threads, one-request-per-event-loop-at-a-time), or an event model (one or many processes, one or many threads, many-requests-per-event-loop-at-one-time) at the same time. I am not sure that splitting request handling across threads (in your example, connection close handled by event on thread A, timeout handled by event on thread B) buys us anything (apart from the complexity you described). It breaks the 1:1: connection mapping to thread (or process) model which is critical to low memory footprint, with thousands of connections, maybe I'm just insane, but all of the servers taking market share, like lighttpd, nginx, etc, all use this model. It also prevents all variations of the slowaris stupidity, because its damn hard to overwhelm the actual connection processing if its all async, and doesn't block a worker.
Re: Events, Destruction and Locking
Paul Querna wrote: Yes, but in a separate process it has fault isolation.. and we can restart it when it fails, neither of which are true for modules using the in-process API directly -- look at the reliability of QMail, or the newer architecture of Google's Chrome, they are both great examples of fault isolation. As is httpd prefork :) I think the key target for the event model is for low-complexity scenarios like shipping raw files, or being a cache, or a reverse proxy. If we have three separate levels, a process, containing threads, containing an event loop, we could allow the behaviour of prefork (many processes, one thread, one-request-per-event-loop-at-a-time), or the bahaviour of worker (one or many processes, many threads, one-request-per-event-loop-at-a-time), or an event model (one or many processes, one or many threads, many-requests-per-event-loop-at-one-time) at the same time. I am not sure that splitting request handling across threads (in your example, connection close handled by event on thread A, timeout handled by event on thread B) buys us anything (apart from the complexity you described). Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Events, Destruction and Locking
Paul Querna wrote: It breaks the 1:1: connection mapping to thread (or process) model which is critical to low memory footprint, with thousands of connections, maybe I'm just insane, but all of the servers taking market share, like lighttpd, nginx, etc, all use this model. It also prevents all variations of the slowaris stupidity, because its damn hard to overwhelm the actual connection processing if its all async, and doesn't block a worker. But as you've pointed out, it makes our heads bleed, and locks slow us down. At the lowest level, the event loop should be completely async, and be capable of supporting an arbitrary (probably very high) number of concurrent connections. If one connection slows or stops (deliberately or otherwise), it won't block any other connections on the same event loop, which will continue as normal. The only requirement is that each request accurately registers event deregistration functions in their cleanups, so that the request is cleanly deregistered and future events canceled on apr_pool_destroy(). The event loop can also choose to proactively kill too-slow connections if certain memory or concurrent connection threshholds are reached. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Events, Destruction and Locking
On Tue, Jul 7, 2009 at 9:39 AM, Paul Querna p...@querna.org wrote: On Tue, Jul 7, 2009 at 8:39 AM, Graham Leggettminf...@sharp.fm wrote: Paul Querna wrote: Nah, 90% of what is done in moduels today should be out of process aka in FastCGI or another method, but out of process. (regardless of MPM) You're just moving the problem from one server to another, the problem remains unsolved. Whether the code runs within httpd space, or fastcgi space, the code still needs to run, and if it's written badly, the code will still leak/crash, and you still have to cater for it. Yes, but in a separate process it has fault isolation.. and we can restart it when it fails, neither of which are true for modules using the in-process API directly -- look at the reliability of QMail, or the newer architecture of Google's Chrome, they are both great examples of fault isolation. Also, it simplifies the programming problem by reducing the number of separate memory and concurrency models that must be accommodated by the application-level code.
Re: Events, Destruction and Locking
This is how I envisioned the async stuff working. -Async event thread is used only for input/output of httpd to/from network* -After we read the headers, we pass the request/connection to the worker threads. Each request is sticky to a thread. Request stuff may block, etc, so this thread pool size is configurable and in mod_status, etc. -any writes out of the request to the clientare passed into the async thread. This may be wrapped in filters, whatever. *We may allow there to be multiple ones of these, ie one for proxies, or have a very well defined way to add watches to this. This is a very simplistic view. I was basically thinking that all conn_rec stuff is handled in the async event thread, all the request_rec stuff is handled in the worker threads. -- Brian Akins
Re: Events, Destruction and Locking
On Tue, Jul 7, 2009 at 12:54 PM, Akins, Brianbrian.ak...@turner.com wrote: This is how I envisioned the async stuff working. -Async event thread is used only for input/output of httpd to/from network* -After we read the headers, we pass the request/connection to the worker threads. Each request is sticky to a thread. Request stuff may block, etc, so this thread pool size is configurable and in mod_status, etc. -any writes out of the request to the clientare passed into the async thread. This may be wrapped in filters, whatever. *We may allow there to be multiple ones of these, ie one for proxies, or have a very well defined way to add watches to this. This is a very simplistic view. I was basically thinking that all conn_rec stuff is handled in the async event thread, all the request_rec stuff is handled in the worker threads. Right, but I think the 'waiting for X' while processing is a very important case, it can get you to a fully async reverse proxy, which opens up lots of possibilities.
Re: Events, Destruction and Locking
Akins, Brian wrote: This is how I envisioned the async stuff working. -Async event thread is used only for input/output of httpd to/from network* -After we read the headers, we pass the request/connection to the worker threads. Each request is sticky to a thread. Request stuff may block, etc, so this thread pool size is configurable and in mod_status, etc. -any writes out of the request to the clientare passed into the async thread. This may be wrapped in filters, whatever. *We may allow there to be multiple ones of these, ie one for proxies, or have a very well defined way to add watches to this. This is a very simplistic view. I was basically thinking that all conn_rec stuff is handled in the async event thread, all the request_rec stuff is handled in the worker threads. The trouble with this is that all you need to do to wedge one of the worker threads is to promise to send two bytes as a request body, and then send one (or zero), then hang. Ideally any async implementation should be 100% async end to end. I don't believe that its necessary though for a single request to be handled by more than one thread. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Events, Destruction and Locking
On 7/7/09 1:02 PM, Graham Leggett minf...@sharp.fm wrote: Ideally any async implementation should be 100% async end to end. I don't believe that its necessary though for a single request to be handled by more than one thread. True. However, what about things that may be process intensive. Ie, running lua in process. And we'd want to run multiple async threads (or processes). One of the issues with lighttpd with multiple processes (to use multiple cores, etc) is that lots of stuff is broken (ie, log files interleave). We just need to be aware of the issues that other servers have uncovered in this area. -- Brian Akins
Re: Events, Destruction and Locking
On 07/07/2009 07:02 PM, Graham Leggett wrote: Ideally any async implementation should be 100% async end to end. I don't believe that its necessary though for a single request to be handled by more than one thread. I agree. I see no reason for multiple threads working on the same request at the same time (at least handler wise). On the other side it may be interesting to develop async handlers that wait for external events like a post body or a database response and that might want to free the thread until this event happens. The same may be interesting for filters. So it should be possible for a request to move over to a different thread, but not more than one thread should be working on the same request at the same time. Regards Rüdiger
Re: Events, Destruction and Locking
On Tue, 2009-07-07 at 16:01 +0200, Graham Leggett wrote: As is httpd prefork :) Yeah, definitely my favourite MPM :-) As far as I understand this, the deal is that we need to have a complete request before we start processing it. Otherwise, we can get stuck and one of our precious resources is tied up for a long time. Is there anything stopping us from having not just fds in listen in that apr_pollset_poll() of prefork.c, but also a bunch of already accepted fds that are waiting for more data to come in? I'm guessing we'd have to use ap_process_http_async_connection() and have multiple ptrans pools, but that should not be all that hard to do. So, the loop would be: - poll() - try assembling a full request from data read so far - process if successful - go back to poll() if not Too naive? -- Bojan
Events, Destruction and Locking
Can't sleep, so finally writing this email I've been meaning to write for about 7 months now :D One of the challenges in the Simple MPM, and to a smaller degree in the Event MPM, is how to manage memory allocation, destruction, and thread safety. A 'simple' example: - 1) Thread A: Client Connection Created - 2) Thread A: Timer Event Added for 10 seconds in the future to detect IO timeout, - 3) Thread B: Client Socket closes in 9.99 seconds. - 4) Thread C: Timer Event for IO timeout is triggered after 10 seconds The simple answer is placing a Mutex around the connection object. Any operation which two threads are working on the connection, locks this Mutex. This has many problems, the first of which is destruction. In this case, Thread B would start destructing the connection, since the socket was closed, but thread C would already be waiting for this mutex and then the object underneath it was just free'ed. To solve this Thread B would unregister all existing (and unfired) triggers/timeouts first. Events would increment a reference count on the connection object, and Thread B would schedule a future event to check this reference count. If the reference count is zero, this timer would free the connection object, if there was still an outstanding reference in a running event, it would schedule itself for a future cleanup attempt. All of this is insanely error prone, difficult to debug, and painful to explain. Pools don't help, but don't really make it worse, and are good enough for the actual cleanup part -- the difficultly lies in knowing *when* you can cleanup an object. A related problem of using Mutex Guards on a connection object is that if a single connection 'locks up' a thread, its feasible forl other worker threads to get stuck waiting for this connection, and we would have no way to 'recover' these lost threads. I think it is possible to write a complete server that deals with all these intricacies and gets everything just 'right', but as soon as you introduce 3rd party module writers, no matter how 'smart' we are, our castle of event goodness will crumble. I am looking for an alternative that doesn't expose all this crazyness of when to free, destruct, or lock things. The best idea I can come up with is for each Connection, it would become 'semi-sticky' to a single thread. Meaning each worker thread would have its own queue of upcoming events to process, and all events for connection X would sit on the same 'queue'. This would prevent two threads waiting for destruction, and other cases of a single connection's mutex locking up all your works, essentially providing basic fault isolation. These queues could be mutable, and you could 'move' a connection between queues, but you would always take all of its events and triggers, and move them together to a different queue. Does the 'connection event queue' idea make sense? I'm not sure I'm expressing the idea fully over email but I'll be at OSCON in a few weeks if anyone wants beer :) -Paul
Re: Events, Destruction and Locking
On Mon, Jul 6, 2009 at 10:20 PM, Paul Quernap...@querna.org wrote: I am looking for an alternative that doesn't expose all this crazyness of when to free, destruct, or lock things. The best idea I can come up with is for each Connection, it would become 'semi-sticky' to a single thread. Meaning each worker thread would have its own queue of upcoming events to process, and all events for connection X would sit on the same 'queue'. This would prevent two threads waiting for destruction, and other cases of a single connection's mutex locking up all your works, essentially providing basic fault isolation. These queues could be mutable, and you could 'move' a connection between queues, but you would always take all of its events and triggers, and move them together to a different queue. Does the 'connection event queue' idea make sense? I think I see where you're going with this...being so dependent upon mutexes seems...like going into a jungle full of guerillas armed with only a dull kitchen knife. So, a connection gets assigned to a 'thread' - but it has only two states: running or waiting for a network event. The critical part is that the thread *never* blocks on network traffic...all the 'network event' thread does is detect yup, ready to go and throws it back to that 'assigned' thread to process the event. Seems trivial enough to do with a serf-centric system. =) I'm not sure I'm expressing the idea fully over email but I'll be at OSCON in a few weeks if anyone wants beer :) I'll take you up on the beer and we can mull it over... -- justin
Re: Events, Destruction and Locking
Paul Querna wrote: Can't sleep, so finally writing this email I've been meaning to write for about 7 months now :D Pools don't help, but don't really make it worse, and are good enough for the actual cleanup part -- the difficultly lies in knowing *when* you can cleanup an object. Pool pre cleanup is meant to deal with such issues. You register a pre-cleanup and it will run before any of the pool objects are actually destroyed. In your case pre-cleanup callback could break the wait loop and make sure you don't reference a zombie object. The only issue left is guarding thread access to a singleton pollset interrupt (we even have pollset_interrupt with latest APR) from a pre-cleanup callback (or simply using a queue to serialize the objects that needs to get removed from the pollset) The exact problem we have/had with Tomcat native where shutting down the server can lead to JVM crash in case the connections are still in the pool waiting for the network event to happen. Regards -- ^(TM)
