I also just realized that your email subject suggests another option for the name: spawn-on-accept-program. It's kinda long but pretty descriptive.
Cheers, Roger On Thu, Sep 23, 2010 at 7:13 PM, Roger Hoover <[email protected]>wrote: > Hi Bruce, > > On Thu, Sep 23, 2010 at 6:22 PM, Bruce Frederiksen <[email protected]>wrote: > >> Roger, >> >> On Thu, Sep 23, 2010 at 6:46 PM, Roger Hoover <[email protected]>wrote: >> >>> Hi Bruce, >>> >>> On Thu, Sep 23, 2010 at 1:41 PM, Bruce Frederiksen >>> <[email protected]>wrote: >>> >>>> I'm new to supervisor and would like a capability that takes the >>>> fcgi-program feature one more step. >>>> >>>> What I want is for supervisord to create a socket (like it does for >>>> fcgi-program), but then also do the accept on it. Each time a connection >>>> comes in, spawn the child process and hand-off the connection to the new >>>> child. From that point forward, supervisord is not involved in the >>>> communication between the connecting client and the spawned server. >>>> >>> >>> Do you mean always spawn a new process per connection? If so, that's how >>> CGI works so you should just use that instead of FCGI. If you mean >>> dynamically spawning a limited number of FCGI processes on demand, then I >>> think an nginx_plugin is the way to go. See A below. >>> >> >> I'm not running http servers here. So they are not running behind nginx >> or apache. Actually, these are one level removed from that. What we have >> is apache -> front-end-server -> back-end-server. I'm talking about the >> connections between the front-end-servers and the back-end-servers (which >> run on different boxes). This connection is over an intranet and does not >> use the http protocol. These connections are long-lived, but I want to give >> the front-end-servers the ability to control the number of back-end-server >> processes depending on work load. >> >> So what I want is to simply connect to the back-end box to spawn a >> back-end-server process if the work load goes up. Then closing the >> connection will terminate the process, if the work load later goes down. >> >> All I need is a process manager that baby-sits these back-end-processes. >> That's where supervisor comes in... >> > > Got it. Thanks for explaining. I think this is the way that inetd/xinetd > work. > >> >> >>> >>>> The reason I want this is to give the client control over how many >>>> server processes are spawned. Supervisord would create the socket when it >>>> starts, but not spawn any of these type of processes until connections come >>>> in. This is a case where I am writing both the client and server, and the >>>> server is only visible on the internal network. >>>> >>>> A. Does anybody know if something like this already exists? >>>> >>> >>> Can what you need be accomplished with this nginx plugin? >>> http://labs.frickle.com/nginx_ngx_supervisord/ >>> >>> I haven't actually tried it yet but it looks like you can configure it to >>> spawn fcgi-programs on demand >>> >>> >>> upstream backend { >>> server 127.0.0.1:8000; >>> server 127.0.0.1:8001; >>> supervisord 127.0.0.1:9001 admin:super; >>> fair; >>> } >>> >>> server { >>> location / { >>> proxy_pass http://backend; >>> } >>> } >>> >>> >>>> B. For those "in the know" on the internal architecture of supervisor, >>>> what pointers do you have on how to implement this? >>>> >>> >>> I think this would be pretty hard to do it in the supervisord process. >>> You have to jump two hurdles. >>> a) You'd have to figure out a reliable way to know when an FCGI socket >>> needs to be cleaned up. If you search recent posts to this lists, you'll >>> see that the currently implementation keeps a count of how many FCGI child >>> processes there are and closes the socket when the count hits zero and >>> creates it when the count goes above zero. I won't go into the details here >>> as to why but this problem would have to be overcome if you want to create >>> the socket before any FCGI children have been spawned. >>> >> >> This problem is due to the different usage of how fcgi-program works. >> Because supervisord is not doing the accept, when there are no child >> processes there is nothing to do the accept and you have a problem. Hence >> the need to close the socket in supervisord so that clients get an immediate >> "server refusing connection" error. >> >> But I want supervisord to always have the socket open and always be doing >> an accept on it. Thus clients will never be turned away. >> >> Another difference is that in the fcgi-program case, supervisord hands off >> the listening socket to each child, which each accept their own >> connections. But in my case, supervisord only hands off the new connection >> sockets. The child processes do not get the listening socket, they only get >> the one new connection socket. The child process never deals with the >> listening socket, and supervisord never deals with the new connection >> sockets (except to hand them off to the new child process). Thus, there is >> a cleaner separation of responsibilities. >> > > OK. This makes sense. As you say, it's a different model from FastCGI so > I think you should create a new type of supervisord "program", something > like "inetd-program", "superserver-program" (inetd is sometimes called the > superserver) or "fork-program" (as opposed to prefork). I think your > servers would be compatible with inetd so that might be a good name. > >> >> b) You can't do any blocking network calls so you'd have to plugin into >>> the main select() loop. I'm not sure how hard or easy that will be. Then I >>> think it would have to work something like this: you'd have to create the >>> socket with the O_NONBLOCK flag set until the first connection comes and set >>> it back to blocking before handing it off to the first child process. Keep >>> calling accept() and checking for EWOULDBLOCK or EAGAIN until it's ready. >>> Then you spawn the appropriate FCGI process to handle it. >>> >> >> Overall, it does not look that complicated. It looks like supervisord has >> the architecture already in place to support this. >> > > I think you should be able to follow a similar pattern as I used for > fcgi-programs and create a new type of program. > >> >> Another question, should I be working on trunk, or do the project owners >> want to do an svn branch for this? >> >> Thank you for your response! >> >> -Bruce >> > >
_______________________________________________ Supervisor-users mailing list [email protected] http://lists.supervisord.org/mailman/listinfo/supervisor-users
