[cgiapp] Tasks after fork()'ing a CGI::Application
Hi, I am forking a CGI::Application to perform a long-running task. The implementation is copied from this ingenious method: http://www.stonehenge.com/merlyn/LinuxMag/col39.html To summarize that article: The child process and begins the long task, logging its progress to a Cache::FileCache object. The parent adds a refresh header and exits. The browser refreshes the same URL and displays the contents of the Cache::FileCache object on each refresh. The child process writes a end-of-job string to the cache. When the refreshing parent finds the string, it removes the header and does not refresh anymore. I would like this to be as robust as possible. Currently, all I do in the child before launching the long-running process is close STDOUT. Is there any other tasks I must perform? Is there a CPAN module that might help clean up things in the child better than this? I tried Proc::Daemon but that closes all file descriptors inherited from the parent(including a few datafile descriptors that I need access to). I can of course re-open them from well know locations. Any advice on making this method more robust would be hugely appreciated. Regards # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Tasks after fork()'ing a CGI::Application
Gurunandan R. Bhat wrote: I would like this to be as robust as possible. Currently, all I do in the child before launching the long-running process is close STDOUT. Is there any other tasks I must perform? Make sure you don't have have any other file handles in use. This includes sockets and database handles. Is there a CPAN module that might help clean up things in the child better than this? I tried Proc::Daemon but that closes all file descriptors inherited from the parent(including a few datafile descriptors that I need access to). I can of course re-open them from well know locations. You really should re-open them. Strange things can happen when your system gets under a decent load that aren't apparent when testing or under a light load when you share file handles/sockets across processes like that. Any advice on making this method more robust would be hugely appreciated. I would actually suggest that you don't fork your web process to do background tasks. The problem is that your web processes are generally much heavier than you need which means wasted resources. It also can cause scalability concerns if you suddenly have lots of background tasks to perform. I'd suggest you look into having a dedicated job queue. You could use one of the existing frameworks like Gearman or TheSchwartz, or you could write you own using something like POE or Proc::Daemon+Parallel::ForkManager. -- Michael Peters Plus Three, LP # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Tasks after fork()'ing a CGI::Application
On Mon, Aug 10, 2009 at 11:18 AM, Michael Petersmpet...@plusthree.com wrote: I would actually suggest that you don't fork your web process to do background tasks. The problem is that your web processes are generally much heavier than you need which means wasted resources. I would definitely second this. Michael suggests using a separate daemon/queue -- that's definitely a good option, but another possibility would be to do a fork+exec in your code. With the exec, the existing process state is replaced with a new process. I've actually done something like this recently, so feel free to ask if you run into issues. The hard part is getting all the state you need into the new process; IPC::Open3 can work for this. -- Dan # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Tasks after fork()'ing a CGI::Application
On Mon, 2009-08-10 at 11:18 -0400, Michael Peters wrote: I would actually suggest that you don't fork your web process to do background tasks. The problem is that your web processes are generally much heavier than you need which means wasted resources. It also can cause scalability concerns if you suddenly have lots of background tasks to perform. I'd suggest you look into having a dedicated job queue. You could use one of the existing frameworks like Gearman or TheSchwartz, or you could write you own using something like POE or Proc::Daemon+Parallel::ForkManager. Thanks. I did seriously consider TheSchwartz and gearman. I have two features that I want to implement: First, I want to pass some feedback to the user who queued the task indicating progress. Second, I must be able to schedule it at a future time. I am currently using Schedule::At to do this. Again both are fragile and as you correctly pointed out, will suffer when scaled. I had not looked at POE so I will now. Thanks once again Regards # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Tasks after fork()'ing a CGI::Application
Gurunandan R. Bhat wrote: Thanks. I did seriously consider TheSchwartz and gearman. I have two features that I want to implement: First, I want to pass some feedback to the user who queued the task indicating progress. I normally use a custom job queue for more flexibility and store the jobs in the database (as serialized data structures). Then my web processes can query the DB to see if the job in question has finished and thus give a progress report to the user. Second, I must be able to schedule it at a future time. I am currently using Schedule::At to do this. Again both are fragile and as you correctly pointed out, will suffer when scaled. My queue also allows this, but I'm not sure how TheSchwartz or Gearman handle that. -- Michael Peters Plus Three, LP # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Tasks after fork()'ing a CGI::Application
Michael Peters wrote: Gurunandan R. Bhat wrote: Thanks. I did seriously consider TheSchwartz and gearman. I have two features that I want to implement: First, I want to pass some feedback to the user who queued the task indicating progress. I normally use a custom job queue for more flexibility and store the jobs in the database (as serialized data structures). Then my web processes can query the DB to see if the job in question has finished and thus give a progress report to the user. Second, I must be able to schedule it at a future time. I am currently using Schedule::At to do this. Again both are fragile and as you correctly pointed out, will suffer when scaled. My queue also allows this, but I'm not sure how TheSchwartz or Gearman handle that. I'm happily using beanstalk for this (http://xph.us/software/beanstalkd/), through Beanstalk::Client. Very light-weight, and very easy to work with. Delaying jobs into the future is also possible (and I use that extensively). When I need to keep the user informed of the current status, I tend to track that in a database, or even in memcached if it isn't vitally important. Rhesa # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Tasks after fork()'ing a CGI::Application
On Mon, 2009-08-10 at 18:17 +0200, Rhesa Rozendaal wrote: Michael Peters wrote: Gurunandan R. Bhat wrote: Thanks. I did seriously consider TheSchwartz and gearman. I have two features that I want to implement: First, I want to pass some feedback to the user who queued the task indicating progress. I normally use a custom job queue for more flexibility and store the jobs in the database (as serialized data structures). Then my web processes can query the DB to see if the job in question has finished and thus give a progress report to the user. Second, I must be able to schedule it at a future time. I am currently using Schedule::At to do this. Again both are fragile and as you correctly pointed out, will suffer when scaled. My queue also allows this, but I'm not sure how TheSchwartz or Gearman handle that. I'm happily using beanstalk for this (http://xph.us/software/beanstalkd/), through Beanstalk::Client. Very light-weight, and very easy to work with. Delaying jobs into the future is also possible (and I use that extensively). When I need to keep the user informed of the current status, I tend to track that in a database, or even in memcached if it isn't vitally important. Thanks a ton!! beanstalkd with its delay = 'secs' seems to be exactly what I wanted. Will serialize progress in a database as you have suggested. Appreciate your help. Regards # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
