[cgiapp] Tasks after fork()'ing a CGI::Application
Hi, I am forking a CGI::Application to perform a long-running task. The implementation is copied from this ingenious method: http://www.stonehenge.com/merlyn/LinuxMag/col39.html To summarize that article: The child process and begins the long task, logging its progress to a Cache::FileCache object. The parent adds a refresh header and exits. The browser refreshes the same URL and displays the contents of the Cache::FileCache object on each refresh. The child process writes a end-of-job string to the cache. When the refreshing parent finds the string, it removes the header and does not refresh anymore. I would like this to be as robust as possible. Currently, all I do in the child before launching the long-running process is close STDOUT. Is there any other tasks I must perform? Is there a CPAN module that might help clean up things in the child better than this? I tried Proc::Daemon but that closes all file descriptors inherited from the parent(including a few datafile descriptors that I need access to). I can of course re-open them from well know locations. Any advice on making this method more robust would be hugely appreciated. Regards # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Tasks after fork()'ing a CGI::Application
Gurunandan R. Bhat wrote: I would like this to be as robust as possible. Currently, all I do in the child before launching the long-running process is close STDOUT. Is there any other tasks I must perform? Make sure you don't have have any other file handles in use. This includes sockets and database handles. Is there a CPAN module that might help clean up things in the child better than this? I tried Proc::Daemon but that closes all file descriptors inherited from the parent(including a few datafile descriptors that I need access to). I can of course re-open them from well know locations. You really should re-open them. Strange things can happen when your system gets under a decent load that aren't apparent when testing or under a light load when you share file handles/sockets across processes like that. Any advice on making this method more robust would be hugely appreciated. I would actually suggest that you don't fork your web process to do background tasks. The problem is that your web processes are generally much heavier than you need which means wasted resources. It also can cause scalability concerns if you suddenly have lots of background tasks to perform. I'd suggest you look into having a dedicated job queue. You could use one of the existing frameworks like Gearman or TheSchwartz, or you could write you own using something like POE or Proc::Daemon+Parallel::ForkManager. -- Michael Peters Plus Three, LP # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Tasks after fork()'ing a CGI::Application
On Mon, Aug 10, 2009 at 11:18 AM, Michael Petersmpet...@plusthree.com wrote: I would actually suggest that you don't fork your web process to do background tasks. The problem is that your web processes are generally much heavier than you need which means wasted resources. I would definitely second this. Michael suggests using a separate daemon/queue -- that's definitely a good option, but another possibility would be to do a fork+exec in your code. With the exec, the existing process state is replaced with a new process. I've actually done something like this recently, so feel free to ask if you run into issues. The hard part is getting all the state you need into the new process; IPC::Open3 can work for this. -- Dan # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Tasks after fork()'ing a CGI::Application
On Mon, 2009-08-10 at 11:18 -0400, Michael Peters wrote: I would actually suggest that you don't fork your web process to do background tasks. The problem is that your web processes are generally much heavier than you need which means wasted resources. It also can cause scalability concerns if you suddenly have lots of background tasks to perform. I'd suggest you look into having a dedicated job queue. You could use one of the existing frameworks like Gearman or TheSchwartz, or you could write you own using something like POE or Proc::Daemon+Parallel::ForkManager. Thanks. I did seriously consider TheSchwartz and gearman. I have two features that I want to implement: First, I want to pass some feedback to the user who queued the task indicating progress. Second, I must be able to schedule it at a future time. I am currently using Schedule::At to do this. Again both are fragile and as you correctly pointed out, will suffer when scaled. I had not looked at POE so I will now. Thanks once again Regards # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Tasks after fork()'ing a CGI::Application
Gurunandan R. Bhat wrote: Thanks. I did seriously consider TheSchwartz and gearman. I have two features that I want to implement: First, I want to pass some feedback to the user who queued the task indicating progress. I normally use a custom job queue for more flexibility and store the jobs in the database (as serialized data structures). Then my web processes can query the DB to see if the job in question has finished and thus give a progress report to the user. Second, I must be able to schedule it at a future time. I am currently using Schedule::At to do this. Again both are fragile and as you correctly pointed out, will suffer when scaled. My queue also allows this, but I'm not sure how TheSchwartz or Gearman handle that. -- Michael Peters Plus Three, LP # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Tasks after fork()'ing a CGI::Application
Michael Peters wrote: Gurunandan R. Bhat wrote: Thanks. I did seriously consider TheSchwartz and gearman. I have two features that I want to implement: First, I want to pass some feedback to the user who queued the task indicating progress. I normally use a custom job queue for more flexibility and store the jobs in the database (as serialized data structures). Then my web processes can query the DB to see if the job in question has finished and thus give a progress report to the user. Second, I must be able to schedule it at a future time. I am currently using Schedule::At to do this. Again both are fragile and as you correctly pointed out, will suffer when scaled. My queue also allows this, but I'm not sure how TheSchwartz or Gearman handle that. I'm happily using beanstalk for this (http://xph.us/software/beanstalkd/), through Beanstalk::Client. Very light-weight, and very easy to work with. Delaying jobs into the future is also possible (and I use that extensively). When I need to keep the user informed of the current status, I tend to track that in a database, or even in memcached if it isn't vitally important. Rhesa # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Tasks after fork()'ing a CGI::Application
On Mon, 2009-08-10 at 18:17 +0200, Rhesa Rozendaal wrote: Michael Peters wrote: Gurunandan R. Bhat wrote: Thanks. I did seriously consider TheSchwartz and gearman. I have two features that I want to implement: First, I want to pass some feedback to the user who queued the task indicating progress. I normally use a custom job queue for more flexibility and store the jobs in the database (as serialized data structures). Then my web processes can query the DB to see if the job in question has finished and thus give a progress report to the user. Second, I must be able to schedule it at a future time. I am currently using Schedule::At to do this. Again both are fragile and as you correctly pointed out, will suffer when scaled. My queue also allows this, but I'm not sure how TheSchwartz or Gearman handle that. I'm happily using beanstalk for this (http://xph.us/software/beanstalkd/), through Beanstalk::Client. Very light-weight, and very easy to work with. Delaying jobs into the future is also possible (and I use that extensively). When I need to keep the user informed of the current status, I tend to track that in a database, or even in memcached if it isn't vitally important. Thanks a ton!! beanstalkd with its delay = 'secs' seems to be exactly what I wanted. Will serialize progress in a database as you have suggested. Appreciate your help. Regards # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
[cgiapp] Routes and get_current_runmode()
I am experiencing a problem with C::A::P::Routes, and I think I know
what is going wrong, but I don’t know why or how to fix it.
Background: I have implemented my application with C::A::Dispatch and
C::A::P::Authentication, and it works well. For various reasons I like
C::A::P::Routes, and would like to use it instead of C::A::Dispatch.
However, C::A::P::Routes doesn’t seem to respect the protected modes
set up by C::A::P::Authentication, and shows the protected modes
without letting the login form kick in.
So, I investigated and discovered that C::A::P::Routes has a
disconnect between what it thinks is the runmode and the value of
$self-get_current_runmode. Here is my test --
sub setup {
my $self = shift;
$self-start_mode('welcome');
$self-routes([
# unprotected runmodes
'/welcome' = 'welcome',
'/login' = 'login',
# protected runmodes
'/mainpage'= 'mainpage',
]);
$self-param(protected_runmodes = ['mainpage']);
}
sub cgiapp_prerun {
my $self = shift;
$self-session-param('__RUN_MODE1', $self-get_current_runmode());
my $protected_runmodes = $self-param('protected_runmodes');
foreach my $rm (@$protected_runmodes) {
if ($self-get_current_runmode() eq $rm) {
$self-prerun_mode('login')
unless($self-session-param('__LOGGED_IN'));
last;
}
}
#$self-prerun_mode('login');
$self-session-param('__RUN_MODE2', $self-get_current_runmode());
}
sub mainpage {
my $self = shift;
my $tmpl = $self-load_tmpl('mainpage.html');
$tmpl-param(runmode1 = $self-session-param('__RUN_MODE1'));
$tmpl-param(runmode2 = $self-session-param('__RUN_MODE2'));
$tmpl-param(debug = $self-routes_dbg); #dumps all the
C::A::P::Routes info
return $tmpl-output;
}
And, when I point my browser to ‘http://myapp/mainpage’ this is what I get
Main Page!
You are now logged in!
Runmode1: welcome
Runmode2: welcome
Debug: $VAR1 = { 'rule_matched: ' = '/mainpage', 'parsed_params: ' =
{}, 'path_received: ' = '/mainpage', 'runmode: ' = 'mainpage' };
See what is going on -- C::A::P::Routes correctly thinks the runmode
is ‘mainpage’ as per the URL. However, $self-get_current_runmode()
returns ‘welcome’ in cgiapp_prerun(), and as a result, the mainpage is
not protected. Now, it could be that C::A::P::Routes is firing *after*
my cgiapp_prerun() protected_mode logic is run. To test that, I
uncommented the #$self-prerun_mode('login'); line in cgiapp_prerun(),
and lo and behold, the runmode is changed to login.
The remedy seems to be that once it has figured out the runmode
correctly, C::A::P::Routes should set the value of
$self-get_current_runmode() to it.
If I have diagnosed the problem correctly, please help me find a fix.
If not, please help me diagnose correctly.
Many thanks,
--
Puneet Kishor http://www.punkish.org
Carbon Model http://carbonmodel.org
Charter Member, Open Source Geospatial Foundation http://www.osgeo.org
Science Commons Fellow, http://sciencecommons.org/about/whoweare/kishor
Nelson Institute, UW-Madison http://www.nelson.wisc.edu
---
Assertions are politics; backing up assertions with evidence is science
===
Sent from Madison, WI, United States
# CGI::Application community mailing list
####
## To unsubscribe, or change your message delivery options, ##
## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp##
####
## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ##
## Wiki: http://cgiapp.erlbaum.net/ ##
####
[cgiapp] Re: Custom Error Messages - CGI::Applicaiton::Plugin::ValidateRM
On Mon, 03 Aug 2009 14:54:10 -0700 a...@spatialsystems.org wrote: I have a textarea field to evaluate and peg errors line by line. So if line 2 and line 5 are invalid, I want to note that as two different errors. It is normal for fields to have more than one constraint, with the possiblity that more than one of them can fail at a time. Usually this is still considered one constraint with one name, but the error messages get appended together. If you want separate tokens for each error message, then that's another story. You could use a field_filter to split the text area into multiple fields before it gets processed by the constraint_methods, and then define unique constraints for each line. Mark -- http://mark.stosberg.com/ # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
[cgiapp] Re: Routes and get_current_runmode()
See what is going on -- C::A::P::Routes correctly thinks the runmode
is ‘mainpage’ as per the URL. However, $self-get_current_runmode()
returns ‘welcome’ in cgiapp_prerun(), and as a result, the mainpage is
not protected. Now, it could be that C::A::P::Routes is firing *after*
my cgiapp_prerun() protected_mode logic is run. To test that, I
uncommented the #$self-prerun_mode('login'); line in cgiapp_prerun(),
and lo and behold, the runmode is changed to login.
I agree with you assessment of where the problem is.
The remedy seems to be that once it has figured out the runmode
correctly, C::A::P::Routes should set the value of
$self-get_current_runmode() to it.
If I have diagnosed the problem correctly, please help me find a fix.
If not, please help me diagnose correctly.
My suggestion for a fix is to see about getting the the routes prerun
before and your prerun behavior to trigger in the other order.
scratches head
maybe this will help?:
Instead of:
sub cgiapp_prerun { ... }
Try
__PACKAGE__-add_callback( prerun = 'my_prerun' );
sub my_prerun { ... }
I'm just guessing. You can also dump this to check the order
of what's registered.
$self-{__INSTALLED_CALLBACKS}
Mark
--
http://mark.stosberg.com/
# CGI::Application community mailing list
####
## To unsubscribe, or change your message delivery options, ##
## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp##
####
## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ##
## Wiki: http://cgiapp.erlbaum.net/ ##
####
Re: [cgiapp] Re: Routes and get_current_runmode()
On Mon, Aug 10, 2009 at 7:46 PM, Mark Stosbergm...@summersault.com wrote:
See what is going on -- C::A::P::Routes correctly thinks the runmode
is ‘mainpage’ as per the URL. However, $self-get_current_runmode()
returns ‘welcome’ in cgiapp_prerun(), and as a result, the mainpage is
not protected. Now, it could be that C::A::P::Routes is firing *after*
my cgiapp_prerun() protected_mode logic is run. To test that, I
uncommented the #$self-prerun_mode('login'); line in cgiapp_prerun(),
and lo and behold, the runmode is changed to login.
I agree with you assessment of where the problem is.
The remedy seems to be that once it has figured out the runmode
correctly, C::A::P::Routes should set the value of
$self-get_current_runmode() to it.
If I have diagnosed the problem correctly, please help me find a fix.
If not, please help me diagnose correctly.
My suggestion for a fix is to see about getting the the routes prerun
before and your prerun behavior to trigger in the other order.
scratches head
maybe this will help?:
Instead of:
sub cgiapp_prerun { ... }
Try
__PACKAGE__-add_callback( prerun = 'my_prerun' );
sub my_prerun { ... }
Doesn't. Still the same problem.
I'm just guessing. You can also dump this to check the order
of what's registered.
$self-{__INSTALLED_CALLBACKS}
I get the following
Installed callbacks: $VAR1 = { 'load_tmpl' = [], 'prerun' = [
'my_prerun' ], 'init' = [] };
Mark
--
http://mark.stosberg.com/
--
Puneet Kishor http://www.punkish.org
Carbon Model http://carbonmodel.org
Charter Member, Open Source Geospatial Foundation http://www.osgeo.org
Science Commons Fellow, http://sciencecommons.org/about/whoweare/kishor
Nelson Institute, UW-Madison http://www.nelson.wisc.edu
---
Assertions are politics; backing up assertions with evidence is science
===
# CGI::Application community mailing list
####
## To unsubscribe, or change your message delivery options, ##
## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp##
####
## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ##
## Wiki: http://cgiapp.erlbaum.net/ ##
####
[cgiapp] Re: Routes and get_current_runmode()
Replying to my own email to document what I have done, and seek
further advice. Please see below --
On Mon, Aug 10, 2009 at 3:35 PM, P Kishorpunk.k...@gmail.com wrote:
I am experiencing a problem with C::A::P::Routes, and I think I know
what is going wrong, but I don’t know why or how to fix it.
Background: I have implemented my application with C::A::Dispatch and
C::A::P::Authentication, and it works well. For various reasons I like
C::A::P::Routes, and would like to use it instead of C::A::Dispatch.
However, C::A::P::Routes doesn’t seem to respect the protected modes
set up by C::A::P::Authentication, and shows the protected modes
without letting the login form kick in.
So, I investigated and discovered that C::A::P::Routes has a
disconnect between what it thinks is the runmode and the value of
$self-get_current_runmode. Here is my test --
sub setup {
my $self = shift;
$self-start_mode('welcome');
$self-routes([
# unprotected runmodes
'/welcome' = 'welcome',
'/login' = 'login',
# protected runmodes
'/mainpage'= 'mainpage',
]);
$self-param(protected_runmodes = ['mainpage']);
}
sub cgiapp_prerun {
my $self = shift;
$self-session-param('__RUN_MODE1', $self-get_current_runmode());
my $protected_runmodes = $self-param('protected_runmodes');
foreach my $rm (@$protected_runmodes) {
if ($self-get_current_runmode() eq $rm) {
$self-prerun_mode('login')
unless($self-session-param('__LOGGED_IN'));
last;
}
}
#$self-prerun_mode('login');
$self-session-param('__RUN_MODE2', $self-get_current_runmode());
}
In cgiapp_prerun, I changed the following line
foreach my $rm (@$protected_runmodes) {
if ($self-get_current_runmode() eq $rm) {
$self-prerun_mode('login')
unless($self-session-param('__LOGGED_IN'));
last;
}
to
if ($self-prerun_mode() eq $rm) {
and now it works as expected.
Looking at the code of C::A::P::Routes, it seems to be setting the
prerun_mode() correctly, but there doesn't seem to be any method for
updating the value of get_current_runmode(). In fact, in the code for
CGI::Application itself there doesn't seem to be any easy facility for
changing the value of get_current_runmode() which seems to get its
value from $self-{__CURRENT_RUNMODE} which seems to be set in
sub run {
..
# Set get_current_runmode() for access by user later
$self-{__CURRENT_RUNMODE} = $rm;
}
To me, C::A::P::Routes seems more in the spirit of CGI::Application
than C::A::D is, and I would rather use Routes, but something needs to
be changed somewhere for it to work properly.
sub mainpage {
my $self = shift;
my $tmpl = $self-load_tmpl('mainpage.html');
$tmpl-param(runmode1 = $self-session-param('__RUN_MODE1'));
$tmpl-param(runmode2 = $self-session-param('__RUN_MODE2'));
$tmpl-param(debug = $self-routes_dbg); #dumps all the
C::A::P::Routes info
return $tmpl-output;
}
And, when I point my browser to ‘http://myapp/mainpage’ this is what I get
Main Page!
You are now logged in!
Runmode1: welcome
Runmode2: welcome
Debug: $VAR1 = { 'rule_matched: ' = '/mainpage', 'parsed_params: ' =
{}, 'path_received: ' = '/mainpage', 'runmode: ' = 'mainpage' };
See what is going on -- C::A::P::Routes correctly thinks the runmode
is ‘mainpage’ as per the URL. However, $self-get_current_runmode()
returns ‘welcome’ in cgiapp_prerun(), and as a result, the mainpage is
not protected. Now, it could be that C::A::P::Routes is firing *after*
my cgiapp_prerun() protected_mode logic is run. To test that, I
uncommented the #$self-prerun_mode('login'); line in cgiapp_prerun(),
and lo and behold, the runmode is changed to login.
The remedy seems to be that once it has figured out the runmode
correctly, C::A::P::Routes should set the value of
$self-get_current_runmode() to it.
If I have diagnosed the problem correctly, please help me find a fix.
If not, please help me diagnose correctly.
Many thanks,
--
Puneet Kishor http://www.punkish.org
Carbon Model http://carbonmodel.org
Charter Member, Open Source Geospatial Foundation http://www.osgeo.org
Science Commons Fellow, http://sciencecommons.org/about/whoweare/kishor
Nelson Institute, UW-Madison http://www.nelson.wisc.edu
---
Assertions are politics; backing up assertions with evidence is science
===
Sent from Madison, WI, United States
# CGI::Application community mailing list
####
## To unsubscribe, or change your message delivery options, ##
## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp##
####
## Web archive:
