[cgiapp] Streaming Status Updates for a Long-running Run Mode
I've written a cgiapp that runs about 20 java processes in the background and I'd like to update the web page that submitted the request (via Ajax) to run them with status, indicating when each process has completed. Currently, my rm method just sits there running the subprocesses (using Paralell::ForkManager, btw) until it's done. This could be upwards of 15 minutes, so the initial request is treated as failed. How can I update my web page with status from this run mode while the subprocesses are running? Thanks! Eric ___ This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group. ___ # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Streaming Status Updates for a Long-running Run Mode
On 10/07/2009 10:33 AM, eric.b...@barclayscapital.com wrote: How can I update my web page with status from this run mode while the subprocesses are running? There's a couple of different ways this can be done. The best is definitely not the easiest but it means creating a separate offline job queue that can run these processes and keep their status in a shared location (like a DB, etc). Then your rm just adds a job to the queue and returns. Then when you're checking on the status via Ajax you need to have another rm that simply checks on the status of the job and returns a flag that could mean success, failure, or try again later (still working). Unfortunately, since you are using Ajax, you can't use the standard NPH approach and just periodically print something to the browser while it's working. Ajax requests don't return (in the javascript) until the entire request is done and it will probably timeout on you. -- Michael Peters Plus Three, LP # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Streaming Status Updates for a Long-running Run Mode
Thanks, Michael. Good point about AJAX not returning until the entire request is done. The approach I'm working on now is to have a global status hash for each file being processed, which I'll update as I run each subprocess and when each completes. Then I've thrown in a quick run mode that simply sends that hash back as JSON and I'm putting some logic into my page to periodically make a request for this runmode so I can update the page with current status info. On a related note, is there a way to have cgiapp send content back to the client during the processing of a run mode instead of at the end when the rm returns? Eric -Original Message- From: cgiapp-boun...@lists.erlbaum.net [mailto:cgiapp-boun...@lists.erlbaum.net] On Behalf Of Michael Peters Sent: Wednesday, October 07, 2009 11:28 AM To: CGI Application Subject: Re: [cgiapp] Streaming Status Updates for a Long-running Run Mode On 10/07/2009 10:33 AM, eric.b...@barclayscapital.com wrote: How can I update my web page with status from this run mode while the subprocesses are running? There's a couple of different ways this can be done. The best is definitely not the easiest but it means creating a separate offline job queue that can run these processes and keep their status in a shared location (like a DB, etc). Then your rm just adds a job to the queue and returns. Then when you're checking on the status via Ajax you need to have another rm that simply checks on the status of the job and returns a flag that could mean success, failure, or try again later (still working). Unfortunately, since you are using Ajax, you can't use the standard NPH approach and just periodically print something to the browser while it's working. Ajax requests don't return (in the javascript) until the entire request is done and it will probably timeout on you. -- Michael Peters Plus Three, LP ___ This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group. ___ # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Streaming Status Updates for a Long-running Run Mode
On 10/07/2009 11:35 AM, eric.b...@barclayscapital.com wrote: The approach I'm working on now is to have a global status hash for each file being processed, which I'll update as I run each subprocess and when each completes. Then I've thrown in a quick run mode that simply sends that hash back as JSON and I'm putting some logic into my page to periodically make a request for this runmode so I can update the page with current status info. I really wouldn't do that. Don't tied up your web server for long running tasks just so that you can wait to show a status to the user. Even something as simple as Unix at for a simple queue would be better. On a related note, is there a way to have cgiapp send content back to the client during the processing of a run mode instead of at the end when the rm returns? No, not for Ajax. And it's not a limitation of cgiapp, but of HTTP/Ajax. For non-Ajax you use Non-Parsed-Headers (NPH) which means you tell C::A to not send headers and you then instead print them yourselves. And then periodically print more things to the client (like some JS that updates a progress bar, etc). But like I said in the other email, this won't work for Ajax. -- Michael Peters Plus Three, LP # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Streaming Status Updates for a Long-running Run Mode
On 10/07/2009 12:24 PM, eric.b...@barclayscapital.com wrote:
I wouldn't do that either, but I've had no end of issues with running
subprocesses from Apache2.
Me: Doctor it hurts when I try to run subprocesses from Apache2.
Doctor: Don't do that :)
What I try to do is have an external process which looks for jobs to do
and then does them. Then my web app simply puts things in the job db
table and the job queue picks them up and processes them, putting the
status (complete, failed, working, etc) back in the db table. Then my
web processes that check the status of jobs checks to see if it's done
and then shows that in the browser.
Almost all web applications that I know have some things that should be
delegated to an external job queue. Even something as simple as sending
an email shouldn't be done from the web processes (could get stuck on
DNS lookups or SMTP waiting, etc).
I just ran across the 'at' solution, which I like quite a bit, however
it turns out that we do not receive the email for the users that the web
server runs under and 'at' sends error and other confirmation
information via email, so I have no idea at this point why my
subprocesses aren't running.
If you want to go the at route instead of writing your own queue (or
using an existing queue) then have at call a Perl script that wraps
around the real process doing the work. Then it could set a failed
status and error messages in the DB or something that you could have
access to. Obviously if this wrapper fails it would have the same
problems, but that should be pretty infrequent since your wrapper could
be really simple.
Understood that you can't do this with Ajax, but how would you do it
with cgiapp in a non-ajax context?
Something like this rough code should work
$self-header_type('none');
print Content-type: text/html\n\n;
$print $main_content; # but don't close the html tag yet
while($not_done) {
# do some work
...
# periodically print so the browser doesn't close the socket
print \0;
# or print some progress that the already printed document knows
# how to deal with
print script type=text/javascriptupdate_progress(10)/script;
}
print /html;
return;
--
Michael Peters
Plus Three, LP
# CGI::Application community mailing list
####
## To unsubscribe, or change your message delivery options, ##
## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp##
####
## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ##
## Wiki: http://cgiapp.erlbaum.net/ ##
####
Re: [cgiapp] Streaming Status Updates for a Long-running Run Mode
-Original Message- From: cgiapp-boun...@lists.erlbaum.net [mailto:cgiapp-boun...@lists.erlbaum.net] On Behalf Of Michael Peters Sent: Wednesday, October 07, 2009 11:44 AM To: CGI Application Subject: Re: [cgiapp] Streaming Status Updates for a Long-running Run Mode On 10/07/2009 11:35 AM, eric.b...@barclayscapital.com wrote: The approach I'm working on now is to have a global status hash for each file being processed, which I'll update as I run each subprocess and when each completes. Then I've thrown in a quick run mode that simply sends that hash back as JSON and I'm putting some logic into my page to periodically make a request for this runmode so I can update the page with current status info. I really wouldn't do that. Don't tied up your web server for long running tasks just so that you can wait to show a status to the user. Even something as simple as Unix at for a simple queue would be better. I wouldn't do that either, but I've had no end of issues with running subprocesses from Apache2. Among them is that of the environment's not being passed to the subprocess, the way apache messes with STDOUT, and the fact that it insists on starting CGI's (mod_perl handlers) in the directory in which they are invoked, regardless of the current working directory of the mod_perl handler. I just ran across the 'at' solution, which I like quite a bit, however it turns out that we do not receive the email for the users that the web server runs under and 'at' sends error and other confirmation information via email, so I have no idea at this point why my subprocesses aren't running. Yes. It's heinous to tie up apache with running subprocesses, but regardless of whether I'm using Apache2::SubProcess-spawn_proc_prog() or IPC::Run3, I still get screwed one way or another. I wrote a wrapper for running via a number of different methods when this problem first emerged after migrating to apache2. I used Module::Pluggable, so all I had to do was to implement a runner plugin that uses 'at', which was nice, but it's failing and I'll have to wait until I get our IT crew to understand and help with the email problem. (don't ask) On a related note, is there a way to have cgiapp send content back to the client during the processing of a run mode instead of at the end when the rm returns? No, not for Ajax. And it's not a limitation of cgiapp, but of HTTP/Ajax. For non-Ajax you use Non-Parsed-Headers (NPH) which means you tell C::A to not send headers and you then instead print them yourselves. And then periodically print more things to the client (like some JS that updates a progress bar, etc). But like I said in the other email, this won't work for Ajax. Understood that you can't do this with Ajax, but how would you do it with cgiapp in a non-ajax context? -- Michael Peters Plus Three, LP ___ This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group. ___ # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Streaming Status Updates for a Long-running Run Mode
* eric.b...@barclayscapital.com [07/10/2009 21:05] : What do you use for your external job queue? Please please take a look at TheSchwartz. http://search.cpan.org/~bradfitz/TheSchwartz-1.07/ Emmanuel # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
