[OT] backend processing of large jobs
I have what is probably a pretty common situation. The user submits a file for processing to my application. It takes many minutes to process this file, so I would like to submit this file as a job to a backend server that can queue up the files as they come in, process them, maintain status information on the queue and individual jobs, and finally deposit the results in a database. I would like to be able to query this backend job server for status on a job (by a job_id) to allow for the pylons web application to check ad lib for job status/completion. I have looked at constructing a standalone XML-RPC server or using twisted, but there are other ways, I am sure. Any suggestions? Thanks, Sean --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to [EMAIL PROTECTED] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: [OT] backend processing of large jobs
On 4/12/07, Alberto Valverde [EMAIL PROTECTED] wrote: On Apr 12, 2007, at 2:33 PM, Sean Davis wrote: I have what is probably a pretty common situation. The user submits a file for processing to my application. It takes many minutes to process this file, so I would like to submit this file as a job to a backend server that can queue up the files as they come in, process them, maintain status information on the queue and individual jobs, and finally deposit the results in a database. I would like to be able to query this backend job server for status on a job (by a job_id) to allow for the pylons web application to check ad lib for job status/completion. I have looked at constructing a standalone XML-RPC server or using twisted, but there are other ways, I am sure. Any suggestions? I've been using http://cheeseshop.python.org/pypi/threadpool/1.2.2 with Pylons' XML-RPC server for something similar succesfully. Thanks, Alberto. That is what I was thinking. Just to followup, if I may, but where do you keep your threadpool so that all requests have access to it? In a global variable? Sean --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to [EMAIL PROTECTED] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: [OT] backend processing of large jobs
On Apr 12, 2007, at 4:25 PM, Sean Davis wrote: Thanks, Alberto. That is what I was thinking. Just to followup, if I may, but where do you keep your threadpool so that all requests have access to it? In a global variable? place it at yourapp/lib/app_globals:Globals binding it to self inside __init__. You'll have it availbale at pylons.g Alberto --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to [EMAIL PROTECTED] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
