Re: Spawning a worker thread from the controller?
On May 7, 12:07 pm, Ksenia Marasanova [EMAIL PROTECTED] wrote: Just wanted to point to a possible alternative way: I usually save jobs like this into the database. The cron job checks every X minutes for a jobs and runs it if needed, changing status to X% completed. It's pretty easy than to display a progress bar or other status notifications in the application. How would you run a job every X minutes? Are you using an external scheduler (unix-cron)? Greetings Christoph --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Spawning a worker thread from the controller?
Just wanted to point to a possible alternative way: I usually save jobs like this into the database. The cron job checks every X minutes for a jobs and runs it if needed, changing status to X% completed. It's pretty easy than to display a progress bar or other status notifications in the application. How would you run a job every X minutes? Are you using an external scheduler (unix-cron)? Yes, unix cron. Example entry in the cron: 0,15,30,45 * * * * /www/sites/mysite/scripts/send_newsletter.py it runs the script every 15 minutes. Ksenia. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Spawning a worker thread from the controller?
Hi Ben, I didn't look at it in great detail -- how do you handle passing status between the pylons threads and the worker threads? This seems to be where I'm having difficulty. thx m On Mon, May 07, 2007 at 07:16:46PM -, ben adam wrote: did you look into one of the suggestions posted earlier i.e. threadpool ? (works for me). ben adam On May 7, 10:10 am, Matt Billenstein [EMAIL PROTECTED] wrote: Thanks Ksenia, that's probably the better way to go from what I'm learning here. m On Mon, May 07, 2007 at 12:07:16PM +0200, Ksenia Marasanova wrote: Op 4-mei-2007, om 22:53 heeft Matt het volgende geschreven: Hi, I have a pylons app which I want to spawn a long-running process from. While this process is running (might be an hour or so), I want the page that kicked off this process to just display status (maybe referesh ever 30 seconds or so)... Does anyone have some pointers on how this should be done or what packages I should be looking into? Just wanted to point to a possible alternative way: I usually save jobs like this into the database. The cron job checks every X minutes for a jobs and runs it if needed, changing status to X% completed. It's pretty easy than to display a progress bar or other status notifications in the application. Ksenia. -- Matt Billenstein [EMAIL PROTECTED]://www.vazor.com/ -- Matt Billenstein [EMAIL PROTECTED] http://www.vazor.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Spawning a worker thread from the controller?
By the way: http://docs.python.org/lib/module-Queue.html Best Regards, -jj On 5/7/07, Matt Billenstein [EMAIL PROTECTED] wrote: Hi Ben, I didn't look at it in great detail -- how do you handle passing status between the pylons threads and the worker threads? This seems to be where I'm having difficulty. thx m On Mon, May 07, 2007 at 07:16:46PM -, ben adam wrote: did you look into one of the suggestions posted earlier i.e. threadpool ? (works for me). ben adam On May 7, 10:10 am, Matt Billenstein [EMAIL PROTECTED] wrote: Thanks Ksenia, that's probably the better way to go from what I'm learning here. m On Mon, May 07, 2007 at 12:07:16PM +0200, Ksenia Marasanova wrote: Op 4-mei-2007, om 22:53 heeft Matt het volgende geschreven: Hi, I have a pylons app which I want to spawn a long-running process from. While this process is running (might be an hour or so), I want the page that kicked off this process to just display status (maybe referesh ever 30 seconds or so)... Does anyone have some pointers on how this should be done or what packages I should be looking into? Just wanted to point to a possible alternative way: I usually save jobs like this into the database. The cron job checks every X minutes for a jobs and runs it if needed, changing status to X% completed. It's pretty easy than to display a progress bar or other status notifications in the application. Ksenia. -- Matt Billenstein [EMAIL PROTECTED]://www.vazor.com/ -- Matt Billenstein [EMAIL PROTECTED] http://www.vazor.com/ -- http://jjinux.blogspot.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Spawning a worker thread from the controller?
If you set self.status, it *shadows* any value that was set on the class. Hence, you'll have a value set on the class *and* a value set on the instance. That's not going to help you if you're looking for the status in the wrong instance ;) If ClassName.status didn't work, I'm worried that's because you're using another instance of ClassName that has it's own version of status that's also shadowing ClassName.status. Best Regards, -jj On 5/4/07, Matt Billenstein [EMAIL PROTECTED] wrote: Yeah, I was doing self.status = blah in the worker thread... The thread the controller is in doesn't seem to ever get this value... Using Class.status = blah doesn't seem to help much either... thx m On Fri, May 04, 2007 at 05:59:15PM -0700, Shannon -jj Behrens wrote: On 5/4/07, Matt Billenstein [EMAIL PROTECTED] wrote: On Fri, May 04, 2007 at 05:27:42PM -0700, Shannon -jj Behrens wrote: On 5/4/07, Matt [EMAIL PROTECTED] wrote: Hi, I have a pylons app which I want to spawn a long-running process from. While this process is running (might be an hour or so), I want the page that kicked off this process to just display status (maybe referesh ever 30 seconds or so)... Does anyone have some pointers on how this should be done or what packages I should be looking into? I've done this before, but not in Pylons. Spawning a long running process or thread is fine. Watch out for Paste because I think Ian put in some code to watch for runaway threads--it's probably fine. Use a timer and AJAX to have the Web page poll the server to see if the work is done. (You could use Comet, but polling is probably fine. Ignore this statement if you don't know what I'm talking about.) In the best case, you can use a progress bar on the page. Just work your way through it; I'm sure you'll be fine. Any recommendations on passing status between the worker thread and the controller when it's polling? I tried just using class variables, but they don't seem to get updated between the threads... How were you updating the class variable? ClassName.variable = foo? If you do self.variable = foo, you're shadowing the class variable. By the way, naturally, watch out for race conditions. Another option is to use the queue module and have the worker thread push messages onto the queue, and the controller can read messages from the queue. I tend to prefer message queues over explicit lock management. Best Regards, -jj -- http://jjinux.blogspot.com/ -- Matt Billenstein [EMAIL PROTECTED] http://www.vazor.com/ -- http://jjinux.blogspot.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Spawning a worker thread from the controller?
You're not using mod_python or something, are you? I was assuming you were using paster serve which uses a single interpreter instance. By the way, if the class member thing is throwing you off, you can also try using a module-level variable. Best Regards, -jj On 5/4/07, Matt Billenstein [EMAIL PROTECTED] wrote: BTW, I'll add that I can do this in a simple standalone python script, spawn a thread and have both that thread and the main thread stuff data into a list or something... thx m On Fri, May 04, 2007 at 06:17:29PM -0700, Matt Billenstein wrote: Yeah, I was doing self.status = blah in the worker thread... The thread the controller is in doesn't seem to ever get this value... Using Class.status = blah doesn't seem to help much either... thx m On Fri, May 04, 2007 at 05:59:15PM -0700, Shannon -jj Behrens wrote: On 5/4/07, Matt Billenstein [EMAIL PROTECTED] wrote: On Fri, May 04, 2007 at 05:27:42PM -0700, Shannon -jj Behrens wrote: On 5/4/07, Matt [EMAIL PROTECTED] wrote: Hi, I have a pylons app which I want to spawn a long-running process from. While this process is running (might be an hour or so), I want the page that kicked off this process to just display status (maybe referesh ever 30 seconds or so)... Does anyone have some pointers on how this should be done or what packages I should be looking into? I've done this before, but not in Pylons. Spawning a long running process or thread is fine. Watch out for Paste because I think Ian put in some code to watch for runaway threads--it's probably fine. Use a timer and AJAX to have the Web page poll the server to see if the work is done. (You could use Comet, but polling is probably fine. Ignore this statement if you don't know what I'm talking about.) In the best case, you can use a progress bar on the page. Just work your way through it; I'm sure you'll be fine. Any recommendations on passing status between the worker thread and the controller when it's polling? I tried just using class variables, but they don't seem to get updated between the threads... How were you updating the class variable? ClassName.variable = foo? If you do self.variable = foo, you're shadowing the class variable. By the way, naturally, watch out for race conditions. Another option is to use the queue module and have the worker thread push messages onto the queue, and the controller can read messages from the queue. I tend to prefer message queues over explicit lock management. Best Regards, -jj --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Spawning a worker thread from the controller?
Hi, I have a pylons app which I want to spawn a long-running process from. While this process is running (might be an hour or so), I want the page that kicked off this process to just display status (maybe referesh ever 30 seconds or so)... Does anyone have some pointers on how this should be done or what packages I should be looking into? thx Matt --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Spawning a worker thread from the controller?
On 5/4/07, Matt [EMAIL PROTECTED] wrote: Hi, I have a pylons app which I want to spawn a long-running process from. While this process is running (might be an hour or so), I want the page that kicked off this process to just display status (maybe referesh ever 30 seconds or so)... Does anyone have some pointers on how this should be done or what packages I should be looking into? I've done this before, but not in Pylons. Spawning a long running process or thread is fine. Watch out for Paste because I think Ian put in some code to watch for runaway threads--it's probably fine. Use a timer and AJAX to have the Web page poll the server to see if the work is done. (You could use Comet, but polling is probably fine. Ignore this statement if you don't know what I'm talking about.) In the best case, you can use a progress bar on the page. Just work your way through it; I'm sure you'll be fine. Best Regards, -jj -- http://jjinux.blogspot.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Spawning a worker thread from the controller?
On Fri, May 04, 2007 at 05:27:42PM -0700, Shannon -jj Behrens wrote: On 5/4/07, Matt [EMAIL PROTECTED] wrote: Hi, I have a pylons app which I want to spawn a long-running process from. While this process is running (might be an hour or so), I want the page that kicked off this process to just display status (maybe referesh ever 30 seconds or so)... Does anyone have some pointers on how this should be done or what packages I should be looking into? I've done this before, but not in Pylons. Spawning a long running process or thread is fine. Watch out for Paste because I think Ian put in some code to watch for runaway threads--it's probably fine. Use a timer and AJAX to have the Web page poll the server to see if the work is done. (You could use Comet, but polling is probably fine. Ignore this statement if you don't know what I'm talking about.) In the best case, you can use a progress bar on the page. Just work your way through it; I'm sure you'll be fine. Any recommendations on passing status between the worker thread and the controller when it's polling? I tried just using class variables, but they don't seem to get updated between the threads... thx m -- Matt Billenstein [EMAIL PROTECTED] http://www.vazor.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Spawning a worker thread from the controller?
On 5/4/07, Matt Billenstein [EMAIL PROTECTED] wrote: On Fri, May 04, 2007 at 05:27:42PM -0700, Shannon -jj Behrens wrote: On 5/4/07, Matt [EMAIL PROTECTED] wrote: Hi, I have a pylons app which I want to spawn a long-running process from. While this process is running (might be an hour or so), I want the page that kicked off this process to just display status (maybe referesh ever 30 seconds or so)... Does anyone have some pointers on how this should be done or what packages I should be looking into? I've done this before, but not in Pylons. Spawning a long running process or thread is fine. Watch out for Paste because I think Ian put in some code to watch for runaway threads--it's probably fine. Use a timer and AJAX to have the Web page poll the server to see if the work is done. (You could use Comet, but polling is probably fine. Ignore this statement if you don't know what I'm talking about.) In the best case, you can use a progress bar on the page. Just work your way through it; I'm sure you'll be fine. Any recommendations on passing status between the worker thread and the controller when it's polling? I tried just using class variables, but they don't seem to get updated between the threads... How were you updating the class variable? ClassName.variable = foo? If you do self.variable = foo, you're shadowing the class variable. By the way, naturally, watch out for race conditions. Another option is to use the queue module and have the worker thread push messages onto the queue, and the controller can read messages from the queue. I tend to prefer message queues over explicit lock management. Best Regards, -jj -- http://jjinux.blogspot.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Spawning a worker thread from the controller?
On May 5, 2007, at 2:59 AM, Shannon -jj Behrens wrote: On 5/4/07, Matt Billenstein [EMAIL PROTECTED] wrote: On Fri, May 04, 2007 at 05:27:42PM -0700, Shannon -jj Behrens wrote: On 5/4/07, Matt [EMAIL PROTECTED] wrote: Hi, I have a pylons app which I want to spawn a long-running process from. While this process is running (might be an hour or so), I want the page that kicked off this process to just display status (maybe referesh ever 30 seconds or so)... Does anyone have some pointers on how this should be done or what packages I should be looking into? I've done this before, but not in Pylons. Spawning a long running process or thread is fine. Watch out for Paste because I think Ian put in some code to watch for runaway threads--it's probably fine. Use a timer and AJAX to have the Web page poll the server to see if the work is done. (You could use Comet, but polling is probably fine. Ignore this statement if you don't know what I'm talking about.) In the best case, you can use a progress bar on the page. Just work your way through it; I'm sure you'll be fine. Any recommendations on passing status between the worker thread and the controller when it's polling? I tried just using class variables, but they don't seem to get updated between the threads... How were you updating the class variable? ClassName.variable = foo? If you do self.variable = foo, you're shadowing the class variable. By the way, naturally, watch out for race conditions. Another option is to use the queue module and have the worker thread push messages onto the queue, and the controller can read messages from the queue. I tend to prefer message queues over explicit lock management. I've found this threadpool implementation useful for a similar task: http://tinyurl.com/yw6xc9 Alberto --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Spawning a worker thread from the controller?
Yeah, I was doing self.status = blah in the worker thread... The thread the controller is in doesn't seem to ever get this value... Using Class.status = blah doesn't seem to help much either... thx m On Fri, May 04, 2007 at 05:59:15PM -0700, Shannon -jj Behrens wrote: On 5/4/07, Matt Billenstein [EMAIL PROTECTED] wrote: On Fri, May 04, 2007 at 05:27:42PM -0700, Shannon -jj Behrens wrote: On 5/4/07, Matt [EMAIL PROTECTED] wrote: Hi, I have a pylons app which I want to spawn a long-running process from. While this process is running (might be an hour or so), I want the page that kicked off this process to just display status (maybe referesh ever 30 seconds or so)... Does anyone have some pointers on how this should be done or what packages I should be looking into? I've done this before, but not in Pylons. Spawning a long running process or thread is fine. Watch out for Paste because I think Ian put in some code to watch for runaway threads--it's probably fine. Use a timer and AJAX to have the Web page poll the server to see if the work is done. (You could use Comet, but polling is probably fine. Ignore this statement if you don't know what I'm talking about.) In the best case, you can use a progress bar on the page. Just work your way through it; I'm sure you'll be fine. Any recommendations on passing status between the worker thread and the controller when it's polling? I tried just using class variables, but they don't seem to get updated between the threads... How were you updating the class variable? ClassName.variable = foo? If you do self.variable = foo, you're shadowing the class variable. By the way, naturally, watch out for race conditions. Another option is to use the queue module and have the worker thread push messages onto the queue, and the controller can read messages from the queue. I tend to prefer message queues over explicit lock management. Best Regards, -jj -- http://jjinux.blogspot.com/ -- Matt Billenstein [EMAIL PROTECTED] http://www.vazor.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Spawning a worker thread from the controller?
BTW, I'll add that I can do this in a simple standalone python script, spawn a thread and have both that thread and the main thread stuff data into a list or something... thx m On Fri, May 04, 2007 at 06:17:29PM -0700, Matt Billenstein wrote: Yeah, I was doing self.status = blah in the worker thread... The thread the controller is in doesn't seem to ever get this value... Using Class.status = blah doesn't seem to help much either... thx m On Fri, May 04, 2007 at 05:59:15PM -0700, Shannon -jj Behrens wrote: On 5/4/07, Matt Billenstein [EMAIL PROTECTED] wrote: On Fri, May 04, 2007 at 05:27:42PM -0700, Shannon -jj Behrens wrote: On 5/4/07, Matt [EMAIL PROTECTED] wrote: Hi, I have a pylons app which I want to spawn a long-running process from. While this process is running (might be an hour or so), I want the page that kicked off this process to just display status (maybe referesh ever 30 seconds or so)... Does anyone have some pointers on how this should be done or what packages I should be looking into? I've done this before, but not in Pylons. Spawning a long running process or thread is fine. Watch out for Paste because I think Ian put in some code to watch for runaway threads--it's probably fine. Use a timer and AJAX to have the Web page poll the server to see if the work is done. (You could use Comet, but polling is probably fine. Ignore this statement if you don't know what I'm talking about.) In the best case, you can use a progress bar on the page. Just work your way through it; I'm sure you'll be fine. Any recommendations on passing status between the worker thread and the controller when it's polling? I tried just using class variables, but they don't seem to get updated between the threads... How were you updating the class variable? ClassName.variable = foo? If you do self.variable = foo, you're shadowing the class variable. By the way, naturally, watch out for race conditions. Another option is to use the queue module and have the worker thread push messages onto the queue, and the controller can read messages from the queue. I tend to prefer message queues over explicit lock management. Best Regards, -jj -- http://jjinux.blogspot.com/ -- Matt Billenstein [EMAIL PROTECTED] http://www.vazor.com/ -- Matt Billenstein [EMAIL PROTECTED] http://www.vazor.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
