Re: [Zope] How to create an ansynchronous method
David Pratt wrote: really interesting.I think it would be a really good thing to see fire and forget with a process you create as a series of steps that can also be queued in one or more queues. Well, I think ZAsync would be great for building and managing the queues, with Stepper steps being used to process them... I also think there is some interesting possibility for something like a Zope instance Stepper where you could trigger a generic daemon to run a thread to process asynchronous tasks (steps) without a ZEO requirement. Huh? That made no sense. Spawning new ZServer threads is evil, I like the fact that Stepper keeps it simple and connects to a ZEO server. It means Stepper can be run from any normal Zope client build and doesn't need to do anything funky. The asynchronous jobs could be queued for long running tasks outside of zope or timed to inject the step requests into zope for the maintenance of a zope instance (in the same vein as ClockServer) but in a single product. I think a combination of Stepper and ZAsync could meet your needs, when you hit specific problems which you need help with, let me or Gary know :-) cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] How to create an ansynchronous method
Ron Bickers wrote: I have no idea. Maybe it'll do what you want, but I don't understand what it really does just from the description. The work I needed to do was external to Zope anyway (reading data from a MySQL database, building PDFs with Reportlab, sending email), so it's actually better that I'm outside of it. The open source Reportlab library right? From what I've seen of their closed source stuff, it's way fast enough to do this all on the fly... cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] How to create an ansynchronous method
Hi David, Just a note in passing to say that what you're really after in this case is Gary Poster's ZASync. Gary's talked to me about getting Stepper (which is basically batch processing triggered via cron using a ZEO connection) to do the work of processing ZAsync's queue, which would be cool, but I don't know if either of us has the need/time to make that happen... cheers, Chris gary's talked with me about doing the David Pratt wrote: Hi Ron. I found the following trying to follow up a bit on what you have suggested. I believe it is similar to what you are doing from your explanation. It may be out of date. I have not attempted to daemonize a process to date so it would be great if you could look at this and comment since I need something to work with. http://mail.python.org/pipermail/python-list/2001-February/030814.html As far as interacting with Zope, I have done something similar to build a site remotely from another server but setup https and sent credentials in the urls. I wonder if there is a way to inject them into Zope another way since the daemon is on the machine. I think the ClockServer injects requests into Zope. I believe it is something similar since credentials still need to be in url to execute something but requests are not exposed to the web doing this. Regards, David On Wednesday, September 14, 2005, at 12:11 AM, Ron Bickers wrote: On Tue September 13 2005 08:16 pm, David Pratt wrote: How is it that you send a signal to the long running process? The long-running process writes a pid file and waits for a SIGUSR1 signal (using Python's signal module). A small External Method, called when the user submits the form, reads the pid file and sends the signal to the process. Is the long running process cronned to look for a record in the database or is this starting another server of some type. It waits for a SIGUSR1 signal from the website, otherwise it's idle. That way it can begin processing immediately but doesn't have to do any periodic checking. I do, however, have a cron job that starts it every 15 minutes in case it dies. The process knows if it's already running, so it's safe to just start it regularly. What I need is something like what you are suggesting - wakes up when there is work in the hopper and chugs along until it is done and then goes to sleep (sort of the way a printing queue works). At the same time it would be great if it was something that had a small RAM footprint or ran without consuming any more than X mbs. The part that waits for the signal is very small, but it loads the Reportlab modules and reads a bunch of data to do the work, so it can get large at times. If it were to just call an external program that dies after doing its work, you could easily keep memory usage low. The other problem I have with this is that it needs to do work in Zope itself since the final docs end up as objects. I have just found Chris Wither's product called Stepper. I am not sure it this is for this type of situation or more for cronned maintenance. I have no idea. Maybe it'll do what you want, but I don't understand what it really does just from the description. The work I needed to do was external to Zope anyway (reading data from a MySQL database, building PDFs with Reportlab, sending email), so it's actually better that I'm outside of it. When I have to run things in Zope from outside, I run curl with a URL of a Python Script that does the work. It's a hack, but I've never had a problem with it. I'm not sure how else you would work in Zope from a process outside of Zope. -- Ron ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] How to create an ansynchronous method
On Fri September 23 2005 04:31 pm, Chris Withers wrote: Ron Bickers wrote: I have no idea. Maybe it'll do what you want, but I don't understand what it really does just from the description. The work I needed to do was external to Zope anyway (reading data from a MySQL database, building PDFs with Reportlab, sending email), so it's actually better that I'm outside of it. The open source Reportlab library right? From what I've seen of their closed source stuff, it's way fast enough to do this all on the fly... What I was doing wasn't fast enough; there were plenty of complaints from customers. But to be fair, it wasn't Reportlab that was slow. The process had to retrieve hundreds (sometimes over a thousand) of name/address records from a database, calculate the length of the longest line in each when printed with a given font, pass that information to Reportlab to format a page that would print each address centered on a label (not center-aligned, but left-aligned with the whole thing in the center of the label), attach the resulting PDF to an email and send it on. I suspect the slowest part was going through each of the hundreds of records and calculating the size, but it really didn't matter; the whole thing took too long to do on the fly. Reportlab is *nice*, BTW. -- Ron ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] How to create an ansynchronous method
Hi Chris. I downloaded ZASync and did a bit of reading and comparing between your Stepper product as well. ZASync at present relies on an older version of Twisted which is now I think into the 2 series so maybe later the products will come together this way. I think Stepper is really interesting.I think it would be a really good thing to see fire and forget with a process you create as a series of steps that can also be queued in one or more queues. I also think there is some interesting possibility for something like a Zope instance Stepper where you could trigger a generic daemon to run a thread to process asynchronous tasks (steps) without a ZEO requirement. The asynchronous jobs could be queued for long running tasks outside of zope or timed to inject the step requests into zope for the maintenance of a zope instance (in the same vein as ClockServer) but in a single product. Regards, David On Friday, September 23, 2005, at 05:33 PM, Chris Withers wrote: Hi David, Just a note in passing to say that what you're really after in this case is Gary Poster's ZASync. Gary's talked to me about getting Stepper (which is basically batch processing triggered via cron using a ZEO connection) to do the work of processing ZAsync's queue, which would be cool, but I don't know if either of us has the need/time to make that happen... ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] How to create an ansynchronous method
Hi. I have a workflow that is triggered by a file upload and the processing of the file can be minutes of processing depending upon the size of the file uploaded. I am concerned about number of threads available to serve zope so I believe this is a good candidate for an asynchronous process. I am looking for some type of outline to do this. Currently a tool does the work that is triggered by workflow script. My hope is to have this process run and send the user an email to advise when the process has completed instead of the user waiting for a response or potentially timing out waiting for one. What steps could I take to make this an ansynchronous process? Also wanted to confirm whether an asynchronous process would free the thread and how one can determine whether a thread has been released. Many thanks. Regards, David ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] How to create an ansynchronous method
On Tue September 13 2005 02:52 pm, David Pratt wrote: Hi. I have a workflow that is triggered by a file upload and the processing of the file can be minutes of processing depending upon the size of the file uploaded. I am concerned about number of threads available to serve zope so I believe this is a good candidate for an asynchronous process. I am looking for some type of outline to do this. Currently a tool does the work that is triggered by workflow script. My hope is to have this process run and send the user an email to advise when the process has completed instead of the user waiting for a response or potentially timing out waiting for one. What steps could I take to make this an ansynchronous process? I had to do something like this when processing a lot of data to create PDF documents to send via email. The time to do so was too long for them to sit and wait, so I created a separate process to do the job. I don't know if this is the best way, but it wasn't very difficult and it has been working without problems for many months. When a user requests the document, I add a record of needed information to a MySQL table (the queue), send a signal to the long-running process (described next) and immediately return a thank you page. A separate-from-Zope long-running Python process waits for a signal, reads the queue table, does what it needs to do, empties the processed items from the queue, and goes idle. It can handle things like getting a signal while it's processing a queue and catching up occassionally if it missed a signal (for whatever reason). Hope this helps. -- Ron ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] How to create an ansynchronous method
Hi Ron. Many thanks for your reply. This is also a long running document processing challenge, there can be 100's (or even thousands) of records which is why the time problem. The code it uses in my tool in the CMF. How is it that you send a signal to the long running process? Is the long running process cronned to look for a record in the database or is this starting another server of some type. What I need is something like what you are suggesting - wakes up when there is work in the hopper and chugs along until it is done and then goes to sleep (sort of the way a printing queue works). At the same time it would be great if it was something that had a small RAM footprint or ran without consuming any more than X mbs. The other problem I have with this is that it needs to do work in Zope itself since the final docs end up as objects. I have just found Chris Wither's product called Stepper. I am not sure it this is for this type of situation or more for cronned maintenance. I want to be able to initiate the process right away but asynchronously from the main zope threads. Regards, David On Tuesday, September 13, 2005, at 06:05 PM, Ron Bickers wrote: On Tue September 13 2005 02:52 pm, David Pratt wrote: Hi. I have a workflow that is triggered by a file upload and the processing of the file can be minutes of processing depending upon the size of the file uploaded. I am concerned about number of threads available to serve zope so I believe this is a good candidate for an asynchronous process. I am looking for some type of outline to do this. Currently a tool does the work that is triggered by workflow script. My hope is to have this process run and send the user an email to advise when the process has completed instead of the user waiting for a response or potentially timing out waiting for one. What steps could I take to make this an ansynchronous process? I had to do something like this when processing a lot of data to create PDF documents to send via email. The time to do so was too long for them to sit and wait, so I created a separate process to do the job. I don't know if this is the best way, but it wasn't very difficult and it has been working without problems for many months. When a user requests the document, I add a record of needed information to a MySQL table (the queue), send a signal to the long-running process (described next) and immediately return a thank you page. A separate-from-Zope long-running Python process waits for a signal, reads the queue table, does what it needs to do, empties the processed items from the queue, and goes idle. It can handle things like getting a signal while it's processing a queue and catching up occassionally if it missed a signal (for whatever reason). Hope this helps. -- Ron ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] How to create an ansynchronous method
On Tue September 13 2005 08:16 pm, David Pratt wrote: How is it that you send a signal to the long running process? The long-running process writes a pid file and waits for a SIGUSR1 signal (using Python's signal module). A small External Method, called when the user submits the form, reads the pid file and sends the signal to the process. Is the long running process cronned to look for a record in the database or is this starting another server of some type. It waits for a SIGUSR1 signal from the website, otherwise it's idle. That way it can begin processing immediately but doesn't have to do any periodic checking. I do, however, have a cron job that starts it every 15 minutes in case it dies. The process knows if it's already running, so it's safe to just start it regularly. What I need is something like what you are suggesting - wakes up when there is work in the hopper and chugs along until it is done and then goes to sleep (sort of the way a printing queue works). At the same time it would be great if it was something that had a small RAM footprint or ran without consuming any more than X mbs. The part that waits for the signal is very small, but it loads the Reportlab modules and reads a bunch of data to do the work, so it can get large at times. If it were to just call an external program that dies after doing its work, you could easily keep memory usage low. The other problem I have with this is that it needs to do work in Zope itself since the final docs end up as objects. I have just found Chris Wither's product called Stepper. I am not sure it this is for this type of situation or more for cronned maintenance. I have no idea. Maybe it'll do what you want, but I don't understand what it really does just from the description. The work I needed to do was external to Zope anyway (reading data from a MySQL database, building PDFs with Reportlab, sending email), so it's actually better that I'm outside of it. When I have to run things in Zope from outside, I run curl with a URL of a Python Script that does the work. It's a hack, but I've never had a problem with it. I'm not sure how else you would work in Zope from a process outside of Zope. -- Ron ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] How to create an ansynchronous method
Hi Ron. I found the following trying to follow up a bit on what you have suggested. I believe it is similar to what you are doing from your explanation. It may be out of date. I have not attempted to daemonize a process to date so it would be great if you could look at this and comment since I need something to work with. http://mail.python.org/pipermail/python-list/2001-February/030814.html As far as interacting with Zope, I have done something similar to build a site remotely from another server but setup https and sent credentials in the urls. I wonder if there is a way to inject them into Zope another way since the daemon is on the machine. I think the ClockServer injects requests into Zope. I believe it is something similar since credentials still need to be in url to execute something but requests are not exposed to the web doing this. Regards, David On Wednesday, September 14, 2005, at 12:11 AM, Ron Bickers wrote: On Tue September 13 2005 08:16 pm, David Pratt wrote: How is it that you send a signal to the long running process? The long-running process writes a pid file and waits for a SIGUSR1 signal (using Python's signal module). A small External Method, called when the user submits the form, reads the pid file and sends the signal to the process. Is the long running process cronned to look for a record in the database or is this starting another server of some type. It waits for a SIGUSR1 signal from the website, otherwise it's idle. That way it can begin processing immediately but doesn't have to do any periodic checking. I do, however, have a cron job that starts it every 15 minutes in case it dies. The process knows if it's already running, so it's safe to just start it regularly. What I need is something like what you are suggesting - wakes up when there is work in the hopper and chugs along until it is done and then goes to sleep (sort of the way a printing queue works). At the same time it would be great if it was something that had a small RAM footprint or ran without consuming any more than X mbs. The part that waits for the signal is very small, but it loads the Reportlab modules and reads a bunch of data to do the work, so it can get large at times. If it were to just call an external program that dies after doing its work, you could easily keep memory usage low. The other problem I have with this is that it needs to do work in Zope itself since the final docs end up as objects. I have just found Chris Wither's product called Stepper. I am not sure it this is for this type of situation or more for cronned maintenance. I have no idea. Maybe it'll do what you want, but I don't understand what it really does just from the description. The work I needed to do was external to Zope anyway (reading data from a MySQL database, building PDFs with Reportlab, sending email), so it's actually better that I'm outside of it. When I have to run things in Zope from outside, I run curl with a URL of a Python Script that does the work. It's a hack, but I've never had a problem with it. I'm not sure how else you would work in Zope from a process outside of Zope. -- Ron ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
