[Zope] dealing with scripts that take too long
Hi there, we're experiencing problems with certain maintenance scripts, which just take too long to complete, so that the browser resets the connection and Zope aborts the transaction. Short of splitting the scripts up into smaller pieces and running them individually (which would be a pain), what can we do? I was thinking we could send data back to the browser, but I cannot figure out a way to do this from a TTW Python script. How can I send data immediately, not only when I 'return printed' after all the processing is done. NPH or so, I believe this was called with plain CGIs. Also, I would be interested in how other people approach this problem. `zopectl run` may be an alternative, but we'd rather not require filesystem access. -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^.*|tr * mailto:; [EMAIL PROTECTED] invalid/expired pgp (sub)keys? use subkeys.pgp.net as keyserver! spamtraps: [EMAIL PROTECTED] arrogance on the part of the meritorious is even more offensive to us than the arrogance of those without merit: for merit itself is offensive. -- friedrich nietzsche signature.asc Description: Digital signature (GPG/PGP) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] dealing with scripts that take too long
snip we're experiencing problems with certain maintenance scripts, which just take too long to complete, so that the browser resets the connection and Zope aborts the transaction. Short of splitting the scripts up into smaller pieces and running them individually (which would be a pain), what can we do? I was thinking we could send data back to the browser, but I cannot figure out a way to do this from a TTW Python script. How can I send data immediately, not only when I 'return printed' after all the processing is done. NPH or so, I believe this was called with plain CGIs. Also, I would be interested in how other people approach this problem. `zopectl run` may be an alternative, but we'd rather not require filesystem access. /snip We are running a *nix environment, and use cron with script files that use either perl LWP or CURL to make http requests that invoke the zope methods/scripts. LWP, CURL (and others) give you the ability to control the timeout parameters. hth Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] dealing with scripts that take too long
On Jan 25, 2006, at 5:17 PM, martin f krafft wrote: we're experiencing problems with certain maintenance scripts, which just take too long to complete, so that the browser resets the connection and Zope aborts the transaction. If these are maintenance scripts that are kicked off manually by admins (as opposed to things that can be automated with cron) then maybe you can use a technique similar to one we use here. For certain long running actions that our users need, we use the Zope Scheduler product. The clicks a button on a UI form, the form then adds the script to the schedule queue as a one shot action. The scheduler clock is actually running on an entirely separate Zope instance on a different machine, so that machines sole duty is to handle these async requests. The script, once completed updates its status, so revisiting the initial form will let the user know the status of their request. (we also give them a view into the schedule queue so if things are taking too long at least they know where they stand.) The Zope server that processes the scheduler queue is set up differently, since it is never has to handle user input. It only has a single thread, and the cache size is larger than normal (the stock zope config is for four threads, and each thread has its own cache pool.) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] dealing with scripts that take too long
martin f krafft schrieb: Hi there, ... I was thinking we could send data back to the browser, but I cannot figure out a way to do this from a TTW Python script. How can I send data immediately, not only when I 'return printed' after all the processing is done. NPH or so, I believe this was called with plain CGIs. Actually it does not have anything to do with NPH, but thats another story. You can just write via context.REQUEST.RESPONSE.write(somestring) When you start doing this, you are switching to streaming mode and anything you return from that script is discarded. This is especially important to know when exceptions occur - you wont see them unless you take precaution try: nasty_things() except Exception,x: response.write(Error: %r\n % x) # or something raise x # dont forget to reraise! Or you look into the error_log object. Also, I would be interested in how other people approach this problem. `zopectl run` may be an alternative, but we'd rather not require filesystem access. zopectl run is actually fine for maintenance. Its also easier to avoid running the same script 100x the same time - as it can happen with HTTP requests. A simple cron job, some locking, report via email ... and you are done. HTH Tino ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] dealing with scripts that take too long
On 25 Jan 2006, at 22:17, martin f krafft wrote: we're experiencing problems with certain maintenance scripts, which just take too long to complete, so that the browser resets the connection and Zope aborts the transaction. I am assuming you use broken browsers that will time out, like IE? Use a more suitable browser like Firefox, those don't time out by default. jens ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
