>> Ian Bicking wrote : >> A process >>generally allows you much more control -- you can kill it off, run it as >>a different user, you don't get caught by the GIL, etc. This works well >>with queuing jobs in a database. I was just looking at remoteD lately as >>well (http://www.neurokode.com/Projects/remoted.php), and it looks like >>a very easy way to do interprocess queuing without a database.
More direct url to remoted : http://remoted.neurokode.com/ PyLinda (http://www-users.cs.york.ac.uk/~aw/pylinda/) looks more experimental but has similar, perhaps more flexible, capabilities. Conceptually I think tuplespaces like PyLinda would seem to be a good fit with web-applications. A cgi-program that coordinated with a TupleSpace to hand in requests and pull-out results could be backed by any number of servers/processes that did the actual work of processing requests and responses. Such a setup would suffer from increased latency caused by all the coordination but could scale well and be very hard to kill. -----Original Message----- From: Ian Bicking [mailto:[EMAIL PROTECTED] Sent: Friday, May 21, 2004 4:16 PM To: John Dickinson Cc: [EMAIL PROTECTED]; Ian Sparks Subject: Re: [Webware-discuss] Task-Queue using Webware... John Dickinson wrote: > Some code is below. It is used to take a bunch (1000s) of files and > merge them into one big file. I hope it helps. The hardest part of > writing it was debugging the worker thread. If everythong doesn't go > just right, then you end up with a bunch of threads out there in > memory taking up cycles. > > I do have one (related) problem with doing this. In another (similar) > application, I tried to use this technique to prevent server > timeouts. The problem was that my worker thread was trying to eval a > huge (~40MB) dictionary. This eval seemed to block everything in the > entire Webware process. Now if I read the Python documentation > correctly, that happens because Python's threads are simulated in the > interpreter and each operation is viewed as atomic. The eval was > taking a long time and would not allow any other thread in the > process to do it's thing. Is this correct? If not, what is happening? > If so, what other way can I do this? Python uses OS threads, so other threads continue operating. However, there is a global interpreter lock (GIL) which is locked for many operations, and you may have encountered that. It usually doesn't cause a problem, but I guess it depends on what you are doing. Imports and some code evaluation, for instance, lock the entire process. Generally I would suggest putting these workers in their own processes -- if not in deployment, at least for development (to make them easier to debug -- i.e., make it usable as a command-line program). A process generally allows you much more control -- you can kill it off, run it as a different user, you don't get caught by the GIL, etc. This works well with queuing jobs in a database. I was just looking at remoteD lately as well (http://www.neurokode.com/Projects/remoted.php), and it looks like a very easy way to do interprocess queuing without a database. -- Ian Bicking / [EMAIL PROTECTED] / http://blog.ianbicking.org ------------------------------------------------------- This SF.Net email is sponsored by: Oracle 10g Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we'll give you the exam FREE. http://ads.osdn.com/?ad_id149&alloc_id66&op=click _______________________________________________ Webware-discuss mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/webware-discuss
