Re: [Python-Dev] Pythonic concurrency - cooperative MT
On 10/1/05, Antoine [EMAIL PROTECTED] wrote: like this with their deferred objects, no? I figure they would need to do something like this too. I will have to check.) A Deferred object is just the abstraction of a callback - or, rather, two callbacks: one for success and one for failure. Twisted is architected around an event loop, which calls your code back when a registered event happens (for example when an operation is finished, or when some data arrives on the wire). Compared to generators, it is a different way of expressing cooperative multi-threading. So, the question is, in Twisted, if I want to defer on an operation that is going to block, say I'm making a call to run a database query that I'm expecting will take much time, and want to yield (defer) for other events to be processed while the query is executed, how do I do that? As far as I remember the Twisted docs I read a long time ago did not provide a solution for that. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
On 10/3/05, Martin Blais [EMAIL PROTECTED] wrote:
On 10/1/05, Antoine [EMAIL PROTECTED] wrote:
like this with their deferred objects, no? I figure they would
need to do something like this too. I will have to check.)
A Deferred object is just the abstraction of a callback - or, rather, two
callbacks: one for success and one for failure. Twisted is architected
around an event loop, which calls your code back when a registered event
happens (for example when an operation is finished, or when some data
arrives on the wire). Compared to generators, it is a different way of
expressing cooperative multi-threading.
So, the question is, in Twisted, if I want to defer on an operation
that is going to block, say I'm making a call to run a database query
that I'm expecting will take much time, and want to yield (defer)
for other events to be processed while the query is executed, how do I
do that? As far as I remember the Twisted docs I read a long time ago
did not provide a solution for that.
Deferreds don't make blocking code non-blocking; they're just a way to
make it nicer to write non-blocking code. There are utilities in
Twisted for wrapping a blocking function call in a thread and having
the result returned in a Deferred, though (see deferToThread). There
is also a lightweight and complete wrapper for DB-API2 database
modules in twisted.enterprise.adbapi, which does the threading
interaction for you.
So, since this then exposes a non-blocking API, you can do stuff like
d = pool.runQuery('SELECT User_ID FROM Users')
d.addCallback(gotDBData)
d2 = ldapfoo.getUser('bob')
d2.addCallback(gotLDAPData)
And both the database call and the ldap request will be worked on concurrently.
--
Twisted | Christopher Armstrong: International Man of Twistery
Radix|-- http://radix.twistedmatrix.com
| Release Manager, Twisted Project
\\\V/// |-- http://twistedmatrix.com
|o O||
wvw-+
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
On 10/2/05, Christopher Armstrong [EMAIL PROTECTED] wrote:
On 10/3/05, Martin Blais [EMAIL PROTECTED] wrote:
On 10/1/05, Antoine [EMAIL PROTECTED] wrote:
like this with their deferred objects, no? I figure they would
need to do something like this too. I will have to check.)
A Deferred object is just the abstraction of a callback - or, rather, two
callbacks: one for success and one for failure. Twisted is architected
around an event loop, which calls your code back when a registered event
happens (for example when an operation is finished, or when some data
arrives on the wire). Compared to generators, it is a different way of
expressing cooperative multi-threading.
So, the question is, in Twisted, if I want to defer on an operation
that is going to block, say I'm making a call to run a database query
that I'm expecting will take much time, and want to yield (defer)
for other events to be processed while the query is executed, how do I
do that? As far as I remember the Twisted docs I read a long time ago
did not provide a solution for that.
Deferreds don't make blocking code non-blocking; they're just a way to
make it nicer to write non-blocking code. There are utilities in
Twisted for wrapping a blocking function call in a thread and having
the result returned in a Deferred, though (see deferToThread). There
is also a lightweight and complete wrapper for DB-API2 database
modules in twisted.enterprise.adbapi, which does the threading
interaction for you.
So, since this then exposes a non-blocking API, you can do stuff like
d = pool.runQuery('SELECT User_ID FROM Users')
d.addCallback(gotDBData)
d2 = ldapfoo.getUser('bob')
d2.addCallback(gotLDAPData)
And both the database call and the ldap request will be worked on
concurrently.
Very nice!
However, if you're using a thread to do just that, it's just using a
part of what threads were designed for: it's really just using the
low-level kernel knowledge about resource access and when they become
ready to wait on the resource, since you're not going to run much
actual code in the thread itself (apart from setting up to do the
blocking call and returning its value).
Now, if we had something in the language that allows us to do
something like that--make the most important potentially blocking
calls asynchronously-- we could implement a more complete scheduler
that could really leverage generators to create a more interesting
concurrency solution with less overhead. For example, imagine that
some class of generators are used as tasks, like we were discussing
before. When you would call the special yield_read() call (a
variation on e.g. os.read() call), there is an implicit yield that
allows other generators which are ready to run until the data is
available, without the overhead of
1. context switching to the helper threads and back;
2. synchronization for communcation with the helper threads (I assume
threads would not be created dynamically, for efficiency. I imagine
there is a pool of helpers waiting to do the async call jobs, and
communication with them to dispatch the call jobs does not come for
free (i.e. locking)).
We really don't need threads at all to do that (at least for the
common blocking calls), just some low-level support for building a
scheduler. Using threads to do that has a cost, it is more or less a
kludge, in that context (but we have nothing better for now).
cheers,
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
Hi. I hear a confusion that is annoying me a bit in some of the discussions on concurrency, and I thought I'd flush my thoughts here to help me clarify some of that stuff, because some people on the list appear to discuss generators as a concurrency scheme, and as far as I know (and please correct me if I'm wrong) they really are not adressing that at all (full explanation below). Before I go on, I must say that I am not in any way an authority on concurrent programming, I'm just a guy who happens to have done a fair amount of threaded programming, so if any of the smart people on the list notice something completely stupid and off the mark that I might be saying here, please feel free to bang on it with the thousand-pound hammer of your hacker-fu and put me to shame (I love to learn). As far as I understand, generators are just a convenient way to program apparently independent control flows (which are not the same as concurrent control flows) in a constrained, structured way, a way that is more powerful than what is allowed by using a stack. By giving up using the stack concept as a fast way to allocate local function variables, it becomes possible to exit and enter chunks of code multiple times, at specific points, within an automatically restored local context (i.e. the local variables, stored on the heap). Generators make it more convenient to do just that: enter and re-enter some code that is expressed as if it would be running in a single execution flow (with explicit points of exit/re-entry, yields). The full monty version of that, is what you get when you write assembly code (*memories of adolescent assembly programming on the C=64 abound here now*): you can JMP anywhere anytime, and a chunk of code (a function) can be reentered anywhere anytime as well, maybe even reentered somewhere else than where it left off. The price to pay for this is one of complexity: in assembly you have to manage restoring the local context yourself (i.e in assembly code this just means restoring the values of some registers which are assumed set and used by the code, like the local variables of a function), and there is no clear grouping of the local scope that is saved. Generators give you that for free: they automatically organize all that local context as belonging to the generator object, and it expresses clear points of exit/re-entry with the yield calls. They are really just a fancy goto, with some convenient assumptions about how control should flow. This happens to be good enough for simplifying a whole class of problems and I suppose the Python and Ruby communities are all learning to love them and use them more and more. (I think the more fundamental consequence of generators is to raise questions about the definition of what a function is: if I have a single chunk of code in which different parts uses two disjoint sets of variables, and it can be entered via a few entry/exit points, is it really one or two or multiple functions? What if different parts share some of the local scope only? Where does the function begin and end? And more importantly, is there a more complex yet stull manageable abstraction that would allow even more flexible control flow than generators allow, straddling the boundaries of what a function is?) You could easily implement something very similar to generators by encapsulating the local scope explicitly in the form of a class, with instance attributes, and having an normal method step() that would be careful about saving state in the object's attributes everytime it returns and restoring state from those attributes everytime it gets called. This is what iterators do. Whenever you want to schedule your object to be running, you call the step() method. So just in that sense generators really aren't all that exciting or new. The main problem that generators solve is that they make this save/restore mechanism automatic, thus allowing you to write a single flow of execution as a normal function with explicit exit points (yield). It's much nicer having that in the language than having to write code that can be restored (especially when you have to write a loop with complex conditions/flow which must run and return only one iteration every time they become runnable). Therefore, as far as I understand it, generators themselves DO NOT implement any form of concurrency. I feel that where generators and concurrency come together is often met with confusion in the discussions I see about them, but maybe that's just me. I see two aspects that allow generators to participate in the elaboration of a concurrency scheme: 1. The convenience of expression of a single execution flow (with explicit interruption points) makes it easy to implement pseudo-concurrency IF AND ONLY IF you consider a generator as an independent unit of control flow (i.e. a task). Whether those generators can run asynchronously is yet undefined and depends on who calls them. 2. In a more
Re: [Python-Dev] Pythonic concurrency - cooperative MT
Hi Martin, [snip] The confusion stems from the fact that two issues are mixed up in this discussion thread: - improving concurrency schemes to make it easier to write well-behaving applications with independent parallel flows - improving concurrency schemes to improve performance when there are several hardware threads available The respective solutions to these problems do not necessarily go hand in hand. To implement that explicitly, you would need an asynchronous version of all the functions that may block on resources (e.g. file open, socket write, etc.), in order to be able to insert a yield statement at that point, after the async call, and there should be a way for the scheduler to check if the resource is ready to be able to put your generator back in the runnable queue. You can also use a helper thread and signal the scheduling loop when some action in the helper thread has finished. It is an elegant solution because it helps you keep a small generic scheduling loop instead of putting select()-like calls in it. (this is how I've implemented timers in my little cooperative multi-threading system, for example) (A question comes to mind here: Twisted must be doing something like this with their deferred objects, no? I figure they would need to do something like this too. I will have to check.) A Deferred object is just the abstraction of a callback - or, rather, two callbacks: one for success and one for failure. Twisted is architected around an event loop, which calls your code back when a registered event happens (for example when an operation is finished, or when some data arrives on the wire). Compared to generators, it is a different way of expressing cooperative multi-threading. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
On Saturday 01 October 2005 22:50, Martin Blais wrote: ... because some people on the list appear to discuss generators as a concurrency scheme, and as far as I know they really are not addressing that at all. Our project started in the context of dealing with the task of a naturally concurrent environment. Specifically the task is that of dealing with large numbers of concurrent connections to a server. As a result, when I've mentioned concurrency it's been due to coming from that viewpoint. In the past I have worked with systems essentially structured in a similar way to Twisted for this kind of problem, but decided against that style for our current project. (Note some people misunderstand my opinions here due to a badly phrased lightning talk ~16 months ago at Europython 2004 - I think twisted is very much best of breed in python for what it does. I just think there //might// be a nicer way. (might :) ) Since I now work in an RD dept I wondered what would happen if instead of the basic approach that underlies systems like twisted what would happen if you took a much more CSP-like approach to building such systems, but using generators rather than threads or explicit state machines. A specific goal was to try and make code simpler for people to work with - with the aim actually of simplifying maintenance as the main by-product. I hadn't heard of anyone trying this approach then and hypothesised it *might* achieve that goal. As a result from day 1 it became clear that where an event based system would normally use a reactor/proactor based approach, that you replace that with a scheduler that repeatedly calls a next method on objects given to it to schedule. In terms of concurrency that is clearly a co-operative multitasking system in the same way as a simplistic event based system is. (Both get more complex in reality when you can't avoid blocking forcing the use of threads for some tasks) So when you say this: explicitly frame the discussion in the context of a single process cooperative scheme that runs on a single processor (at any one time). This is spot on. However, any generator can be farmed off and run in a thread. Any communications you did with the generator can be wrapped via Queues then - forming a controlled bridge between the threads. Similarly we're currently looking at using non-blocking pipes and pickling to communicate with generators running in a forked environment. As a result if you write your code as generators it can migrate to a threaded or process based environment, and scale across multiple processes (and hence processors) if tools to perform this migration are put in place. We're a little way off doing that, but this looks to be highly reasonable. As far as I understand, generators are just a convenient way to They give you code objects that can do a return and continue later. This isn't really the same as the ability to just do a goto into random points in a function. You can only go back to the point the generator yielded at (unless someone has a perverse trick :-). You could easily implement something very similar to generators by encapsulating the local scope explicitly in the form of a class, with instance attributes, and having an normal method step() that would be careful about saving state in the object's attributes everytime it returns and restoring state from those attributes everytime it gets called. For a more explicit version of this we have a (deliberately naive) C++ version of generators our core concurrency system. Mechanism is here: http://tinyurl.com/7gaol , example use here: http://tinyurl.com/bgwro That does precisely that. (except we use a next() method there :-) Therefore, as far as I understand it, generators themselves DO NOT implement any form of concurrency. By themselves, they don't. They can be used to deal with concurrency though. 2. In a more advanced framework/language, perhaps some generators could be considered to always be possible to run asynchronously, ruled by a system of true concurrency with some kind of scheduling algorithm that oversees that process. Whether this has been implemented by many is still a mystery to me, This is what we do. Our tutorial we've given to trainees (one of whom have had very little experience of even programming) was able to pick up our approach quickly due to our tutorial. This requires them to implement a mini-version of the framework, which might actually aid the discussion here since it very clearly shows the core of our system. (nb it is however a simplified version) I previously posted a link to it, which is here: http://kamaelia.sourceforge.net/MiniAxon/ but I can see how a low-level library that provides asynchronously running execution vehicles for each CPU could be used to manage and run a pool of shared generator objects in a way that is better (for a specific application) than the relatively uninformed scheduling provided
Re: [Python-Dev] Pythonic concurrency - cooperative MT
(C) That scheduler is non-preemptive. A single greedy generator can starve all the others. Instead of looking at this as a problem, you could look at it as a feature. Since generators can't be switched at arbitrary places, the programmer has to define his/her synchronization points explicitly. Let me contrast this: - in preemptive MT, threads can switch at every moment by default, and you have to explicitly wrap some pieces of code in critical sections (or other primitives) to protect the semantics of your program; every time you forget a critical section, you introduce a bug - in cooperative MT, threads can only switch where the programmer explicitly allows it; every time you forget a synchronization point, you give your program worse performance (latency), but you *don't* introduce a bug So you have a scheme where you seek optimal performance (latency) but you take the risk of a huge number of difficult bugs, and you have a scheme where good performance needs more careful coding *but* the paradigm avoids difficult bugs /by construction/. By the way, the cooperative MT is exactly the Twisted approach, except implemented in a different manner (event loop instead of explicit cooperative tasks). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
(C) That scheduler is non-preemptive. A single greedy generator can starve all the others. Antoine Instead of looking at this as a problem, you could look at it Antoine as a feature. Apple looked at it as a feature for years. Not anymore. wink Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
Le vendredi 30 septembre 2005 à 08:32 -0500, [EMAIL PROTECTED] a écrit : (C) That scheduler is non-preemptive. A single greedy generator can starve all the others. Antoine Instead of looking at this as a problem, you could look at it Antoine as a feature. Apple looked at it as a feature for years. Not anymore. wink You are missing the context: - at the OS level, it is not good to have cooperative MT because any badly-written app (there are lots of ;-)) can cause lack of responsiveness for the whole machine - at the application level, cooperative MT can be good to enforce robust semantics without disrupting other apps The latter is what we are discussing AFAIK: a different concurrency scheme inside a single application - not accross the whole OS or desktop. Actually, I think different concurrency schemes must be chosen and mixed depending on the semantics. For example, when you have a networked GUI app, it can be good to have the network in one preemptive thread (using e.g. Twisted) and the GUI in another preemptive thread (using GTK, wx...). This is because GUI and network have different latency characteristics and requirements. (of course, you can replace preemptive thread with process in the above description) So whatever innovatice concurrency scheme Python may come out, it should still be mixable with more traditional concurrency schemes, because required properties vary wildly even inside a single app. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
On 9/30/05, Antoine Pitrou [EMAIL PROTECTED] wrote: (C) That scheduler is non-preemptive. A single greedy generator can starve all the others. Instead of looking at this as a problem, you could look at it as a feature. Since generators can't be switched at arbitrary places, the programmer has to define his/her synchronization points explicitly. I use this approach extensively, using tasks which are defined using generators. The scheduler I developed for this can be viewed here: http://metaplay.dyndns.org:82/svn/nanothreads/nanothreads.py Synchronization using yield statements is an important feature, as I use these cooperative threads for game development. These threads are _very_ useful for games, as they have very low overhead, and allow the programmer to exercise more control over their execution. Sw. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
On 9/30/05, Antoine Pitrou [EMAIL PROTECTED] wrote: Le vendredi 30 septembre 2005 à 08:32 -0500, [EMAIL PROTECTED] a écrit : (C) That scheduler is non-preemptive. A single greedy generator can starve all the others. Antoine Instead of looking at this as a problem, you could look at it Antoine as a feature. Apple looked at it as a feature for years. Not anymore. wink You are missing the context: - at the OS level, it is not good to have cooperative MT because any badly-written app (there are lots of ;-)) can cause lack of responsiveness for the whole machine - at the application level, cooperative MT can be good to enforce robust semantics without disrupting other apps The latter is what we are discussing AFAIK: a different concurrency scheme inside a single application - not accross the whole OS or desktop. I like this analysis. Actually, I think different concurrency schemes must be chosen and mixed depending on the semantics. For example, when you have a networked GUI app, it can be good to have the network in one preemptive thread (using e.g. Twisted) and the GUI in another preemptive thread (using GTK, wx...). This is because GUI and network have different latency characteristics and requirements. (of course, you can replace preemptive thread with process in the above description) I'm skeptical of this. I think a traditional GUI+network app can combine the network and UI tasks in one process -- usually they interact somewhat and that's where the bugs show up. Also note that this is not an application area where MP is very interesting -- most of the time you're blocked waiting for the user or for a socket (or likely both :) so a single CPU is all you need. I've never heard someone complain that the GIL is in the way for these types of apps. So whatever innovatice concurrency scheme Python may come out, it should still be mixable with more traditional concurrency schemes, because required properties vary wildly even inside a single app. I don't think you've proved that yet. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
Hi, I've never heard someone complain that the GIL is in the way for these types of apps. I've never said so either. I was just saying that it can be useful to mix cooperative threading and preemptive threading in the same app, i.e. have different domains of cooperative threading which are preemptively scheduled by the OS. That has nothing to do with the GIL, I think (but I don't know much in Python internals). For instance the mix between Twisted and GUI event loops is a very common topic on the twisted mailing-list, because people have trouble getting it right inside a single system thread. The Twisted people have started advocating something called the ThreadedSelectReactor, which I haven't looked at but whose name suggests that some operations are performed in a helper system thread. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
On Fri, 30 Sep 2005 17:26:27 +0200, Antoine Pitrou [EMAIL PROTECTED] wrote: Hi, I've never heard someone complain that the GIL is in the way for these types of apps. I've never said so either. I was just saying that it can be useful to mix cooperative threading and preemptive threading in the same app, i.e. have different domains of cooperative threading which are preemptively scheduled by the OS. That has nothing to do with the GIL, I think (but I don't know much in Python internals). For instance the mix between Twisted and GUI event loops is a very common topic on the twisted mailing-list, because people have trouble getting it right inside a single system thread. The Twisted people have started advocating something called the ThreadedSelectReactor, which I haven't looked at but whose name suggests that some operations are performed in a helper system thread. Advocating might be putting it too strongly :) Experimenting with describes the current state of things most accurately. The problem it aims to solve is integration with cooperative threading systems which don't work very well. An example of such a loop is the wx event loop. Whenever a modal dialog is displayed or a menu is activated, wx's loop stops cooerating. The only way we've found thus far to avoid this is to run wx in a separate thread, so even when it stops cooperating, network code can still get scheduled. Jp ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
Hi Jp, Le vendredi 30 septembre 2005 à 12:20 -0400, Jp Calderone a écrit : Advocating might be putting it too strongly :) Experimenting with describes the current state of things most accurately. Ok :) The problem it aims to solve is integration with cooperative threading systems which don't work very well. An example of such a loop is the wx event loop. Whenever a modal dialog is displayed or a menu is activated, wx's loop stops cooerating. This specific problem hides the more general problem, which is that GUI and network activities have different typical latencies. As I explained on the twisted ML, a GUI needs very good response times to feel friendly (typically below 20 or even 10 ms.), whereas some network protocols have non-trivial calculations which can go above 100 ms. Moreover, you don't want a sudden flood of network events to block events in the GUI. This is why even without considering wx's specificities, it is still useful to keep GUI and network activities in separate threads. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
I was just saying that it can be useful to mix cooperative threading and preemptive threading in the same app, i.e. have different domains of cooperative threading which are preemptively scheduled by the OS. That has nothing to do with the GIL, I think (but I don't know much in Python internals). This is one of the interesting questions I'd like to answer (but which probably requires some kind of mathematician to prove one way or another): Does a complete concurrency solution require both a preemptive task management system and a cooperative one? OS Processes are generally quite limited in number (although I understand that BeOS was designed to create tons of lightweight processes, which changed that particular picture). In Java, you can usually create several hundred threads before it craps out. Simulations and games could easily have thousands or even hundreds of thousands of independent simulation units. If there are limits to the number of concurrency drivers (threads, etc.), then that would suggest that at some point you have to move to a cooperative system. Which would suggest that a complete concurrency solution might require both. That wouldn't be my ideal. My ideal would be a single solution that would scale up to large numbers of concurrency units, and that wouldn't require the programmer to remember to explicitly yield control. Whether my ideal is possible is a question I'd like to answer. Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED] Contains electronic books: Thinking in Java 3e Thinking in C++ 2e Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
On Fri, 2005-09-30 at 18:33 +0200, Antoine Pitrou wrote: Hi Jp, Le vendredi 30 septembre 2005 à 12:20 -0400, Jp Calderone a écrit : Advocating might be putting it too strongly :) Experimenting with describes the current state of things most accurately. Ok :) The problem it aims to solve is integration with cooperative threading systems which don't work very well. An example of such a loop is the wx event loop. Whenever a modal dialog is displayed or a menu is activated, wx's loop stops cooerating. That is wx's problem. Try PyGTK some day; I hear it's really good! ;-) This specific problem hides the more general problem, which is that GUI and network activities have different typical latencies. As I explained on the twisted ML, a GUI needs very good response times to feel friendly (typically below 20 or even 10 ms.), whereas some network protocols have non-trivial calculations which can go above 100 ms. With PyGTK a typical solution for this is to use a generator function executing an idle function, which would make the non-trivial calculations, but yield control back to the main loop periodically, in order to process GUI events. For example, see the last code block in http://www.async.com.br/faq/pygtk/index.py?req=showfile=faq23.020.htp Moreover, you don't want a sudden flood of network events to block events in the GUI. Process one event at a time, after each event give back control to the main loop, and give low priority to socket IO events. Problem solved. This is why even without considering wx's specificities, it is still useful to keep GUI and network activities in separate threads. You are considering a scenario that seldom happens to design a solution that is far too complicated for most cases. Regards. -- Gustavo J. A. M. Carneiro [EMAIL PROTECTED] [EMAIL PROTECTED] The universe is always one step beyond logic ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
