Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
As others have reported: if you have a gtk reactor and a wx reactor to support, in addition to a regular Win32 reactor (e.g. PythonWin), your approach still won't work. They all do GetMessage behind the scenes (sp?), yet you would run them in multiple threads, so they get their own messages. Plus they are not thread-safe. I'm not sure what your point is here. Let me see if I can clarify things, at least from my side. I suggest it is possible to implement a PerfectReactor. I define this PerfectReactor as a singleton top-level reactor which runs in the main thread of a Python application. All incoming events would be serialized to this single reactor, which would dispatch them to their appropriate listeners. When asked to monitor disparate event systems, the PerfectReactor would act as a concentrator of multiple other reactors, with various ugly things going on hidden in the implementation. The goal would be that modular libraries of event listeners could be written in an agnostic manner, simply requesting to listen to whatever events they need to, and the main application wouldn't have to know in advance what reactors to configure or how to configure them. You seem to assert it is not possible to implement such a PerfectReactor, and list counter-examples. So far so good. If you're saying it is impossible for any implementation to simultaneously support more than one of the Win32, gtk, and wx reactors: fine, this hypothetical PerfectReactor can't do the impossible, but that's not an argument against a PerfectReactor. If it *is* possible to support more than one of these simultaneously through precise configuration, I assert that a PerfectReactor could support that configuration too, and the goal would be to have it automatically configure itself. You point out the gtk reactor and the wx reactor must be run in the main thread, because they run their own message pumps. What I had in mind was that, when you asked to start listening to Win32 events, the Win32 reactor would tell the PerfectReactor I must be run in the main thread, and it would (re-) configure that to be the main thread's listener; the notification mechanism for other threads saying there's a new event in your queue would switch to using Win32 messages. If the gtk and wx reactors had the same restriction, then whichever was first introduced to the PerfectReactor first would win, and introducing subsequent reactors would throw an exception. (Again, can't do the impossible.) However, if gtk and wx expose enough of the plumbing of their message pumps, this central Win32 message pump could identify messages intended for the gtk and wx reactor message pumps and inject them. The gtk and wx reactors wouldn't call GetMessage, but they could call the rest of their message loops. Therefore they could all be run in the main thread. (And yes, behind the scenes is correct.) if the PerfectReactor is to support Tcl/Tk, it needs to run the Tcl even loop in the Tcl thread, which is normally the main thread (in Tkinter, it's the thread where you originally created the tkapp object). OTOH, PerfectReactor may also need to run some Win32 event loop in the same thread. Again, I'm not sure of your point. If you're saying it is impossible to run multithreaded Tcl and any message pump (Win32, gtk, wx) in the same application, then fine, PerfectReactor cannot do the impossible. If it is only possible to do with clever configuration, then I maintain a PerfectReactor could self-configure to support this configuration at runtime. Cheers, /larry/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
[EMAIL PROTECTED] wrote: When you boil it down, Twisted's event loop is just a notification for a connection was made, some data was received on a connection, a connection was closed, and a few APIs to listen or initiate different kinds of connections, start timed calls, and communicate with threads. All of the platform details of how data is delivered to the connections are abstracted away. How do you propose we would make a less specific event mechanism? But that is exactly the problem I have with Twisted. For HTTP it creates its own set of notifications instead of structuring the code similar to SocketServer (UDP and TCP), BaseHTTPServer, SimpleHTTPServer etc which are well understood in the python community and e.g. used by medusa and asyncore. Having to completely restructure one's own code is a high price. Giving control away into a big framework that calls my own code for not so easy to understand reasons (for a twisted noob) does not give me a warm feeling. It's o.k. for complex applications like web servers but for small networking applications I'd like to have a chance to understand what's going on. Asyncore is so simple that it's easy to follow when I let it do the select() for me. That said, I conclude that the protocol implementations are superb but unfortunately to tightly coupled to the Twisted philosophy, sitting in the middle, trying to orchestrate instead of being easy to integrate. Joachim ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 16 Feb, 06:30 pm, [EMAIL PROTECTED] wrote: I suggest it is possible to implement a PerfectReactor. I don't think this constitutes a sufficient existence proof. Perhaps you could write a prototype? There are a bunch of existing reactors you could either import or copy/paste from to bootstrap it, so you wouldn't necessarily need to do many of the platform-specific hacks it has already implemented for those loops. Personally I am highly skeptical that this is possible, and I didn't find any of your arguments convincing, but I would be ecstatic if it worked; it isn't *fun* maintaining 12 or so subtly incompatible implementations of the same interface :). I also think this discussion would be more appropriate to continue on the Twisted list, as we have veered pretty far afield of stdlib enhancements and are now talking about multiplexing method implementations, but I'll follow it wherever it continues. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
I was the one on the Stackless list who last September or so proposed the idea of monkeypatching and I'm including that idea in my presentation for PyCon. See my early rough draft at http://www.stackless.com/pipermail/stackless/2007-February/002212.html which contains many details about using Stackless, though none on the Stackless implementation. (A lot on how to tie things together.) So people know, I am an applications programmer and not a systems programmer. Things like OS-specific event mechanisms annoy and frustrate me. If I could do away with hardware and still write useful programs I would. I have tried 3 times to learn Twisted. The first time I found and reported various problems and successes. See emails at http://www.twistedmatrix.com/pipermail/twisted-python/2003-June/thread.html The second time was to investigate a way to report upload progress: http://twistedmatrix.com/trac/ticket/288 and the third was to compare Allegra and Twisted http://www.dalkescientific.com/writings/diary/archive/2006/08/28/levels_of_abstraction.html In all three cases I've found it hard to use Twisted because the code didn't do as I expected it to do and when something went wrong I got results which were hard to interpret. I believe others have similar problems and is one reason Twisted is considered to be a big, complicated, inseparable hairy mess. I find the Stackless code also hard to understand. Eg, I don't know where the watchdog code is for the run() command. It uses several layers of macros and I haven't been able get it straight in my head. However, so far I've not run into strange errors in Stackless that I have in Twisted. I find the normal Python code relatively easy to understand. Stackless only provides threadlets. It does no I/O. Richard Tew developed a stacklesssocket module which emulates the API for the stdlib socket module. I tweaked it a bit and showed that by doing the monkeypatch import stacklesssocket import sys sys.modules[socket] = stacklesssocket then code like urllib.urlopen became Stackless compatible. Eg, in my PyCon talk draft I show something like import slib # must monkeypatch before any other module imports socket slib.use_monkeypatch() import urllib2 import time import hashlib def fetch_and_reverse(host): t1 = time.time() s = urllib2.urlopen(http://+host+/;).read()[::-1] dt = time.time() - t1 digest = hashlib.md5(s).hexdigest() print hash of %r/ = %s in %.2f s % (host, digest, dt) slib.main_tasklet(fetch_and_reverse)(www.python.org) slib.main_tasklet(fetch_and_reverse)(docs.python.org) slib.main_tasklet(fetch_and_reverse)(planet.python.org) slib.run_all() where the three fetches occur in parallel. The choice of asyncore is, I think, done because 1) it prevents needing an external dependency, 2) asyncore is smaller and easier to understand than Twisted, and 3) it was for demo/proof of concept purposes. While tempting to improve that module I know that Twisted has already gone though all the platform-specific crap and I don't want to go through it again myself. I don't want to write a reactor to deal with GTK, and one for OS X, and one for ... Another reason I think Twisted is considered tangled-up Deep Magic, only for Wizards Of The Highest Order is because it's infused with event-based processing. I've done a lot of SAX processing and I can say that few people think that way or want to go through the process of learning how. Compare, for example, the following f = urllib2.urlopen(http://example.com/;) for i, line in enumerate(f): print (%06d % i), repr(line) with the normal equivalent in Twisted or other async-based system. Yet by using the Stackless socket monkeypatch, this same code works in an async framework. And the underlying libraries have a much larger developer base than Twisted. Want NNTP? import nntplib Want POP3? import poplib Plenty of documentation about them too. On the Stackless mailing list I have proposed someone work on a talk for EuroPython titled Stackless and Twisted. Andrew Francis has been looking into how to do that. All the earlier quotes were lifted from glyph. Here's another: When you boil it down, Twisted's event loop is just a notification for a connection was made, some data was received on a connection, a connection was closed, and a few APIs to listen or initiate different kinds of connections, start timed calls, and communicate with threads. All of the platform details of how data is delivered to the connections are abstracted away.. How do you propose we would make a less specific event mechanism? What would I need to do to extract this Twisted core so I could replace asyncore? I know at minimum I need twisted.internet and twisted.python (the latter for logging) and twisted.persisted for styles.Ephemeral. But I say this hesitantly recalling the frustrations I had in dealing with a connection error in Twisted, described in the aforementioned
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/14/07, Greg Ewing [EMAIL PROTECTED] wrote: Thomas Wouters wrote: *I* don't like the idea of something in the Python installation deciding which reactor to use. I wouldn't mind if some way were provided of changing the reactor if you want. I'd just like to see a long term goal of making it unnecessary as far as possible. In any case, your idea requires a lot of changes in external, non-Python code -- PyGTK simply exposes the GTK mainloop, which couldn't care less about Python's idea of a perfect event reactor model. On unix at least, I don't think it should be necessary to change gtk, only pygtk. If it can find out the file descriptor of the connection to the X server, it can plug that into the reactor, and then call gtk_main_iteration_do() whenever something comes in on it. A similar strategy ought to work for any X11-based toolkit that exposes a function to perform one iteration of its main loop. Mileage on other platforms may vary. The PerfectReactor can be added later, all current reactors aliased to it, and no one would have to change a single line of code. Sure. The other side to all this is the client side, i.e. the code that installs event callbacks. At the moment there's no clear direction to take, so everyone makes their own choice -- some use asyncore, some use Twisted, some use the gtk event loop, some roll their own, etc. There is no single PerfectReactor. There are several use cases where you need to wait on 1 different event systems, which guarantees at least two OS threads (and two event loops). In general it's nice to have a single Python event loop (the reactor) to act on said threads (e.g. something just sitting on a mutex waiting for messages) but waiting for IO to occur should *probably* happen on one or more ancillary threads -- one per event system (e.g. select, GTK, WaitForMultipleEvents, etc.) -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, Feb 15, 2007 at 09:19:30AM -0500, Jean-Paul Calderone wrote:
That feels like 6 layers too many, given that
_logrun(selectable, _drdw, selectable, method, dict)
return context.call({ILogContext: newCtx}, func, *args, **kw)
return self.currentContext().callWithContext(ctx, func, *args, **kw)
return func(*args, **kw)
getattr(selectable, method())
klass(number, string)
are all generic calls.
I know function calls are expensive in Python, and method calls even more
so... but I still don't understand this issue. Twisted's call stack is too
deep? It is fair to say it is deep, I guess, but I don't see how that is a
problem. If it is, I don't see how it is specific to this discussion.
It's hard to debug the resulting problem. Which level of the *12*
levels in the stack trace is responsible for a bug? Which of the *6*
generic calls is calling the wrong thing because a handler was set up
incorrectly or the wrong object provided? The code is so 'meta' that
it becomes effectively undebuggable.
--amk
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, 15 Feb 2007 10:46:05 -0500, A.M. Kuchling [EMAIL PROTECTED] wrote:
On Thu, Feb 15, 2007 at 09:19:30AM -0500, Jean-Paul Calderone wrote:
That feels like 6 layers too many, given that
_logrun(selectable, _drdw, selectable, method, dict)
return context.call({ILogContext: newCtx}, func, *args, **kw)
return self.currentContext().callWithContext(ctx, func, *args, **kw)
return func(*args, **kw)
getattr(selectable, method())
klass(number, string)
are all generic calls.
I know function calls are expensive in Python, and method calls even more
so... but I still don't understand this issue. Twisted's call stack is too
deep? It is fair to say it is deep, I guess, but I don't see how that is a
problem. If it is, I don't see how it is specific to this discussion.
It's hard to debug the resulting problem. Which level of the *12*
levels in the stack trace is responsible for a bug? Which of the *6*
generic calls is calling the wrong thing because a handler was set up
incorrectly or the wrong object provided? The code is so 'meta' that
it becomes effectively undebuggable.
I've debugged plenty of Twisted applications. So it's not undebuggable. :)
Application code tends to reside at the bottom of the call stack, so Python's
traceback order puts it right where you're looking, which makes it easy to
find. For any bug which causes something to be set up incorrectly and only
later manifests as a traceback, I would posit that whether there is 1 frame or
12, you aren't going to get anything useful out of the traceback. Standard
practice here is just to make exception text informative, I think, but this is
another general problem with Python programs and event loops, not one specific
to either Twisted itself or the particular APIs Twisted exposes.
As a personal anecdote, I've never once had to chase a bug through any of the
6 generic calls singled out. I can't think of a case where I've helped any
one else who had to do this, either. That part of Twisted is very old, it is
_very_ close to bug-free, and application code doesn't have very much control
over it at all. Perhaps in order to avoid scaring people, there should be a
way to elide frames from a traceback (I don't much like this myself, I worry
about it going wrong and chopping out too much information, but I have heard
other people ask for it)?
Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Bob Ippolito wrote: There is no single PerfectReactor. There are several use cases where you need to wait on 1 different event systems, which guarantees at least two OS threads (and two event loops). In general it's nice to have a single Python event loop (the reactor) to act on said threads (e.g. something just sitting on a mutex waiting for messages) but waiting for IO to occur should *probably* happen on one or more ancillary threads -- one per event system (e.g. select, GTK, WaitForMultipleEvents, etc.) Why couldn't PerfectReactor be a reactor for other reactors? A sort of concentrator for these multiple event systems and multiple threads. You ask to listen to sockets, so it instantiates a singleton PerfectReactor which instantiates a select() reactor and listens to it directly in a single-threaded manner. If you then ask to listen to Win32 messages, the PerfectReactor instantiates a GetMessage() reactor. Then, realizing it has two event systems, it spawns a thread for each child reactor with a listener that serializes the incoming events into the PerfectReactor's queue. Bingo, your application doesn't need to be written thread-safe, PerfectReactor is platform-agnostic, and you don't have to know in advance all the event types you might ever listen to. Sorry if this is a dumb question (or if I'm mangling the terminology), /larry/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Martin v. Löwis wrote: Now, for these generalized event loops, it may not be possible anymore to combine all event sources into a single blocking call. Right, that's why my proposal assumed that each disparate event source would need its own thread. Ah, threads :-( It turns out that you need to invoke GetMessage in the context of the thread in which the window was created. In a different thread, you won't get any messages. Oof! I'm embarrassed to have forgotten that. But that's not a fatal problem. It means that on Windows the PerfectReactor must service the blocking GetMessage loop, and these other threads notify the PerfectReactor of new events by sending a message. (Either that, or, it could poll GetMessage and its incoming event queue without ever blocking. But that is obviously suboptimal.) I think I've done this sort of thing before, in fact. Of course, in the absence of any windows, the Windows PerfectReactor could fall back to a mutex. Abstract this inside PerfectReactor and its event sources wouldn't notice the difference. Integrating with threads might be a solution in some cases, and a problem in others. You can't assume it is a universal solution. Universal? Yeah, I doubt it too. But perhaps it would be good enough for nearly all cases. In the cases where it wasn't, it could throw an I can't listen to that type of event right now exception, forcing you to fall back to preconfiguring your central reactor by hand. Anyway, like many folks I'm hoping this whole conversation results in establishing basic standard duck-typed conventions--what other languages might call interfaces--for event senders and receivers. In which case this non-universal threaded approach would simply be one of several to choose from. I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable. Thanks for the thoughtful reply, /larry/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Ah, threads :-( It turns out that you need to invoke GetMessage in the context of the thread in which the window was created. In a different thread, you won't get any messages. I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable. I've tried something similar on Linux, with gtk an wx. You can run the gtk main loop in its own thread, but because gtk is not thread safe, you have to grab a mutex everytime you run gtk code outside the thread the mainloop is running in. So you have to surround your calls to the gtk api with calls to gtk.threads_enter and gtk.threads_leave. Except for callbacks of course, because they are executed in the main thread... Doable, but not fun. The same goes for wx. Then all hell breaks loose when you try to use both gtk and wx at the same time. That's because on Linux, the wx main loop calls the gtk mainloop behind the scenes. As far as I know, that problem can not be solved from python. So yes that strategy can work, but it's no silver bullet. Cheers, Baptiste ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/15/07, Baptiste Carvello [EMAIL PROTECTED] wrote: Ah, threads :-( It turns out that you need to invoke GetMessage in the context of the thread in which the window was created. In a different thread, you won't get any messages. I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable. I've tried something similar on Linux, with gtk an wx. You can run the gtk main loop in its own thread, but because gtk is not thread safe, you have to grab a mutex everytime you run gtk code outside the thread the mainloop is running in. So you have to surround your calls to the gtk api with calls to gtk.threads_enter and gtk.threads_leave. Except for callbacks of course, because they are executed in the main thread... Doable, but not fun. The same goes for wx. Then all hell breaks loose when you try to use both gtk and wx at the same time. That's because on Linux, the wx main loop calls the gtk mainloop behind the scenes. As far as I know, that problem can not be solved from python. So yes that strategy can work, but it's no silver bullet. And it's worse on Windows and Mac OS X where some GUI API calls *must* happen on a particular thread or they simply don't work. -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Jean-Paul Calderone [EMAIL PROTECTED] wrote: On Thu, 15 Feb 2007 02:36:22 -0700, Andrew Dalke [EMAIL PROTECTED] wrote: [snip] 2) asyncore is smaller and easier to understand than Twisted, While I hear this a lot, applications written with Twisted _are_ shorter and contain less irrelevant noise in the form of boilerplate than the equivalent asyncore programs. This may not mean that Twisted programs are easier to understand, but it is at least an objectively measurable metric. In my experience, the boilerplate is generally incoming and outgoing buffers. If both had better (optional default) implementations, and perhaps a way of saying use the default implementations of handle_close, etc., then much of the boilerplate would vanish. People would likely implement a found_terminator method and be happy. and 3) it was for demo/proof of concept purposes. While tempting to improve that module I know that Twisted has already gone though all the platform-specific crap and I don't want to go through it again myself. I don't want to write a reactor to deal with GTK, and one for OS X, and one for ... Now if we can only figure out a way for everyone to benefit from this without tying too many brains up in knots. :) Whenever I need to deal with these kinds of things (in wxPython specifically), I usually set up a wxTimer to signal asyncore.poll(timeout=0), but I'm lazy, and rarely need significant throughput in my GUI applications. [snip] Yet by using the Stackless socket monkeypatch, this same code works in an async framework. And the underlying libraries have a much larger developer base than Twisted. Want NNTP? import nntplib Want POP3? import poplib Plenty of documentation about them too. This is going to come out pretty harshly, for which I can only apologize in advance, but it bears mention. The quality of protocol implementations in the standard library is bad. As in not good. Twisted's NNTP support is better (even if I do say so myself - despite only having been working on by myself, when I knew almost nothing about Twisted, and having essentially never been touched since). Twisted's POP3 support is fantastically awesome. Next to imaplib, twisted.mail.imap4 is a sparkling diamond. And each of these implements the server end of the protocol as well: you won't find that in the standard library for almost any protocol. Protocol support is hit and miss. NNTP in Python could be better, but that's not an asyncore issue (being that nntplib isn't implemented using asyncore), that's an NNTP in Python could be done better issue. Is it worth someone's time to patch it, or should they just use Twisted? Well, if we start abandoning stdlib modules, because they can always use Twisted, then we may as well just ship Twisted with Python. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Martin v. Löwis wrote: That is insufficient. The gtk main loop has more input sources than just the connection to X: - timers - idle handlers - child handlers - additional file descriptors - a generalzed 'event source' When gtk is not the central event mechanism, there's no need to use the gtk event loop for these things -- you can just use the central event mechanism directly. The pygtk APIs for setting these up can redirect them to the appropriate place, to accommodate existing code that uses the gtk event loop for them. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, 15 Feb 2007 13:55:31 -0800, Josiah Carlson [EMAIL PROTECTED] wrote: Jean-Paul Calderone [EMAIL PROTECTED] wrote: [snip] Now if we can only figure out a way for everyone to benefit from this without tying too many brains up in knots. :) Whenever I need to deal with these kinds of things (in wxPython specifically), I usually set up a wxTimer to signal asyncore.poll(timeout=0), but I'm lazy, and rarely need significant throughput in my GUI applications. And I guess you also don't mind that on OS X this is often noticably broken? :) [snip] Protocol support is hit and miss. NNTP in Python could be better, but that's not an asyncore issue (being that nntplib isn't implemented using asyncore), that's an NNTP in Python could be done better issue. Is it worth someone's time to patch it, or should they just use Twisted? Well, if we start abandoning stdlib modules, because they can always use Twisted, then we may as well just ship Twisted with Python. We could always replace the stdlib modules with thin compatibility layers based on the Twisted protocol implementations. It's trivial to turn an asynchronous API into a synchronous one. I think you are correct in marking this an unrelated issue, though. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Jean-Paul Calderone [EMAIL PROTECTED] wrote: On Thu, 15 Feb 2007 13:55:31 -0800, Josiah Carlson [EMAIL PROTECTED] wrote: Jean-Paul Calderone [EMAIL PROTECTED] wrote: [snip] Now if we can only figure out a way for everyone to benefit from this without tying too many brains up in knots. :) Whenever I need to deal with these kinds of things (in wxPython specifically), I usually set up a wxTimer to signal asyncore.poll(timeout=0), but I'm lazy, and rarely need significant throughput in my GUI applications. And I guess you also don't mind that on OS X this is often noticably broken? :) I don't own a Mac, and so far, of the perhaps dozen or so Mac users of the software that does this, I've heard no reports of it being broken. From what I understand, wxTimers work on all supported platforms (which includes OSX), and if asyncore.poll() is broken on Macs, then someone should file a bug report. If it's asyncore's fault, assign it to me, otherwise someone with Mac experience needs to dig into it. [snip] Protocol support is hit and miss. NNTP in Python could be better, but that's not an asyncore issue (being that nntplib isn't implemented using asyncore), that's an NNTP in Python could be done better issue. Is it worth someone's time to patch it, or should they just use Twisted? Well, if we start abandoning stdlib modules, because they can always use Twisted, then we may as well just ship Twisted with Python. We could always replace the stdlib modules with thin compatibility layers based on the Twisted protocol implementations. It's trivial to turn an asynchronous API into a synchronous one. I think you are correct in marking this an unrelated issue, though. If the twisted folks (or anyone else) want to implement a shim that pretends to be nntplib, it's their business whether calling twisted.internet.monkeypatch.nntplib() does what the name suggests. :) That is to say, I don't believe anyone would be terribly distraught if there was an easy way to use Twisted without drinking the kool-aid. Then again, I do believe that it makes sense to patch the standard library whenever possible - if Twisted has better parsing of nntp, smtp, pop3, imap4, etc. responses, then perhaps we should get the original authors to sign a PSF contributor agreement, and we could translate whatever is better. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Wed, Feb 14, 2007, [EMAIL PROTECTED] wrote: As I said, I don't have time to write the PEPs myself, but I might fix some specific bugs if there were a clear set of issues preventing this from moving forward. Better integration with the standard library would definitely be a big win for both Twisted and Python. Here's where I'm coming from: My first experience with Twisted was excellent: I needed a web server in fifteen minutes to do my PyCon presentation, and it Just Worked. My second experience with Twisted? Well, I didn't really have one. My first experience was Twisted 1.1, and when I tried installing 2.0 on my Mac (about 1.5 years ago), it just didn't work. Combined with the difficulty of using the documentation and the fact that I was in a hurry, I rejected the Twisted solution. (My company needed an FTP server that did a callback every time a file got uploaded -- something that I expect would be very simple for Twisted.) -- Aahz ([EMAIL PROTECTED]) * http://www.pythoncraft.com/ I disrespectfully agree. --SJM ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 02:22 pm, [EMAIL PROTECTED] wrote: On Wed, Feb 14, 2007, [EMAIL PROTECTED] wrote: As I said, I don't have time to write the PEPs myself, but I might fix some specific bugs if there were a clear set of issues preventing this from moving forward. Better integration with the standard library would definitely be a big win for both Twisted and Python. Here's where I'm coming from: My first experience with Twisted was excellent: I needed a web server in fifteen minutes to do my PyCon presentation, and it Just Worked. My second experience with Twisted? Well, I didn't really have one. Thanks for the feedback, Aahz... I'm a little confused, though. Is this just a personal anecdote, or were you trying to make a larger point about Twisted's suitability for the stdlib? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
[EMAIL PROTECTED] schrieb: On 02:20 am, [EMAIL PROTECTED] wrote: If Twisted is designed so that it absolutely *has* to use its own special event mechanism, and everything else needs to be modified to suit its requirements, then it's part of the problem, not part of the solution. I've often heard this complaint, usually of the form that's twisted-specific. The thing is, Twisted isn't specific. Its event mechanism isn't special. In fact it's hard to imagine how it might be made less special than it currently is. Whatever your event dispatch mechanism, *some* code has to be responsible for calling the OS API of select(), WaitForMultipleEvents(), g_main_loop_run(), or whatever. Twisted actually imposes very few requirements for code to participate in this, and was designed from day one specifically to be a generalized mainloop mechanism which would not limit you to one underlying multiplexing implementation, event-dispatch mechanism, or operating system if you used its API. When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more. And they didn't even work too well. The problems I remember were that the win32reactor was limited to a only handful of handles, the wxreactor didn't process events when a wx modal dialog boy was displayed, and so on. Has this changed? Thanks, Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 08:52 pm, [EMAIL PROTECTED] wrote: When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more. Yes. That's intentional. Each of those systems offers its own event loop, and each reactor implements the basic operations in terms of those loops. They all have the same API. Application code does 'from twisted.internet import reactor; reactor.listenTCP', 'reactor.callLater', etc. Only the very top-most level decides which reactor the application will use. And they didn't even work too well. The problems I remember were that the win32reactor was limited to a only handful of handles, the wxreactor didn't process events when a wx modal dialog boy was displayed, and so on. Has this changed? win32eventreactor is limited to 64 handles because WaitForMultipleObjects is limited to 64 handles. wxreactor's event loop is ghastly and actually did pause when a modal dialog box is displayed (that has since been fixed). Process support on win32 now works in the default select reactor as well as the gtk reactor, so win32reactor is mostly a curiosity at this point (useful mainly if you want to implement your own GDI-based GUI, as PyUI did at one point), and its limitations are not as serious for Twisted as a whole. In other words, Twisted exposes the platform limitations in its platform-specific event loop implementations, and only works around them where it's possible to do so without masking platform functionality. For servers, the epoll, poll, and select reactors work just fine. The select reactor does have a maximum of FD_SETSIZE simultaneous sockets as well, but it is very easy to switch reactors if you need something more scalable. For clients, the best GUI toolkit for Twisted applications at this point is GTK, but WX, QT and Cocoa can all be made to work with a minimum of hassle. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
[EMAIL PROTECTED] wrote: On 08:52 pm, [EMAIL PROTECTED] wrote: When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more. Only the very top-most level decides which reactor the application will use. This is a worry, because it implies that there has to *be* a top level that knows what kind of reactor the whole application will use, and all parts of the application need to know that they will be getting their reactor from that top level. That may not be the case. For example, you want to incorporate some piece of event-driven code written by someone else into your gtk application. But it wasn't written with gtk in mind, so it doesn't know to use a gtkreactor, or how to get a reactor that it can use from your application. This is not my idea of what another poster called a one-stop shop -- a common API that different pieces of code can call independently without having to know about each other. To my mind, there shouldn't be a reactor object exposed to the application at all. There should just be functions for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used. So at this point I'm skeptical that the Twisted API for these things should be adopted as-is. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Greg Ewing wrote: [EMAIL PROTECTED] wrote: On 08:52 pm, [EMAIL PROTECTED] wrote: When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more. Only the very top-most level decides which reactor the application will use. This is a worry, because it implies that there has to *be* a top level that knows what kind of reactor the whole application will use, and all parts of the application need to know that they will be getting their reactor from that top level. That may not be the case. For example, you want to incorporate some piece of event-driven code written by someone else into your gtk application. But it wasn't written with gtk in mind, so it doesn't know to use a gtkreactor, or how to get a reactor that it can use from your application. If the borrowed code takes a reactor parameter then presumably the top-level code can pass the appropriate reactor type in. If the borrowed code uses a fixed reactor then it's difficult to see how changes to the Twisted API could help. Incorporating some piece of event-driven code written by someone else implies specific assumptions about event types and delivery, surely. That's why it's difficult to port code between GUI toolkits, for example, and even more so to write code that runs on several toolkits without change. This is not my idea of what another poster called a one-stop shop -- a common API that different pieces of code can call independently without having to know about each other. To my mind, there shouldn't be a reactor object exposed to the application at all. There should just be functions for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used. You seem to be arguing for libraries that contain platform dependencies to handle multiple platforms. Glyph seems to prefer the ability for the library caller to pass in handlers for platform-dependent features. So at this point I'm skeptical that the Twisted API for these things should be adopted as-is. Since Glyph has already stated his opinion that Twisted isn't yet ready for adoption as-is this doesn't add to the discussion. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/14/07, Greg Ewing [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: On 08:52 pm, [EMAIL PROTECTED] wrote: When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more. Only the very top-most level decides which reactor the application will use. This is a worry, because it implies that there has to *be* a top level that knows what kind of reactor the whole application will use, and all parts of the application need to know that they will be getting their reactor from that top level. That may not be the case. For example, you want to incorporate some piece of event-driven code written by someone else into your gtk application. But it wasn't written with gtk in mind, so it doesn't know to use a gtkreactor, or how to get a reactor that it can use from your application. This is not my idea of what another poster called a one-stop shop -- a common API that different pieces of code can call independently without having to know about each other. To my mind, there shouldn't be a reactor object exposed to the application at all. There should just be functions for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used. Eh, your own example seems to argue the opposite. If the choice for reactor was made somewhere deep inside the library, how does it know to use the GTK reactor? Only the code using GTK knows that it's going to need the GTK reactor. The generic code just uses whatever reactor was selected, automatically. The reason the decision should be made at the topmost level is that this is the most visible and adaptive location. A library or framework has to handle a boundless number of variations of environment, use and co-operation with other libraries and frameworks, whereas the topmost script knows exactly what it wants to do. A library that uses PyGTK can say 'I need the GTK reactor', but it would be wrong. All it needs is a reactor that interacts nicely with GTK. If there are multiple reactors that integrate nicely with GTK, there is no reason for the library to insist on one particular implementation. And it could be in the way, if a second library needs a different kind of integration and it knows of a reactor that supports both that *and* GTK integration. And if the whole system breaks, for some reason, the first place the programmer will look is in his own code, not deep inside another library. For what it's worth, I think a single event-handling mainloop API in the standard library is a *great* idea. Using part of Twisted for it (suitably adapted) is an even greater idea: a whole lot of work and brainpower went into Twisted's core, and it wasn't wasted. Duplicating the effort would be a big mistake and a terrible waste. A from-scratch implementation of an event-handling mechanism will not solve any of the issues most people have with Twisted. It will have many more bugs than Twisted (all the ones Twisted had and fixed, sometimes very obscure and hard-to-find bugs), it will not be significantly less complex (Twisted is as complex as it is because it had to be, to do what it is supposed to do) and it will not require any less of a brain-warp to understand (because that's just what event-driven programming takes.) If I wasn't short on free time (spending most of it on Py3k and, soon, py3k-forward-compatibility) I would do all the necessary Twisted-integration work myself :-) -- Thomas Wouters [EMAIL PROTECTED] Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 12:31 am, [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: On 08:52 pm, [EMAIL PROTECTED] wrote: When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more. Only the very top-most level decides which reactor the application will use. This is a worry, because it implies that there has to *be* a top level that knows what kind of reactor the whole application will use, and all parts of the application need to know that they will be getting their reactor from that top level. The default reactor is the most portable one, 'select', and if no other reactor is installed, that's the one that will be used. That may not be the case. For example, you want to incorporate some piece of event-driven code written by someone else into your gtk application. But it wasn't written with gtk in mind, so it doesn't know to use a gtkreactor, or how to get a reactor that it can use from your application. from twisted.internet import reactor is the way you get at the reactor, regardless of which one is currently installed. There can only be one reactor active at any given time, because at the very bottom of the event-handling machinery _some_ platform multiplexing API must be called, and that is mostly what the reactor represents. The GTK reactor does not have its own API. It simply allows you to use GTK APIs as well, by back-ending to the glib mainloop. That is, in fact, the whole point of having a reactor API in the first place. This is not my idea of what another poster called a one-stop shop -- a common API that different pieces of code can call independently without having to know about each other. To my mind, there shouldn't be a reactor object exposed to the application at all. There should just be functions for setting up callbacks. That's what the Twisted reactor object *is*, exactly. Functions (well, methods) for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used. The deep inside the library decision is actually a policy decision made by a server's administrator, or dependent upon the GUI library being used if you need to interact with a GUI event loop. Perhaps the default selection could be better, but installing a reactor is literally one line of code, or a single command-line option to the twistd daemon. See: http://twistedmatrix.com/projects/core/documentation/howto/choosing-reactor.html It is completely transparent to the application, _unless_ the application wants to make use of platform-specific features. See the following for more information: http://www.cs.wustl.edu/~schmidt/PDF/reactor-siemens.pdf although technically Twisted's reactor is more like the slightly higher level POSA proactor pattern; asyncore is more like a true reactor in the sense discussed in that paper. Twisted exposes various APIs for setting up callbacks exactly as you describe: http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IReactorTCP.html http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IReactorTime.html So at this point I'm skeptical that the Twisted API for these things should be adopted as-is Given that your supporting arguments were almost exactly the opposite of the truth in every case, I think this conclusion should be re-examined :). If you're interested in how a normal Twisted application looks, see this tutorial: http://twistedmatrix.com/projects/core/documentation/howto/servers.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Steve Holden wrote: If the borrowed code takes a reactor parameter then presumably the top-level code can pass the appropriate reactor type in. Since there should only be one reactor at a time in any given application, it shouldn't have to be passed in -- it could be held in a global variable deep inside the library. Only the code which creates the reactor initially needs to know about that variable, or even that there is such a thing as a reactor. Incorporating some piece of event-driven code written by someone else implies specific assumptions about event types and delivery, surely. It requires agreement on how to specify the event types and what to do in response, but that's all it should require. The way I envisage it, setting up an event callback should be like opening a file -- there's only one way to do it, and you don't have to worry about what the rest of the application is doing. You don't have to get passed an object that knows how to open files -- it's a fundamental service provided by the system. You just use it. That's why it's difficult to port code between GUI toolkits, for example, and even more so to write code that runs on several toolkits without change. Just in case it's not clear, the events I'm talking about are things like file and socket I/O, not GUI events. Trying to use two different GUIs at once is not something I'm addressing. Rather, you should be able to write code that does e.g. some async socket I/O, and embed it in a GUI app using e.g. gtk, without having to modify it to take account of the fact that it's working in a gtk environment, or having to parameterise it to allow for such things. You seem to be arguing for libraries that contain platform dependencies to handle multiple platforms. I'm arguing that as much of the platform dependency as possible should be in the asyncore library (or whatever replaces it). The main application code *might* have to give it a hint such as this app uses gtk, but no more than that. And ideally, I'd prefer it not to even have to do that -- pygtk should do whatever is necessary to hook itself into asyncore if at all possible, not the other way around. Since Glyph has already stated his opinion that Twisted isn't yet ready for adoption as-is this doesn't add to the discussion. Okay, but one of the suggestions made seemed to be why not just use the Twisted API. I'm putting forward a possible reason. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Thomas Wouters wrote: If the choice for reactor was made somewhere deep inside the library, how does it know to use the GTK reactor? In my ideal world, there wouldn't actually be a gtk reactor -- there would only be a Linux reactor, a MacOSX reactor, a Windows reactor, etc. Things like pygtk would be adapted to hook into the platform's standard reactor. Less ideal would be for pygtk to intall a gtk reactor when it gets imported. The danger with this approach is that two libraries could fight over which kind of reactor to use. You suggest that the top level could choose some other reactor that is compatible with both libraries. That seems like a rather hit-and-miss approach -- such a reactor might exist, or it might not. If not, you're out of luck. And the chance of finding a suitable reactor gets smaller as the number of libraries increases. The reason the decision should be made at the topmost level is that this is the most visible and adaptive location. This is where my vision is fundamentally different: you shouldn't have to *make* a decision in the first place. All event-driven libraries should be made to use the same substrate on any given platform. Then they can coexist without the need for any top-level choices. I know that will be hard to do, but it's the only way out of this mess that I can see. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/14/07, Greg Ewing [EMAIL PROTECTED] wrote: I know that will be hard to do, but it's the only way out of this mess that I can see. That depends on what you consider messy about it. *I* don't like the idea of something in the Python installation deciding which reactor to use. It's my application, and I'm damn well going to tell it what to do. If that means it doesn't work as I expected, it's my own fault :-) In any case, your idea requires a lot of changes in external, non-Python code -- PyGTK simply exposes the GTK mainloop, which couldn't care less about Python's idea of a perfect event reactor model. While those issues are being settled, we'll have to cope with selecting the right reactor manually. It's not all that different from what you want, in any case. The PerfectReactor can be added later, all current reactors aliased to it, and no one would have to change a single line of code. -- Thomas Wouters [EMAIL PROTECTED] Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Phillip J. Eby wrote: peak.events, for example, lets you have multiple event loops running in the same or different threads. Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use threads. And... how do you run multiple event loops simultaneously in the *same* thread? That sounds self-contradictory to me. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/14/07, Greg Ewing [EMAIL PROTECTED] wrote: Phillip J. Eby wrote: peak.events, for example, lets you have multiple event loops running in the same or different threads. Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use threads. And... how do you run multiple event loops simultaneously in the *same* thread? That sounds self-contradictory to me. If all (or all-but-one) of them have a 'run one iteration' method, you can call that from the 'main' mainloop. Or you can design all mainloops as coroutines and have them call each other. (I haven't looked at Phillip's approach at all, but something tells me coroutines are involved :-) -- Thomas Wouters [EMAIL PROTECTED] Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, 15 Feb 2007 15:47:39 +1300, Greg Ewing [EMAIL PROTECTED] wrote: Steve Holden wrote: If the borrowed code takes a reactor parameter then presumably the top-level code can pass the appropriate reactor type in. Since there should only be one reactor at a time in any given application, it shouldn't have to be passed in -- it could be held in a global variable deep inside the library. Only the code which creates the reactor initially needs to know about that variable, or even that there is such a thing as a reactor. Whether or not the premise here is accurate may be out of scope for this thread. Or it may not be. I dunno. However, I do want to point out that it is not necessarily correct that there should be only one reactor at a time in a given application. PJE has already explained that peak.events can have multiple reactors. Twisted is tied to one, but this may not always be the case. Whether there is a default reactor for applications that don't care about the ability to have more than one at a time is yet another question which may be worth examining. These are the kinds of things which should be spelled out in a PEP, including the rationale for any particular policy decisions (which should be kept to an absolute minimum) are made. Incorporating some piece of event-driven code written by someone else implies specific assumptions about event types and delivery, surely. It requires agreement on how to specify the event types and what to do in response, but that's all it should require. The way I envisage it, setting up an event callback should be like opening a file -- there's only one way to do it, and you don't have to worry about what the rest of the application is doing. You don't have to get passed an object that knows how to open files -- it's a fundamental service provided by the system. You just use it. If we suppose that files and sockets are supported in roughly the same way, and we suppose that sockets are supported in the way that Twisted supports them, then there is no difficulty supporting files in this way. :) That's why it's difficult to port code between GUI toolkits, for example, and even more so to write code that runs on several toolkits without change. Just in case it's not clear, the events I'm talking about are things like file and socket I/O, not GUI events. Trying to use two different GUIs at once is not something I'm addressing. Alright, good. Getting two different GUI libraries to play together is a pretty hairy task indeed, and well worth keeping separate from this one. :) Rather, you should be able to write code that does e.g. some async socket I/O, and embed it in a GUI app using e.g. gtk, without having to modify it to take account of the fact that it's working in a gtk environment, or having to parameterise it to allow for such things. Excellent. To be clear, this is how the Twisted model works, with respect to integration with GUI toolkits. I would not enjoy working with a system in which this was not the case. You seem to be arguing for libraries that contain platform dependencies to handle multiple platforms. I'm arguing that as much of the platform dependency as possible should be in the asyncore library (or whatever replaces it). Certainly. Library code doesn't care if the event loop is driven by select or poll or epoll or /dev/poll or kqueue or aio or iocp or win32 events or realtime signals or kaio or whatever gnarly thing is hidden in gtk or whatever gnarly thing is hidden inside qt or whatever gnarly thing is hidden inside COM or whatever gnarly thing is hidden inside wxWidgets. It cares about what features are available. It requests them somehow, and uses them. If they are unavailable, then it can decide whether the lack is catastrophic and give up or if it can be worked around somehow. The way a Twisted application does this is based on interfaces. Assuming interfaces continue to not be present in the stdlib, a stdlib event loop would have to find some other API for presenting this information, but it is not a very hard problem to solve. The main application code *might* have to give it a hint such as this app uses gtk, but no more than that. And ideally, I'd prefer it not to even have to do that -- pygtk should do whatever is necessary to hook itself into asyncore if at all possible, not the other way around. There is some advantage to declaring things up front, lest you get into the situation where you are partway through using code which will suddenly begin to demand Gtk at the same time as you are partway through using code which will suddenly begin to demand Qt, at which point you are in trouble. But this is another minor point. Since Glyph has already stated his opinion that Twisted isn't yet ready for adoption as-is this doesn't add to the discussion. Okay, but one of the suggestions made seemed to be why not just use the Twisted API. I'm putting forward a possible reason. So far, it sounds like the
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, 15 Feb 2007 16:18:40 +1300, Greg Ewing [EMAIL PROTECTED] wrote: [snip] This is where my vision is fundamentally different: you shouldn't have to *make* a decision in the first place. All event-driven libraries should be made to use the same substrate on any given platform. Then they can coexist without the need for any top-level choices. I know that will be hard to do, but it's the only way out of this mess that I can see. Thomas already pointed this out, but I'm repeating it anyway. This vision represents an impossible reality at present. You will not get Gtk or Qt or wxWidgets to use Python's event notification API. If you are really very interested in solving this problem, go to the developers of each platform those toolkits run on and sell them on a unified event notification API. Once they have adopted, implemented, and deployed it, you can go to the Gtk, Qt, and wxWidgets teams and tell them to port all of their code to that new API. Then, you can have a unified model in Python. Until then, the practical compromise with almost zero negative consequences (sometimes, one extra piece of configuration will be required - compare this to how the logging module works ;) is to optionally allow explicit reactor selection. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
At 04:25 PM 2/15/2007 +1300, Greg Ewing wrote: Phillip J. Eby wrote: peak.events, for example, lets you have multiple event loops running in the same or different threads. Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use threads. And... how do you run multiple event loops simultaneously in the *same* thread? When one is nested inside the other. This isn't a common need, but it's occasionally useful if you need to switch back and forth between blocking and non-blocking code. For example, suppose that you have some code that wants to offer a synchronous interface to an asynchronous library... and the synchronous code is being called from a FastCGI accept event loop. The inner code can't use the outer event loop, because the outer loop isn't going to proceed until the inner code is finished. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Thomas Wouters wrote: If all (or all-but-one) of them have a 'run one iteration' method, you can call that from the 'main' mainloop. But without some way of blocking until an event arrives for *either* loop, you have to resort to some kind of busy polling, which is not elegant. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Thomas Wouters wrote: *I* don't like the idea of something in the Python installation deciding which reactor to use. I wouldn't mind if some way were provided of changing the reactor if you want. I'd just like to see a long term goal of making it unnecessary as far as possible. In any case, your idea requires a lot of changes in external, non-Python code -- PyGTK simply exposes the GTK mainloop, which couldn't care less about Python's idea of a perfect event reactor model. On unix at least, I don't think it should be necessary to change gtk, only pygtk. If it can find out the file descriptor of the connection to the X server, it can plug that into the reactor, and then call gtk_main_iteration_do() whenever something comes in on it. A similar strategy ought to work for any X11-based toolkit that exposes a function to perform one iteration of its main loop. Mileage on other platforms may vary. The PerfectReactor can be added later, all current reactors aliased to it, and no one would have to change a single line of code. Sure. The other side to all this is the client side, i.e. the code that installs event callbacks. At the moment there's no clear direction to take, so everyone makes their own choice -- some use asyncore, some use Twisted, some use the gtk event loop, some roll their own, etc. If it were made known that asyncore or some other thing in the stdlib was intended to become the standard, then it would give people some guidance as to how to write future event-driven code. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Phillip J. Eby wrote: At 04:25 PM 2/15/2007 +1300, Greg Ewing wrote: Phillip J. Eby wrote: peak.events, for example, lets you have multiple event loops running in the same or different threads. Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use threads. And... how do you run multiple event loops simultaneously in the *same* thread? When one is nested inside the other. This isn't a common need, but it's occasionally useful if you need to switch back and forth between blocking and non-blocking code. For example, suppose that you have some code that wants to offer a synchronous interface to an asynchronous library... and the synchronous code is being called from a FastCGI accept event loop. The inner code can't use the outer event loop, because the outer loop isn't going to proceed until the inner code is finished. I actually have some code that works a little bit like this. I have a SelectLoop class that the application knows about and explicitly attaches transport objects to, and queues events on the loop. I have two ways to perform some actions in a blocking way. 1) I can create a new SelectLoop and move some transports temporarily from the main loop to the temporary loop to ignore all other events for a while. 2) I can re-enter the SelectLoop run() method and keep reacting to all events. Once the right event occurs - I exit from the nested run() and continue on. This does require a lot more mental effort though. Note that my code is nowhere near as ambitious as Twisted, but I watned to have the above flexibility. The code really only has to work on AIX and Linux, and in a limited way, Windows. - Dave ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/14/07, Phillip J. Eby [EMAIL PROTECTED] wrote: When one is nested inside the other. This isn't a common need, but it's occasionally useful if you need to switch back and forth between blocking and non-blocking code. For example, suppose that you have some code that wants to offer a synchronous interface to an asynchronous library... and the synchronous code is being called from a FastCGI accept event loop. The inner code can't use the outer event loop, because the outer loop isn't going to proceed until the inner code is finished. This would also let you wrap sys.stdout.write() in a nested event loop so as to allow print statements to still work while you use have it set to non-blocking mode, but I could see it argued that using print statements at all is wrong at that point. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Thomas Wouters schrieb: If all (or all-but-one) of them have a 'run one iteration' method, you can call that from the 'main' mainloop. That doesn't really work (and neither do variations involving coroutines. Either the 'run one iteration' method blocks until one even arrives, in which case it may block for a long time while other loops have events pending (which then won't get processed). Or each 'run one iteration' method has a polling mode (e.g. with a time-out), in which case you get busy wait (you can fiddle with the time-out value to trade business vs. responsiveness). The 'run one iteration' approach doesn't support writing applications that are single-threaded, responsive, and idle when there is nothing to do. I can't believe that coroutines help in any way here. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Greg Ewing schrieb: On unix at least, I don't think it should be necessary to change gtk, only pygtk. If it can find out the file descriptor of the connection to the X server, it can plug that into the reactor, and then call gtk_main_iteration_do() whenever something comes in on it. That is insufficient. The gtk main loop has more input sources than just the connection to X: - timers can be registered, which are called when the time comes - idle handlers can be registered which are called when there are no other events - child handlers are invoked when a child process terminates - additional file descriptors can be registered (probably used for sockets primarily) - a generalzed 'event source' can be hooked into it, with C functions for prepare, check, dispatch, and finalize See http://www.gtk.org/api/2.6/glib/glib-The-Main-Event-Loop.html Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
