Re: [Python-Dev] microthreading vs. async io
"Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > [EMAIL PROTECTED] schrieb: > > Asyncore *only* implements asynchronous IO -- any "tasks" performed in > > its context are the direct result of an IO operation, so it's hard to > > say it implements cooperative multitasking (and Josiah can correct me if > > I'm wrong, but I don't think it intends to). > > I'm trying to correct you: By your definition, asyncore implements > cooperative multi-tasking. You didn't define 'task', but I understand > it as 'separate activity'. With asyncore, you can, for example, run > a web server and an IRC server in a single thread, as separate tasks, > and asyncore deals with the scheduling between these tasks. > In your terminology, it is based on continuations: the chunk you specify > is the event handler. > > Indeed, asyncore's doc string starts with > > # There are only two ways to have a program on a single > # processor do "more than one thing at a time". > > and goes on suggesting that asyncore provides one of them. Well, when asyncore was written, it probably didn't have coroutines, generator trampolines, etc., what we would consider today, in this particular context, "cooperative multithreading". What that documentation /should/ have said is... # There are at least two ways to have a program on a single # processor do "more than one thing at a time". Then go on to describe threads, a 'polling for events' approach like asyncore, then left the rest for someone else to add later. I'll add it as my first patch to asyncore. - Josiah ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
On Thu, Feb 15, 2007 at 12:48:48AM +, Steve Holden wrote: > Given that they say "a camel is a horse designed by a committee" Metaphors can go that far but not farther. And, BTW, camels are very suited for their environments... I am not afraid of committees for large tasks. Well, that has to be a small committee ruling by a cleverest ruler. > require a > single individual with good language design skills and the ability to > veto features that were out of line with the design requirements. A lot > like a BDFL, really. Of course, but I don't know if "CP4E" idea is still on his agenda and with what priority. Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
I was the one on the Stackless list who last September or so
proposed the idea of monkeypatching and I'm including that
idea in my presentation for PyCon. See my early rough draft
at http://www.stackless.com/pipermail/stackless/2007-February/002212.html
which contains many details about using Stackless, though
none on the Stackless implementation. (A lot on how to tie things together.)
So people know, I am an applications programmer and not a
systems programmer. Things like OS-specific event mechanisms
annoy and frustrate me. If I could do away with hardware and
still write useful programs I would.
I have tried 3 times to learn Twisted. The first time I found
and reported various problems and successes. See emails at
http://www.twistedmatrix.com/pipermail/twisted-python/2003-June/thread.html
The second time was to investigate a way to report upload
progress: http://twistedmatrix.com/trac/ticket/288
and the third was to compare Allegra and Twisted
http://www.dalkescientific.com/writings/diary/archive/2006/08/28/levels_of_abstraction.html
In all three cases I've found it hard to use Twisted because
the code didn't do as I expected it to do and when something
went wrong I got results which were hard to interpret. I
believe others have similar problems and is one reason Twisted
is considered to be "a big, complicated, inseparable hairy mess."
I find the Stackless code also hard to understand. Eg,
I don't know where the watchdog code is for the "run()"
command. It uses several layers of macros and I haven't
been able get it straight in my head. However, so far
I've not run into strange errors in Stackless that I
have in Twisted.
I find the normal Python code relatively easy to understand.
Stackless only provides threadlets. It does no I/O.
Richard Tew developed a "stacklesssocket" module which emulates
the API for the stdlib "socket" module. I tweaked it a
bit and showed that by doing the monkeypatch
import stacklesssocket
import sys
sys.modules["socket"] = stacklesssocket
then code like "urllib.urlopen" became Stackless compatible.
Eg, in my PyCon talk draft I show something like
import slib
# must monkeypatch before any other module imports "socket"
slib.use_monkeypatch()
import urllib2
import time
import hashlib
def fetch_and_reverse(host):
t1 = time.time()
s = urllib2.urlopen("http://"+host+"/";).read()[::-1]
dt = time.time() - t1
digest = hashlib.md5(s).hexdigest()
print "hash of %r/ = %s in %.2f s" % (host, digest, dt)
slib.main_tasklet(fetch_and_reverse)("www.python.org")
slib.main_tasklet(fetch_and_reverse)("docs.python.org")
slib.main_tasklet(fetch_and_reverse)("planet.python.org")
slib.run_all()
where the three fetches occur in parallel.
The choice of asyncore is, I think, done because 1) it
prevents needing an external dependency, 2) asyncore is
smaller and easier to understand than Twisted, and
3) it was for demo/proof of concept purposes. While
tempting to improve that module I know that Twisted
has already gone though all the platform-specific crap
and I don't want to go through it again myself. I don't
want to write a reactor to deal with GTK, and one for
OS X, and one for ...
Another reason I think Twisted is considered "tangled-up
Deep Magic, only for Wizards Of The Highest Order" is because
it's infused with event-based processing. I've done a lot
of SAX processing and I can say that few people think that
way or want to go through the process of learning how.
Compare, for example, the following
f = urllib2.urlopen("http://example.com/";)
for i, line in enumerate(f):
print ("%06d" % i), repr(line)
with the normal equivalent in Twisted or other
async-based system.
Yet by using the Stackless socket monkeypatch, this
same code works in an async framework. And the underlying
libraries have a much larger developer base than Twisted.
Want NNTP? "import nntplib" Want POP3? "import poplib"
Plenty of documentation about them too.
On the Stackless mailing list I have proposed someone work
on a talk for EuroPython titled "Stackless and Twisted".
Andrew Francis has been looking into how to do that.
All the earlier quotes were lifted from glyph. Here's another:
> When you boil it down, Twisted's event loop is just a
> notification for "a connection was made", "some data was
> received on a connection", "a connection was closed", and
> a few APIs to listen or initiate different kinds of
> connections, start timed calls, and communicate with threads.
> All of the platform details of how data is delivered to the
> connections are abstracted away.. How do you propose we
> would make a less "specific" event mechanism?
What would I need to do to extract this Twisted core so
I could replace asyncore? I know at minimum I need
"twisted.internet" and "twisted.python" (the latter for
logging) and "twisted.persisted" for "styles.Ephemeral".
But I say this hesitantly recalling the frustrations
I had in dealing with a co
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/14/07, Greg Ewing <[EMAIL PROTECTED]> wrote:
> Thomas Wouters wrote:
>
> > *I* don't like the idea of something in the Python installation
> > deciding which reactor to use.
>
> I wouldn't mind if some way were provided of changing
> the reactor if you want. I'd just like to see a long
> term goal of making it unnecessary as far as possible.
>
> > In any case, your idea requires a lot of changes in external, non-Python
> > code -- PyGTK simply exposes the GTK mainloop, which couldn't care less
> > about Python's idea of a perfect event reactor model.
>
> On unix at least, I don't think it should be necessary
> to change gtk, only pygtk. If it can find out the file
> descriptor of the connection to the X server, it can
> plug that into the reactor, and then call
> gtk_main_iteration_do() whenever something comes in
> on it.
>
> A similar strategy ought to work for any X11-based
> toolkit that exposes a function to perform one
> iteration of its main loop.
>
> Mileage on other platforms may vary.
>
> > The PerfectReactor can be added later, all current reactors
> > aliased to it, and no one would have to change a single line
> > of code.
>
> Sure.
>
> The other side to all this is the client side, i.e. the
> code that installs event callbacks. At the moment there's
> no clear direction to take, so everyone makes their own
> choice -- some use asyncore, some use Twisted, some use
> the gtk event loop, some roll their own, etc.
There is no single PerfectReactor. There are several use cases where
you need to wait on >1 different event systems, which guarantees at
least two OS threads (and two event loops). In general it's nice to
have a single Python event loop ("the reactor") to act on said threads
(e.g. something just sitting on a mutex waiting for messages) but
waiting for IO to occur should *probably* happen on one or more
ancillary threads -- one per event system (e.g. select, GTK,
WaitForMultipleEvents, etc.)
-bob
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Trial balloon: microthreads library in stdlib
On 2/14/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote in response to [EMAIL PROTECTED]: > As far as I can tell, you still haven't even clearly expressed what your > needs are, let alone whether or not Twisted is suitable. In the reply > you're citing, you said that "this" sounded like something "low level" that > "twisted would be written on top of" - but the "this" you were talking > about, based on your previous messages, sounded like monkeypatching the > socket and asyncore modules to provide asynchronous file I/O based on the > platform-specific IOCP API for Windows. I don't know Richard's needs nor requirements. I do know the mechanism of the monkeypatch he's talking about. I describe it a bit in my draft for my Stackless talk at PyCon http://www.stackless.com/pipermail/stackless/2007-February/002212.html It uses asyncore for the I/O and a scheduler which can figure out if there are other running tasklets and how long is needed until the next tasklet needs to wake up. Yes, this is another reactor. Yes, it's not widely cross-platform. Yes, it doesn't work with gtk and other GUI frameworks. Yes, as written it doesn't handle threads. But also yes, it's a different style of writing reactors because of how Stackless changes the control flow. I and others would like to see something like the "stacklesssocket" implemented on top of Twisted. Andrew Francis is looking in to it but I don't know to what degree of success he's had. Err, looking through the email archive, he's has had some difficulties in doing a Twisted/Stackless integration. I don't think Twisted people understand Stackless well enough (nor obviously he Twisted) or what he's trying to do. > >It is a large dependency and it is a secondary framework. > > Has it occurred to you that it is a "large" dependency not because we like > making bloated and redundant code, but because it is doing something that is > actually complex and involved? Things like Twisted's support for NNTP, POP3, etc. aren't needed with the monkeypatch approach because the standard Python libraries will work, with Stackless and the underlying asyc library collaborating under the covers. So those parts of Twisted aren't needed or relevant. Twisted is, after all, many things. > I thought that I provided several reasons before as well, but let me state > them as clearly as I can here. Twisted is a large and mature framework with > several years of development and an active user community. The pluggable > event loop it exposes is solving a hard problem, and its implementation > encodes a lot of knowledge about how such a thing might be done. It's also > tested on a lot of different platforms. > > Writing all this from scratch - even a small subset of it - is a lot of > work, and is unlikely to result in something robust enough for use by all > Python programs without quite a lot of effort. It would be a lot easier to > get the Twisted team involved in standardizing event-driven communications > in Python. Every other effort I'm aware of is either much smaller, entirely > proprietary, or both. Again, I would love to be corrected here, and shown > some other large event-driven framework in Python that I was heretofore > unaware of. Sorry for the long quote; wasn't sure how to trim it. I made this point elsewhere and above but feel it's important enough to emphasize once more. Stackless lets people write code which looks like blocking code but which is not. The blocking functions are forwarded to the reactor, which does whatever it's supposed to do, and the results returned. Because most Python networking code will work unchanged (assuming changes to the underlying APIs to work with Stackless, as we hack now through monkeypatches), a microthreads solution almost automatically and trivially gains a large working code base, documentation, and active developer base. There does not need to be "some other large event-driven framework in Python" because all that's needed is the 13kLOC of reactor code from twisted.internet and not 140+kLOC in all of Twisted. > Standardization is much easier to achieve when you have > multiple interested parties with different interests to accommodate. As > Yitzhak Rabin used to say, "you don't engage in API standardization with > your friends, you engage in API standardization with your enemies" - or... > something like that. I thought you made contracts even with -- or especially with -- your friends regarding important matters so that both sides know what they are getting into and so you don't become enemies in the future. > You say that you weren't proposing an alternate implementation of an event > loop core, so I may be reacting to something you didn't say at all. > However, again, at least Tristan thought the same thing, so I'm not the > only one. For demonstration (and for my case pedagogical reasons) we have an event core loop. I would rather pull out the parts I need from Twisted. I don't know how. I don't
Re: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax
Greg Ewing wrote: > Steve Holden wrote: > >> A further data point is that modern machines seem to give timing >> variabilities due to CPU temperature variations even if you always eat >> exactly the same thing. > > Oh, great. Now we're going to have to run our > benchmarks in a temperature-controlled oven... > ... with the fan running at constant speed :) -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax
On Thursday 15 February 2007 21:48, Steve Holden wrote: > Greg Ewing wrote: > > Steve Holden wrote: > >> A further data point is that modern machines seem to give > >> timing variabilities due to CPU temperature variations even if > >> you always eat exactly the same thing. > > > > Oh, great. Now we're going to have to run our > > benchmarks in a temperature-controlled oven... > > ... with the fan running at constant speed :) Unless the fans are perfectly balanced, small changes in gravity are going to affect the rate at which they spin. So I guess the position of the moon will affect it :-) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microthreading vs. async io
[EMAIL PROTECTED] wrote: [...] > microtreading: > Exploiting language features to use cooperative multitasking in tasks > that "read" like they are single-threaded. > > asynchronous IO: > Performing IO to/from an application in such a way that the > application does not wait for any IO operations to complete, but > rather polls for or is notified of the readiness of any IO operations. > > [...] > Asyncore *only* implements asynchronous IO -- any "tasks" performed in > its context are the direct result of an IO operation, so it's hard to > say it implements cooperative multitasking (and Josiah can correct me if > I'm wrong, but I don't think it intends to). > > Much of the discussion here has been about creating a single, unified > asynchronous IO mechanism that would support *any* kind of cooperative > multitasking library. I have opinions on this ($0.02 each, bulk > discounts available), but I'll keep them to myself for now. > Talking only about async I/O in order to write cooperative tasks that "smell" single threaded is to restricted IMO. If there are a number of cooperative tasks that "read" single-threaded (or sequential) than the goal is to avoid a _blocking operation_ in any of them because the other tasks could do useful things in the meantime. But there are a number of different blocking operations, not only async IO (which is easily handled by select()) but also: - waiting for a child process to exit - waiting for a posix thread to join() - waiting for a signal/timer - ... Kevent (kernel event) on BSD e.g. tries to provide a common infrastructure to provide a file descriptor where one can push some conditions onto and select() until one of the conditions is met. Unfortunately, thread joining is not covered by it, so one cannot wait (without some form of busy looping) until one of the conditions is true if thread joining is one of them, but for all the other cases it would be possible. There are many other similar approaches (libevent, notify, to name a few). So in order to avoid blocking in a task, I'd prefer that the task: - declaratively specifies what kind of conditions (events) it wants to wait for. (API) If that declaration is a function call, then this function could implicitely yield if the underlying implementation would be stackless or greenlet based. Kevent on BSD systems already has a usable API for defining the conditions by structures and there is also a python module for it. The important point IMO is to have an agreed API for declaring the conditions a task wants to wait for. The underlying implementation in a scheduler would be free to use whatever event library it wants to use. E.g. have a wait(events = [], timeout = -1) method would be sufficient for most cases, where an event would specify - resource type (file, process, timer, signal, ...) - resource id (fd, process id, timer id, signal number, ...) - filter/flags (when to fire, e.g. writable, readable exception for fd, ...) - ... the result could be a list of events that have "fired", more or less similar to the events in the argument list, but with added information on the exact condition. The task would return from wait(events) when at least 1 of the conditions is met. The task then knows e.g. that an fd is readable and can then do the read() on its own in the way it likes to do it, without being forced to let some uber framework do the low level IO. Just the waiting for conditions without blocking the application is important. I have implemented something like the above, based on greenlets. In addition to the event types specified by BSD kevent(2) I've added a TASK and CHANNEL resource type for the events, so that I can wait for tasks to complete or send/receive messages to/from other tasks without blocking the application. But the implementation is not the important thing, the API is, and then we can start writing competing implementations. Joachim ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Feb 15, 2007, at 6:27 AM, Anthony Baxter wrote: > On Thursday 15 February 2007 21:48, Steve Holden wrote: >> Greg Ewing wrote: >>> Steve Holden wrote: A further data point is that modern machines seem to give timing variabilities due to CPU temperature variations even if you always eat exactly the same thing. >>> >>> Oh, great. Now we're going to have to run our >>> benchmarks in a temperature-controlled oven... >> >> ... with the fan running at constant speed :) > > Unless the fans are perfectly balanced, small changes in gravity are > going to affect the rate at which they spin. So I guess the > position of the moon will affect it :-) Except on Tuesdays. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRdRKcXEjvBPtnXfVAQJUNAP/ebCrt2RV2pmKTElHUyzWIWxPFCqIiIuF FDkiSx4x/ZZtOcdXlJHOA6lBWjuPAxe1jxE9BVpNy/6qCtky+mpnM5nXqIeAlQUk XByguxKmsxF4HWSlk6GJ4hZWbZqsMdFiw8WZttCihQJwmr58rDMTUKRHss2IPOhL egl76Tg1gE4= =OTwQ -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax
Barry Warsaw wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On Feb 15, 2007, at 6:27 AM, Anthony Baxter wrote: > >> On Thursday 15 February 2007 21:48, Steve Holden wrote: >>> Greg Ewing wrote: Steve Holden wrote: > A further data point is that modern machines seem to give > timing variabilities due to CPU temperature variations even if > you always eat exactly the same thing. Oh, great. Now we're going to have to run our benchmarks in a temperature-controlled oven... >>> >>> ... with the fan running at constant speed :) >> >> Unless the fans are perfectly balanced, small changes in gravity are >> going to affect the rate at which they spin. So I guess the >> position of the moon will affect it :-) > > Except on Tuesdays. [off-list, because "this is getting silly" ...] Anthony's antipodean antecedents (alliteration, all right?) remind me we will also have to factor Coriolis effects in. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax
Steve Holden wrote: > Barry Warsaw wrote: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> On Feb 15, 2007, at 6:27 AM, Anthony Baxter wrote: >> >>> On Thursday 15 February 2007 21:48, Steve Holden wrote: Greg Ewing wrote: > Steve Holden wrote: >> A further data point is that modern machines seem to give >> timing variabilities due to CPU temperature variations even if >> you always eat exactly the same thing. > Oh, great. Now we're going to have to run our > benchmarks in a temperature-controlled oven... ... with the fan running at constant speed :) >>> Unless the fans are perfectly balanced, small changes in gravity are >>> going to affect the rate at which they spin. So I guess the >>> position of the moon will affect it :-) >> Except on Tuesdays. > [off-list, because "this is getting silly" ...] Apparently the "off-list" part was a desire that didn't become reality. Sorry, I'll shut up now. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microthreading vs. async io
On 2/15/07, Joachim König-Baltes <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED] wrote: > E.g. have a wait(events = [], timeout = -1) method would be sufficient > for most cases, where an event would specify I agree with everything except this. A simple function call would have O(n) cost, thus being unacceptable for servers with many open connections. Instead you need it to maintain a set of events and let you add or remove from that set as needed. > I have implemented something like the above, based on greenlets. I assume greenlets would be an internal implementation detail, not exposed to the interface? -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, 15 Feb 2007 02:36:22 -0700, Andrew Dalke <[EMAIL PROTECTED]> wrote: >I was the one on the Stackless list who last September or so >proposed the idea of monkeypatching and I'm including that >idea in my presentation for PyCon. See my early rough draft >at http://www.stackless.com/pipermail/stackless/2007-February/002212.html >which contains many details about using Stackless, though >none on the Stackless implementation. (A lot on how to tie things together.) > >So people know, I am an applications programmer and not a >systems programmer. Things like OS-specific event mechanisms >annoy and frustrate me. If I could do away with hardware and >still write useful programs I would. What a wonderful world it would be. :) > > [snip] > >In all three cases I've found it hard to use Twisted because >the code didn't do as I expected it to do and when something >went wrong I got results which were hard to interpret. I >believe others have similar problems and is one reason Twisted >is considered to be "a big, complicated, inseparable hairy mess." > >I find the Stackless code also hard to understand. Eg, >I don't know where the watchdog code is for the "run()" >command. It uses several layers of macros and I haven't >been able get it straight in my head. However, so far >I've not run into strange errors in Stackless that I >have in Twisted. > As you point out below, however, Twisted and stackless achieve different goals. >I find the normal Python code relatively easy to understand. > >Stackless only provides threadlets. It does no I/O. >Richard Tew developed a "stacklesssocket" module which emulates >the API for the stdlib "socket" module. I tweaked it a >bit and showed that by doing the monkeypatch > > import stacklesssocket > import sys > sys.modules["socket"] = stacklesssocket > >then code like "urllib.urlopen" became Stackless compatible. >Eg, in my PyCon talk draft I show something like > It may be of interest to you to learn that a Twisted developer implement this model several years ago. It has not been further developed for a handful of reasons, at the core of which is the fact that it is very similar to pre-emptive threading in terms of application-level complexity. You gave several examples of the use of existing code which expects a blocking socket interface and which "just works" when the socket module is changed in this way. However, this is a slight simplification. Code written without expecting a context switch (exactly what happens when a socket operation is performed in this model) is not necessarily correct when context switches are suddenly introduced. Consider this extremely trivial example: x = 0 def foo(conn): global x a = x + 1 b = ord(conn.recv(1)) x = a + b return x Clearly, foo is not threadsafe. Global mutable state is a terrible, terrible thing. The point to note is that by introducing a context switch at the conn.recv(1) call, the same effect is achieved as by any other context switch: it becomes possible for foo to return an inconsistent result or otherwise corrupt its own state if another piece of code violates its assumptions and changes x while it is waiting for the recv call to complete. Is urllib2 threadsafe? I have heard complaints that it is not. I have looked at the code, and at least in its support for caching, it appears not to be. Perhaps it can be made threadsafe, but in requiring that, the advantage of having a whole suite of modules which will "just work" with a transparently context switching socket module are mostly lost. > > [snip - urllib2/tasklet example] > >The choice of asyncore is, I think, done because 1) it >prevents needing an external dependency, But if some new event loop is introduced into the standard library, then using it also will not require an external dependency. ;) >2) asyncore is >smaller and easier to understand than Twisted, While I hear this a lot, applications written with Twisted _are_ shorter and contain less irrelevant noise in the form of boilerplate than the equivalent asyncore programs. This may not mean that Twisted programs are easier to understand, but it is at least an objectively measurable metric. >and >3) it was for demo/proof of concept purposes. >While >tempting to improve that module I know that Twisted >has already gone though all the platform-specific crap >and I don't want to go through it again myself. I don't >want to write a reactor to deal with GTK, and one for >OS X, and one for ... > Now if we can only figure out a way for everyone to benefit from this without tying too many brains up in knots. :) > >Another reason I think Twisted is considered "tangled-up >Deep Magic, only for Wizards Of The Highest Order" is because >it's infused with event-based processing. I've done a lot >of SAX processing and I can say that few people think that >way or want to go through the process of learning how. > >Compare, for example, the following > > f = urllib2.
Re: [Python-Dev] microthreading vs. async io
Adam Olsen wrote: > I agree with everything except this. A simple function call would > have O(n) cost, thus being unacceptable for servers with many open > connections. Instead you need it to maintain a set of events and let > you add or remove from that set as needed. We can learn from kevent here, it already has EV_ADD, EV_DELETE, EV_ENABLE, EV_DISABLE, EV_ONESHOT flags. So the event-conditions would stay "in the scheduler" (per task) so that they can fire multiple times without the need to be handled over again and again. Thanks, that's exactly the discussion I'd like to see, discussing about a simple API. >> I have implemented something like the above, based on greenlets. > > I assume greenlets would be an internal implementation detail, not > exposed to the interface? Yes, you could use stackless, perhaps even Twisted, but I'm not sure if that would work because the requirement for the "reads single-threaded" is the simple wait(...) function call that does a yield (over multiple stack levels down to the function that created the task), something that is only provided by greenlet and stackless to my knowledge. Joachim ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microthreading vs. async io
On 2/15/07, Joachim König-Baltes <[EMAIL PROTECTED]> wrote: > Adam Olsen wrote: > >> I have implemented something like the above, based on greenlets. > > > > I assume greenlets would be an internal implementation detail, not > > exposed to the interface? > Yes, you could use stackless, perhaps even Twisted, > but I'm not sure if that would work because the requirement for the > "reads single-threaded" is the simple wait(...) function call that does > a yield > (over multiple stack levels down to the function that created the task), > something that is only provided by greenlet and stackless to my knowledge. I don't think we're on the same page then. The way I see it you want a single async IO implementation shared by everything while having a collection of event loops that cooperate "just enough". The async IO itself would likely end up being done in C. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microthreading vs. async io
Adam Olsen schrieb: > I don't think we're on the same page then. The way I see it you want > a single async IO implementation shared by everything while having a > collection of event loops that cooperate "just enough". The async IO > itself would likely end up being done in C. > No, I'd like to have: - An interface for a task to specifiy the events it's interested in, and waiting for at least one of the events (with a timeout). - an interface for creating a task (similar to creating a thread) - an interface for a schedular to manage the tasks When a task resumes after a wait(...) it knows which of the events it was waiting for have fired. It can then do whatever it wants, do the low level non blocking IO on its own or using something else. Of course, as these are cooperative tasks, they still must be careful not to block, e.g. not reading to much from a file descriptor that is readable, but these problems have been solved in a lot of libraries, and I would not urge the task to use a specific way to accomplish its "task". The problem solved by this approach is to allow a number of cooperating threads to wait for an event without the need to busy loop or block by delegating the waiting to a central instance, the scheduler. How this efficient waiting is implemented is the responsability of the scheduler, but the schedular would not do the (possibly blocking) io operation, it would only guaranty to continue a task, when it can do an IO operation without blocking. Joachim ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, Feb 15, 2007 at 09:19:30AM -0500, Jean-Paul Calderone wrote:
> >That feels like 6 layers too many, given that
> > _logrun(selectable, _drdw, selectable, method, dict)
> > return context.call({ILogContext: newCtx}, func, *args, **kw)
> > return self.currentContext().callWithContext(ctx, func, *args, **kw)
> > return func(*args, **kw)
> > getattr(selectable, method())
> > klass(number, string)
> >
> >are all generic calls.
>
> I know function calls are expensive in Python, and method calls even more
> so... but I still don't understand this issue. Twisted's call stack is too
> deep? It is fair to say it is deep, I guess, but I don't see how that is a
> problem. If it is, I don't see how it is specific to this discussion.
It's hard to debug the resulting problem. Which level of the *12*
levels in the stack trace is responsible for a bug? Which of the *6*
generic calls is calling the wrong thing because a handler was set up
incorrectly or the wrong object provided? The code is so 'meta' that
it becomes effectively undebuggable.
--amk
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] generic async io (was: microthreading vs. async io)
On Thu, Feb 15, 2007 at 04:28:17PM +0100, Joachim K?nig-Baltes wrote: > No, I'd like to have: > > - An interface for a task to specifiy the events it's interested in, and > waiting for at least one of the events (with a timeout). > - an interface for creating a task (similar to creating a thread) > - an interface for a schedular to manage the tasks I think this discussion would be facilitated by teasing the first bullet-point from the latter two: the first deals with async IO, while the latter two deal with cooperative multitasking. It's easy to write a single package that does both, but it's much harder to write *two* fairly generic packages with a clean API between them, given the varied platform support for async IO and the varied syntax and structures (continuations vs. microthreads, in my terminology) for multitasking. Yet I think that division is exactly what's needed. Since you asked (I'll assume the check for $0.02 is in the mail), I think a strictly-async-IO library would offer the following: - a sleep queue object to which callables can be added - wrappers for all/most of the stdlib blocking IO operations which add the operation to the list of outstanding operations and return a sleep queue object - some relatively easy method of extending that for new IO operations - a poll() function (for multitasking libraries) and a serve_forever() loop (for asyncore-like uses, where all the action is IO-driven) The mechanisms for accomplishing all of that on the chosen platform would be an implementation detail, possibly with some intitialization "hinting" from the application. The library would also need to expose its platform-based limitations (can't wait on thd.join(), can only wait on 64 fd's, etc.) to the application for compatibility-checking purposes. Thoughts? Dustin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microthreading vs. async io
Joachim König-Baltes wrote: > The problem solved by this approach is to allow a number of cooperating > threads to wait for an event without the need to busy loop or block by > delegating > the waiting to a central instance, the scheduler. How this efficient > waiting is > implemented is the responsability of the scheduler, but the schedular would > not do the (possibly blocking) io operation, it would only guaranty to > continue a task, when it can do an IO operation without blocking. > > From the point of view of the task, it only has to sprinkle a number of wait(...) calls before doing blocking calls, so there is no need to use callbacks or writing the inherently sequential code upside down. That is the gain I'm interested in. The style used in asyncore, inheriting from a class and calling return in a method and being called later at a different location (different method) just interrupts the sequential flow of operations and makes it harder to understand. The same is true for all other strategies using callbacks or similar mechanisms. All this can be achieved with a multilevel yield() that is hidden in a function call. So the task does a small step down (wait) in order to jump up (yield) to the scheduler without disturbing the eye of the beholder. Joachim ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, 15 Feb 2007 10:46:05 -0500, "A.M. Kuchling" <[EMAIL PROTECTED]> wrote:
>On Thu, Feb 15, 2007 at 09:19:30AM -0500, Jean-Paul Calderone wrote:
>> >That feels like 6 layers too many, given that
>> > _logrun(selectable, _drdw, selectable, method, dict)
>> > return context.call({ILogContext: newCtx}, func, *args, **kw)
>> > return self.currentContext().callWithContext(ctx, func, *args, **kw)
>> > return func(*args, **kw)
>> > getattr(selectable, method())
>> > klass(number, string)
>> >
>> >are all generic calls.
>>
>> I know function calls are expensive in Python, and method calls even more
>> so... but I still don't understand this issue. Twisted's call stack is too
>> deep? It is fair to say it is deep, I guess, but I don't see how that is a
>> problem. If it is, I don't see how it is specific to this discussion.
>
>It's hard to debug the resulting problem. Which level of the *12*
>levels in the stack trace is responsible for a bug? Which of the *6*
>generic calls is calling the wrong thing because a handler was set up
>incorrectly or the wrong object provided? The code is so 'meta' that
>it becomes effectively undebuggable.
I've debugged plenty of Twisted applications. So it's not undebuggable. :)
Application code tends to reside at the bottom of the call stack, so Python's
traceback order puts it right where you're looking, which makes it easy to
find. For any bug which causes something to be set up incorrectly and only
later manifests as a traceback, I would posit that whether there is 1 frame or
12, you aren't going to get anything useful out of the traceback. Standard
practice here is just to make exception text informative, I think, but this is
another general problem with Python programs and event loops, not one specific
to either Twisted itself or the particular APIs Twisted exposes.
As a personal anecdote, I've never once had to chase a bug through any of the
6 "generic calls" singled out. I can't think of a case where I've helped any
one else who had to do this, either. That part of Twisted is very old, it is
_very_ close to bug-free, and application code doesn't have very much control
over it at all. Perhaps in order to avoid scaring people, there should be a
way to elide frames from a traceback (I don't much like this myself, I worry
about it going wrong and chopping out too much information, but I have heard
other people ask for it)?
Jean-Paul
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Bob Ippolito wrote:
> There is no single PerfectReactor. There are several use cases where
> you need to wait on >1 different event systems, which guarantees at
> least two OS threads (and two event loops). In general it's nice to
> have a single Python event loop ("the reactor") to act on said threads
> (e.g. something just sitting on a mutex waiting for messages) but
> waiting for IO to occur should *probably* happen on one or more
> ancillary threads -- one per event system (e.g. select, GTK,
> WaitForMultipleEvents, etc.)
Why couldn't PerfectReactor be a reactor for other reactors? A sort of
concentrator for these multiple event systems and multiple threads.
You ask to listen to sockets, so it instantiates a singleton
PerfectReactor which instantiates a select() reactor and listens to it
directly in a single-threaded manner. If you then ask to listen to
Win32 messages, the PerfectReactor instantiates a GetMessage() reactor.
Then, realizing it has two "event systems", it spawns a thread for each
child reactor with a listener that serializes the incoming events into
the PerfectReactor's queue. Bingo, your application doesn't need to be
written thread-safe, PerfectReactor is platform-agnostic, and you don't
have to know in advance all the event types you might ever listen to.
Sorry if this is a dumb question (or if I'm mangling the terminology),
/larry/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microthreading vs. async io
At 11:00 PM 2/14/2007 -0600, [EMAIL PROTECTED] wrote: >Instead, I would like to concentrate on producing a small, clean, >consistent, generator-based microthreading library. I've seen several >such libraries (including the one in PEP 342, which is fairly skeletal), >and they all work *almost* the same way, but differ in, for example, the >kinds of values that can be yielded, their handling of nested calls, and >the names for the various "special" values one can yield. Which is one reason that any "standard" coroutine library would need to be extensible with respect to the handling of yielded values, e.g. by using a generic function to implement the yielding. See the 'yield_to' function in the example I posted. Actually, the example I posted would work fine as a microthreads core by adding a thread-local variable that points to some kind of "scheduler" to replace the Twisted scheduling functions I used. It would need to be a variable, because applications would have to be able to replace it, e.g. with the Twisted reactor. In other words, the code I posted isn't really depending on Twisted for anything but reactor.callLater() and the corresponding .isActive() and .cancel() methods of the objects it returns. If you add a default implementation of those features that can be replaced with the Twisted reactor, and dropped the code that deals with Deferreds and TimeoutErrors, you'd have a nice standalone library. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Larry Hastings schrieb:
> Bob Ippolito wrote:
>> There is no single PerfectReactor. There are several use cases where
>> you need to wait on >1 different event systems, which guarantees at
>> least two OS threads (and two event loops). In general it's nice to
>> have a single Python event loop ("the reactor") to act on said threads
>> (e.g. something just sitting on a mutex waiting for messages) but
>> waiting for IO to occur should *probably* happen on one or more
>> ancillary threads -- one per event system (e.g. select, GTK,
>> WaitForMultipleEvents, etc.)
> Why couldn't PerfectReactor be a reactor for other reactors? A sort of
> concentrator for these multiple event systems and multiple threads.
Because you can't write a single function that blocks (idle) until
an event in one of the systems occurs. None of the system is truly
asynchronous. All APIs, eventually, rely on synchronous, blocking,
operating system calls. In Unix, the call to make is typically
select or poll, and you know it is safe to make because you have
all even sources as file descriptors and you know nothing else
needs to be done before one of the events occurs.
Now, for these generalized event loops, it may not be possible anymore
to combine all event sources into a single blocking call. If you have
timeouts, you have to find out what the minimum timeout is (to make
it the timeout for the blocking call, assuming that supports a timeout).
If you have idle tasks, you have to invoke all idle tasks of all
reactors (which you may not be able to get because there is no API
to fetch them).
Even more difficult: if these use different OS event mechanisms
(on systems that have multiple such systems). E.g. try combinging
a Win32 GetMessage/PostMessage loop with a WaitForMultipleObjects
loop with a WinSock select call; then add the Win32 RPC libraries
to that (i.e. try to determine the named pipe handles you have to
pass to WaitForMultipleObjects).
> You ask to listen to sockets, so it instantiates a singleton
> PerfectReactor which instantiates a select() reactor and listens to it
> directly in a single-threaded manner. If you then ask to listen to
> Win32 messages, the PerfectReactor instantiates a GetMessage() reactor.
> Then, realizing it has two "event systems", it spawns a thread for each
> child reactor with a listener that serializes the incoming events into
> the PerfectReactor's queue.
Ah, threads :-( It turns out that you need to invoke GetMessage in the
context of the thread in which the window was created. In a different
thread, you won't get any messages. (This is indeed one of the biggest
limitations of USER32, which was only overcome in Vista when the
"native" GUI programming model is based on DirectX - the problem is
then of course solved only for applications that use the DirectX UI
API).
> Bingo, your application doesn't need to be
> written thread-safe, PerfectReactor is platform-agnostic, and you don't
> have to know in advance all the event types you might ever listen to.
>
> Sorry if this is a dumb question (or if I'm mangling the terminology),
Integrating with threads might be a solution in some cases, and
a problem in others. You can't assume it is a universal solution.
Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microthreading vs. async io
On Thu, Feb 15, 2007 at 04:51:30PM +0100, Joachim K?nig-Baltes wrote:
> The style used in asyncore, inheriting from a class and calling return
> in a method
> and being called later at a different location (different method) just
> interrupts the
> sequential flow of operations and makes it harder to understand. The same is
> true for all other strategies using callbacks or similar mechanisms.
>
> All this can be achieved with a multilevel yield() that is hidden in a
> function call.
> So the task does a small step down (wait) in order to jump up (yield) to
> the scheduler
> without disturbing the eye of the beholder.
I agree -- I find that writing continuations or using asyncore's
structure makes spaghetti out of functionality that requires multiple
blocking operations inside looping or conditional statements. The best
example, for me, was writing a complex site-specific web spider that had
to fetch 5-10 pages in a certain sequence, where each step in that
sequence depended on the results of the previous fetches. I wrote it in
Twisted, but the proliferation of nested callback functions and chained
deferreds made my head explode while trying to debug it. With a decent
microthreading library, that could look like:
def fetchSequence(...):
fetcher = Fetcher()
yield fetcher.fetchHomepage()
firstData = yield fetcher.fetchPage('http://...')
if someCondition(firstData):
while True:
secondData = yield fetcher.fetchPage('http://...')
# ...
if someOtherCondition(secondData): break
else:
# ...
which is *much* easier to read and debug. (FWIW, after I put my head
back together, I rewrote the app with threads, and it now looks like the
above, without the yields. Problem is, throttlling on fetches means 99%
of my threads are blocked on sleep() at any given time, which is just
silly).
All that said, I continue to contend that the microthreading and async
IO operations are separate. The above could be implemented relatively
easily in Twisted with a variant of the microthreading module Phillip
posted earlier. It could also be implemented atop a bare-bones
microthreading module with Fetcher using asyncore on the backend, or
even scheduler urllib.urlopen() calls into OS threads. Presumably, it
could run in NanoThreads and Kamaelia too, among others.
What I want is a consistent syntax for microthreaded code, so that I
could write my function once and run it in *all* of those circumstances.
Dustin
P.S. For the record -- I've written lots of other apps in Twisted with
great success; this one just wasn't a good fit.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microthreading vs. async io
On Thu, 15 Feb 2007 10:36:21 -0600, [EMAIL PROTECTED] wrote:
> [snip]
>
>def fetchSequence(...):
> fetcher = Fetcher()
> yield fetcher.fetchHomepage()
> firstData = yield fetcher.fetchPage('http://...')
> if someCondition(firstData):
>while True:
> secondData = yield fetcher.fetchPage('http://...')
> # ...
> if someOtherCondition(secondData): break
> else:
># ...
Ahem:
from twisted.internet import reactor
from twisted.internet.defer import inlineCallbacks
from twisted.web.client importt getPage
@inlineCallbacks
def fetchSequence(...):
homepage = yield getPage(homepage)
firstData = yield getPage(anotherPage)
if someCondition(firstData):
while:
secondData = yield getPage(wherever)
if someOtherCondition(secondData):
break
else:
...
So as I pointed out in another message in this thread, for several years it
has been possible to do this with Twisted. Since Python 2.5, you can do it
exactly as I have written above, which looks exactly the same as your example
code.
Is the only problem here that this style of development hasn't had been made
visible enough?
Jean-Paul
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microthreading vs. async io
At 11:47 AM 2/15/2007 -0500, Jean-Paul Calderone wrote: >Is the only problem here that this style of development hasn't had been made >visible enough? You betcha. I sure as heck wouldn't have bothered writing the module I did, if I'd known it already existed. Or at least only written whatever parts that the module doesn't do. The Twisted used by the current version of Chandler doesn't include this feature yet, though, AFAICT. But this is excellent; it means people will be able to write plugins that do network I/O without needing to grok CPS. They'll still need to be able to grok some basics (like not blocking the reactor), but this is good to know about. Now I won't have to actually test that module I wrote. ;-) You guys should be trumpeting this - it's real news and in fact a motivator for people to upgrade to Python 2.5 and whatever version of Twisted supports this. You just lifted a *major* barrier to using Twisted for client-oriented tasks, although I imagine there probably needs to be even more of it. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microthreading vs. async io
On 2/15/07, Joachim König-Baltes <[EMAIL PROTECTED]> wrote: > Adam Olsen schrieb: > > I don't think we're on the same page then. The way I see it you want > > a single async IO implementation shared by everything while having a > > collection of event loops that cooperate "just enough". The async IO > > itself would likely end up being done in C. > > > No, I'd like to have: > > - An interface for a task to specifiy the events it's interested in, and > waiting > for at least one of the events (with a timeout). > - an interface for creating a task (similar to creating a thread) > - an interface for a schedular to manage the tasks My tasks are transient and only wait on one thing at a time (if not waiting for the scheduler to let them run!). I have my own semantics for creating tasks that incorporate exception propagation. My exception propagation (and return handling) require a scheduler with specific support for them. Net result is that I'd have to wrap all you provide, if not monkey-patching it because it doesn't provide the interfaces to let me wrap it properly. All I want is a global select.poll() object that all the event loops can hook into and each will get a turn to run after each call. Well, that, plus I want it to work around all the platform-specific peculiarities. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microthreading vs. async io
On 2/15/07, Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > Is the only problem here that this style of development hasn't had been made > visible enough? Perhaps not the only problem, but definitely a big part of it. I looked for such a thing in twisted after python 2.5 came out and was unable to find it. If I had I may not have bothered to update my own microthreads to use python 2.5 (my proof-of-concept was based on real threads). -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Martin v. Löwis wrote: > Now, for these generalized event loops, it may not be possible anymore > to combine all event sources into a single blocking call. Right, that's why my proposal assumed that each disparate event source would need its own thread. > Ah, threads :-( It turns out that you need to invoke GetMessage in the > context of the thread in which the window was created. In a different > thread, you won't get any messages. Oof! I'm embarrassed to have forgotten that. But that's not a fatal problem. It means that on Windows the PerfectReactor must service the blocking GetMessage loop, and these other threads notify the PerfectReactor of new events by sending a message. (Either that, or, it could poll GetMessage and its incoming event queue without ever blocking. But that is obviously suboptimal.) I think I've done this sort of thing before, in fact. Of course, in the absence of any windows, the Windows PerfectReactor could fall back to a mutex. Abstract this inside PerfectReactor and its event sources wouldn't notice the difference. > Integrating with threads might be a solution in some cases, and a > problem in others. You can't assume it is a universal solution. Universal? Yeah, I doubt it too. But perhaps it would be "good enough for nearly all cases". In the cases where it wasn't, it could throw an "I can't listen to that type of event right now" exception, forcing you to fall back to preconfiguring your central reactor by hand. Anyway, like many folks I'm hoping this whole conversation results in establishing basic standard duck-typed conventions--what other languages might call "interfaces"--for event senders and receivers. In which case this non-universal threaded approach would simply be one of several to choose from. I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable. Thanks for the thoughtful reply, /larry/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microthreading vs. async io
On Thu, Feb 15, 2007 at 11:47:27AM -0500, Jean-Paul Calderone wrote: > Is the only problem here that this style of development hasn't had been made > visible enough? Yep -- I looked pretty hard about two years ago, and although I haven't been looking for that specifically since, I haven't heard anything about it. The API docs don't provide a good way to find things like this, and the Twisted example tutorial didn't mention it at my last check. So if we have an in-the-field implementation of this style of programming (call it what you will), is it worth *standardizing* that style so that it's the same in Twisted, my library, and anyone else's library that cares to follow the standard? Dustin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] generic async io (was: microthreading vs. async io)
[EMAIL PROTECTED] wrote: > > I think this discussion would be facilitated by teasing the first > bullet-point from the latter two: the first deals with async IO, while > the latter two deal with cooperative multitasking. > > It's easy to write a single package that does both, but it's much harder > to write *two* fairly generic packages with a clean API between them, > given the varied platform support for async IO and the varied syntax and > structures (continuations vs. microthreads, in my terminology) for > multitasking. Yet I think that division is exactly what's needed. Hmm. Now, please, people, don't take offence, but I don't know how to phrase this tactfully :-( The 'threading' approach to asynchronous I/O was found to be a BAD IDEA back in the 1970s, was abandoned in favour of separating asynchronous I/O from threading, and God alone knows why it was reinvented - except that most of the people with prior experience had died or retired :-( Let's go back to the days when asynchronous I/O was the norm, and I/O performance critical applications drove the devices directly. In those days, yes, that approach did make sense. But it rapidly ceased to do so with the advent of 'semi-intelligent' devices and the virtualisation of I/O by the operating system. That was in the mid-1970s. Nowadays, ALL devices are semi-intelligent and no system since Unix has allowed applications direct access to devices, except for specialised HPC and graphics. We used to get 90% of theoretical peak performance on mainframes using asynchronous I/O from clean, portable applications, but it was NOT done by treating the I/O as threads and controlling their synchronisation by hand. In fact, quite the converse! It was done by realising that asynchronous I/O and explicit threading are best separated ENTIRELY. There were two main models: Streaming, as in most languages (Fortran, C, Python, but NOT in POSIX). The key properties here are that the transfer boundaries have no significance, only heavyweight synchronisation primitives (open, close etc.) provide any constraints on when data are actually transferred and (for very high performance) buffers are unavailable from when a transfer is started to when it is checked. If copying is acceptable, the last constraint can be dropped. In the simple case, this allows the library/system to reblock and perform transfers asynchronously. In the more advanced case, the application has to use multiple buffering (at least double), but can get full performance without any form of threading. IBM MVT applications used to get up to 90% without hassle in parallel with computation and using only a single thread (well, there was only a single CPU, anyway). The other model is transactions. This has the property that there is a global commit primitive, and the order of transfers is undefined between commits. Inter alia, it means that overlapping transfers are undefined behaviour, whether in a single thread or in multiple threads. BSP uses this model. The MPI-2 design team included a lot of ex-mainframe people and specifies both models. While it is designed for parallel applications, the I/O per se is not controlled like threads. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] generic async io (was: microthreading vs. async io)
On Thu, Feb 15, 2007 at 07:46:59PM +, Nick Maclaren wrote:
> [EMAIL PROTECTED] wrote:
> >
> > I think this discussion would be facilitated by teasing the first
> > bullet-point from the latter two: the first deals with async IO, while
> > the latter two deal with cooperative multitasking.
> >
> > It's easy to write a single package that does both, but it's much harder
> > to write *two* fairly generic packages with a clean API between them,
> > given the varied platform support for async IO and the varied syntax and
> > structures (continuations vs. microthreads, in my terminology) for
> > multitasking. Yet I think that division is exactly what's needed.
>
> The 'threading' approach to asynchronous I/O was found to be a BAD
> IDEA back in the 1970s, was abandoned in favour of separating
> asynchronous I/O from threading, and God alone knows why it was
> reinvented - except that most of the people with prior experience
> had died or retired :-(
Knowing the history of something like this is very helpful, but I'm not
sure what you mean by this first paragraph. I think I'm most unclear
about the meaning of "The 'threading' approach to asynchronous I/O"?
Its opposite ("separating asynchronous I/O from threading") doesn't
illuminate it much more. Could you elaborate?
Dustin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Py2.6 ideas
* Teach vars() to work with classes defining __slots__. Essentially, __slots__ are just an implementation detail designed to make instances a bit more compact. * Make the docstring writable for staticmethods, classmethods, and properties. We did this for function objects and it worked-out well. * Have staticmethods and classmethods automatically fill-in a docstring from the wrapped function. An editor's tooltips would benefit nicely. * Add a pure python named_tuple class to the collections module. I've been using the class for about a year and found that it greatly improves the usability of tuples as records. http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 * Give builtin types a __name__ attribute so that we have a uniform way of accessing type names. * Enhance doctest with a method that computes updated doctest results. If the only change I make to a matrix suite is to add commas to long numbers in the matrix repr(), then it would be nice to have an automated way to update the matrix output on all the other test cases in the module. * add an optional position argument to heapq.heapreplace to allow an arbitrary element to be updated in-place and then have the heap condition restored. I've now encountered three separate parties trying to figure-out how to do this. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] generic async io (was: microthreading vs. async io)
[EMAIL PROTECTED] wrote:
>
> Knowing the history of something like this is very helpful, but I'm not
> sure what you mean by this first paragraph. I think I'm most unclear
> about the meaning of "The 'threading' approach to asynchronous I/O"?
> Its opposite ("separating asynchronous I/O from threading") doesn't
> illuminate it much more. Could you elaborate?
I'll try. Sorry about being unclear - it is one of my failings.
Here is an example draft of some interfaces:
Threading
-
An I/O operation passes a buffer, length, file and action and receives a
token back. This token can be queried for completion, waited on and so
on, and is cancelled by waiting on it and getting a status back. I.e.
it is a thread-like object. This is the POSIX-style operation, and is
what I say cannot be made to work effectively.
Streaming
-
An I/O operation either writes some data to a stream or reads some data
from it; such actions are sequenced within a thread, but not between
threads (even if the threads coordinate their I/O). Data written goes
into limbo until it is read, and there is no way for a reader to find
the block boundaries it was written with or whether data HAS been
written. A non-blocking read merely tests if data are ready for
reading, which is not the same.
There are no positioning operations, and only open, close and POSSIBLY a
heavyweight synchronise or rewind (both equivalent to close+open) force
written data to be transferred. Think of Fortran sequential I/O without
BACKSPACE or C I/O without ungetc/ungetchar/fseek/fsetpos.
Transactions
An I/O operation either writes some data to a file or reads some data
from it. There is no synchronisation of any form until a commit. If
two transfers between a pair of commits overlap (including file length
changes), the behaviour is undefined. All I/O includes its own
positioning, and no positioning is relative.
Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email: [EMAIL PROTECTED]
Tel.: +44 1223 334761Fax: +44 1223 334679
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] UserDict revamp
> On 2/15/07, Raymond Hettinger <[EMAIL PROTECTED]> wrote: >> I would like to be the one to migrate it to Py3.0. [Steve] > No complaints here. Not that you need my permission of course. ;-) Thanks, I had already started working on this one. Of course, everyone is welcome to contribute. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] generic async io
[EMAIL PROTECTED] wrote: > I think this discussion would be facilitated by teasing the first > bullet-point from the latter two: the first deals with async IO, while > the latter two deal with cooperative multitasking. > > It's easy to write a single package that does both, but it's much harder > to write *two* fairly generic packages with a clean API between them, > given the varied platform support for async IO and the varied syntax and > structures (continuations vs. microthreads, in my terminology) for > multitasking. Yet I think that division is exactly what's needed. > > Since you asked (I'll assume the check for $0.02 is in the mail), I > think a strictly-async-IO library would offer the following: > > - a sleep queue object to which callables can be added > - wrappers for all/most of the stdlib blocking IO operations which >add the operation to the list of outstanding operations and return >a sleep queue object >- some relatively easy method of extending that for new IO operations > - a poll() function (for multitasking libraries) and a serve_forever() >loop (for asyncore-like uses, where all the action is IO-driven) A centralized approach of wrapping all blocking IO operation in stdlib could only work in pure python applications. What about extensions that integrate e.g. gtk2, gstreamer and other useful libraries that come with their own low level IO. Python is not the right place to solve this problem, and there are so many C-Libraries which tried it, e.g. gnu-pth tries to implement pthreads on a single-threaded OS. But none of these approaches is perfect. E.g. if you want to read 5 bytes from a fd, you can use FIONREAD on a socket and get the number of bytes available from the OS, so you can be sure to not block, but FIONREAD on a normal file fd (e.g. on a NFS mount) will not tell you, how many bytes the OS has prefetched, so you might block, even if you are reading only 1 byte. I think it's best to decide how to do the low level IO for each case in the task. It knows what's it's doing and how to avoid blocking. Therefore I propose to decouple the waiting for a condition/event from the actual blocking operation. And to avoid the blocking, there is no need to reinvent the wheel, the socket module already provides ways to avoid it for network IO and a lot of C libraries exist to do it in a portable way, but none is perfect. And based on these events it's much easier to design a schedular than to write one which also has to do the non blocking IO operations in order to give the tasks the illusion of a blocking operation. The BSD kevent is the most powerful event waiting mechanism with kernel support (as it unifies the waiting on different events on different resources like fd, process, timers, signals) but its API can be emulated to a sudden degree in the other event mechanism like notify on Linux or Niels Provos' libevent. The real showstopper for making the local event waiting easy are the missing coroutines or at least a form of non local goto like setjmp/longjump in C (that's what greenlets provides), remember that yield() only suspends the current function, so every function on the stack must be prepared to handle the yield, even if they are not interested in it (hiding this fact with decorators does not make it better IMO) Joachim ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
>> Ah, threads :-( It turns out that you need to invoke GetMessage in the >> context of the thread in which the window was created. In a different >> thread, you won't get any messages. > > I'd be interested to hear about other situations where threading would > cause a problem. My suspicion is that Windows is the hard one, and as > I've shown that one is solvable. > > I've tried something similar on Linux, with gtk an wx. You can run the gtk main loop in its own thread, but because gtk is not thread safe, you have to grab a mutex everytime you run gtk code outside the thread the mainloop is running in. So you have to surround your calls to the gtk api with calls to gtk.threads_enter and gtk.threads_leave. Except for callbacks of course, because they are executed in the main thread... Doable, but not fun. The same goes for wx. Then all hell breaks loose when you try to use both gtk and wx at the same time. That's because on Linux, the wx main loop calls the gtk mainloop behind the scenes. As far as I know, that problem can not be solved from python. So yes that strategy can work, but it's no silver bullet. Cheers, Baptiste ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/15/07, Baptiste Carvello <[EMAIL PROTECTED]> wrote: > >> Ah, threads :-( It turns out that you need to invoke GetMessage in the > >> context of the thread in which the window was created. In a different > >> thread, you won't get any messages. > > > > I'd be interested to hear about other situations where threading would > > cause a problem. My suspicion is that Windows is the hard one, and as > > I've shown that one is solvable. > > > > > I've tried something similar on Linux, with gtk an wx. > > You can run the gtk main loop in its own thread, but because gtk is not thread > safe, you have to grab a mutex everytime you run gtk code outside the thread > the > mainloop is running in. So you have to surround your calls to the gtk api with > calls to gtk.threads_enter and gtk.threads_leave. Except for callbacks of > course, because they are executed in the main thread... Doable, but not fun. > > The same goes for wx. Then all hell breaks loose when you try to use both gtk > and wx at the same time. That's because on Linux, the wx main loop calls the > gtk > mainloop behind the scenes. As far as I know, that problem can not be solved > from python. > > So yes that strategy can work, but it's no silver bullet. And it's worse on Windows and Mac OS X where some GUI API calls *must* happen on a particular thread or they simply don't work. -bob ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > On Thu, 15 Feb 2007 02:36:22 -0700, Andrew Dalke <[EMAIL PROTECTED]> wrote: [snip] > >2) asyncore is > >smaller and easier to understand than Twisted, > > While I hear this a lot, applications written with Twisted _are_ shorter and > contain less irrelevant noise in the form of boilerplate than the equivalent > asyncore programs. This may not mean that Twisted programs are easier to > understand, but it is at least an objectively measurable metric. In my experience, the boilerplate is generally incoming and outgoing buffers. If both had better (optional default) implementations, and perhaps a way of saying "use the default implementations of handle_close, etc.", then much of the boilerplate would vanish. People would likely implement a found_terminator method and be happy. > >and > >3) it was for demo/proof of concept purposes. > >While > >tempting to improve that module I know that Twisted > >has already gone though all the platform-specific crap > >and I don't want to go through it again myself. I don't > >want to write a reactor to deal with GTK, and one for > >OS X, and one for ... > > Now if we can only figure out a way for everyone to benefit from this without > tying too many brains up in knots. :) Whenever I need to deal with these kinds of things (in wxPython specifically), I usually set up a wxTimer to signal asyncore.poll(timeout=0), but I'm lazy, and rarely need significant throughput in my GUI applications. [snip] > >Yet by using the Stackless socket monkeypatch, this > >same code works in an async framework. And the underlying > >libraries have a much larger developer base than Twisted. > >Want NNTP? "import nntplib" Want POP3? "import poplib" > >Plenty of documentation about them too. > > This is going to come out pretty harshly, for which I can only apologize in > advance, but it bears mention. The quality of protocol implementations in the > standard library is bad. As in "not good". Twisted's NNTP support is better > (even if I do say so myself - despite only having been working on by myself, > when I knew almost nothing about Twisted, and having essentially never been > touched since). Twisted's POP3 support is fantastically awesome. Next to > imaplib, twisted.mail.imap4 is a sparkling diamond. And each of these > implements the server end of the protocol as well: you won't find that in the > standard library for almost any protocol. Protocol support is hit and miss. NNTP in Python could be better, but that's not an asyncore issue (being that nntplib isn't implemented using asyncore), that's an "NNTP in Python could be done better" issue. Is it worth someone's time to patch it, or should they just use Twisted? Well, if we start abandoning stdlib modules, "because they can always use Twisted", then we may as well just ship Twisted with Python. - Josiah ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Wrapping up 'dynamic attribute' discussion
I've sent an updated version of PEP 363 to the editors, which
includes the following summary of the discussion. I hope I've
captured the important points, but please let me know if there's
something important I've left out or misrepresented.
- - - - 8< - - - -
Mailing Lists Discussion
Initial posting of this PEP in draft form was to python-ideas on
20070209 [2], and the response was generally positive. The PEP was
then posted to python-dev on 20070212 [3], and an interesting
discussion ensued. A brief summary:
Initially, there was reasonable (but not unanimous) support for the
idea, although the precise choice of syntax had a more mixed
reception. Several people thought the "." would be too easily
overlooked, with the result that the syntax could be confused with a
method/function call. A few alternative syntaxes were suggested:
obj.(foo)
obj.[foo]
obj.{foo}
obj{foo}
obj.*foo
obj->foo
obj<-foo
[EMAIL PROTECTED]
obj.[[foo]]
with "obj.[foo]" emerging as the preferred one. In this initial
discussion, the two-argument form was universally disliked, so it
was to be taken out of the PEP.
Discussion then took a step back to whether this particular feature
provided enough benefit to justify new syntax. As well as requiring
coders to become familiar with the new syntax, there would also be
the problem of backward compatibility --- code using the new syntax
would not run on older pythons.
Instead of new syntax, a new "wrapper class" was proposed, with the
following specification / conceptual implementation suggested by
Martin Loewis:
class attrs:
def __init__(self, obj):
self.obj = obj
def __getitem__(self, name):
return getattr(self.obj, name)
def __setitem__(self, name, value):
return setattr(self.obj, name, value)
def __delitem__(self, name):
return delattr(self, name)
def __contains__(self, name):
return hasattr(self, name)
This was considered a cleaner and more elegant solution to the
original problem. The decision was made that the present PEP did
not meet the burden of proof for the introduction of new syntax, a
view which had been put forward by some from the beginning of the
discussion. The wrapper class idea was left open as a possibility
for a future PEP.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
> On Thu, 15 Feb 2007 10:46:05 -0500, "A.M. Kuchling" <[EMAIL PROTECTED]> wrote:
> >It's hard to debug the resulting problem. Which level of the *12*
> >levels in the stack trace is responsible for a bug? Which of the *6*
> >generic calls is calling the wrong thing because a handler was set up
> >incorrectly or the wrong object provided? The code is so 'meta' that
> >it becomes effectively undebuggable.
On 2/15/07, Jean-Paul Calderone <[EMAIL PROTECTED]> wrote,
> I've debugged plenty of Twisted applications. So it's not undebuggable. :)
Hence the word "effectively". Or are you offering to be on-call
within 5 minutes for anyone wanting to debug code? Not very
scalable that.
The code I was talking about took me an hour to track down
and I could only find the location be inserting a "print traceback"
call to figure out where I was.
> Application code tends to reside at the bottom of the call stack, so Python's
> traceback order puts it right where you're looking, which makes it easy to
> find.
As I also documented, Twisted tosses a lot of the call stack. Here
is the complete and full error message I got:
Error: [Failure instance: Traceback (failure with no frames):
twisted.internet.error.ConnectionRefusedError: Connection was refused
by other side: 22: Invalid argument.
]
I wrote the essay at
http://www.dalkescientific.com/writings/diary/archive/2006/08/28/levels_of_abstraction.html
to, among others, show just how hard it is to figure things
out in Twisted.
> For any bug which causes something to be set up incorrectly and only
> later manifests as a traceback, I would posit that whether there is 1 frame or
> 12, you aren't going to get anything useful out of the traceback.
I posit that tracebacks are useful.
Consider:
def blah():
make_network_request("A")
make_network_request("B")
where "A" and "B" are encoded as part of a HTTP POST payload
to the same URI.
If there's an error in the network connection - eg, the implementation
for "B" on the server dies so the connection closes w/o a response -
then knowning that the call for "B" failed and not "A" is helpful
during debugging.
The low level error message cannot report that. Yes, I could put
my own try blocks around everything and contextualize all of the
error messages so they are semantically correct for the given
level of code. But that I would be a lot of code, hard to test, and
not cost effective.
> Standard practice here is just to make exception text informative,
> I think,
If you want to think of it as "exception text" then consider that
the stack trace is "just" more text for the message.
>but this is another general problem with Python programs
> and event loops, not one specific to either Twisted itself or the
> particular APIs Twisted exposes.
The thread is "Twisted Isn't Specific", as a branch of a discussion
on microthreads in the standard library. As someone experimenting
with Stackless and how it can be used on top of an async library
I feel competent enough to comment on the latter topic.
As someone who has seen the reverse Bozo bit set by Twisted
people on everyone who makes the slightest comment towards
using any other async library, and has documented evidence as
to just why one might do so, I also feel competent enough to
comment on the branch topic.
My belief is that there are too many levels of generiticity in
Twisted. This makes is hard for an outsider to come in and
use the system. By "use" I include 1) understanding how the
parts go together, 2) diagnose problems and 3) adding new
features that Twisted doesn't support.
Life is trade offs. A Twisted trade off is generiticity at the
cost of understandability. Remember, this is all my belief,
backed by examples where I did try to understand. My
experience with other networking packages have been much
easier, including with asyncore and Allegra. They are not
as general purpose, but it's hard for me to believe the extra
layers in Twisted are needed to get that extra whatever
functionality.
My other belief is that async programming is hard for most
people, who would rather do "normal" programming instead
of "inside-out" programming. Because of this 2nd belief I
am interested in something like Stackless on top of an
async library.
> As a personal anecdote, I've never once had to chase a bug through any of the
> 6 "generic calls" singled out. I can't think of a case where I've helped any
> one else who had to do this, either. That part of Twisted is very old, it is
> _very_ close to bug-free, and application code doesn't have very much control
> over it at all. Perhaps in order to avoid scaring people, there should be a
> way to elide frames from a traceback (I don't much like this myself, I worry
> about it going wrong and chopping out too much information, but I have heard
> other people ask for it)?
Even though I said some of this earlier I'll elaborate for clarification.
The specific bug I was tracking down had *no* traceback. There w
[Python-Dev] Summary: rejection of 'dynamic attribute' syntax
[EMAIL PROTECTED] wrote:
> > > I really, really wish that every feature proposal for Python had
to meet
> > > some burden of proof
Ben North wrote:
> > This is what I understood the initial posting to python-ideas to be
> > about.
[EMAIL PROTECTED] wrote:
> I'm suggesting that the standards of the community in _evaluating_ to
> the proposals should be clearer
Perhaps I didn't need to take your initial comments personally then,
sorry :-)
I do see what you're pointing out: the later part of the "dynamic
attribute" discussion was where the question of whether python really
needs new syntax for this was addressed, and the outcome made the
earlier discussion of "x.[y]" vs "x.(y)" vs "x.{y}" vs "x->y"
etc. irrelevant.
Ben.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
On 2/15/07, Raymond Hettinger <[EMAIL PROTECTED]> wrote: > * Add a pure python named_tuple class to the collections module. I've been > using the class for about a year and found that it greatly improves the > usability of tuples as records. > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 Hm, but why would they still have to be tuples? Why not just have a generic 'record' class? > * Give builtin types a __name__ attribute so that we have a uniform way of > accessing type names. This already exists, unless I misunderstand you: Python 2.2.3+ (#94, Jun 4 2003, 08:24:18) [GCC 2.96 2731 (Red Hat Linux 7.3 2.96-113)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> int.__name__ 'int' >>> > * Enhance doctest with a method that computes updated doctest results. If the > only change I make to a matrix suite is to add commas to long numbers in the > matrix repr(), then it would be nice to have an automated way to update the > matrix output on all the other test cases in the module. What I could have used for Py3k (specifically for the 2to3 transformer): mods to the DocTestParser class that allow you to exactly reproduce the input string from the return value of parse(). Unrelated, I'd like the tokenize module to be more easily extensible. E.g. I'd like to add new tokens, and I'd like to be able to change its whitespace handling. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
Guido van Rossum wrote: > On 2/15/07, Raymond Hettinger <[EMAIL PROTECTED]> wrote: >> * Add a pure python named_tuple class to the collections module. >> I've been using the class for about a year and found that it greatly >> improves the usability of tuples as records. >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 > > Hm, but why would they still have to be tuples? Why not just have a > generic 'record' class? Hmm - possibilities. "record" definitely has greater connotations of heterogeneous elements than "tuple", which would put paid to the constant arguments that "a tuple is really just an immutable list". list - primarily intended for homogeneous elements record - primarily intended for heterogeneous elements, elements are (optionally?) named and have mutable and immutable versions of each. Maybe the current list syntax would then continue to create a mutable list, and the current tuple syntax would create an immutable record (with no element names) i.e. the current tuple. Tim Delaney ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Martin v. Löwis wrote: > That is insufficient. The gtk main loop has more input > sources than just the connection to X: > - timers > - idle handlers > - child handlers > - additional file descriptors > - a generalzed 'event source' When gtk is not the central event mechanism, there's no need to use the gtk event loop for these things -- you can just use the central event mechanism directly. The pygtk APIs for setting these up can redirect them to the appropriate place, to accommodate existing code that uses the gtk event loop for them. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
>> Hm, but why would they still have to be tuples? Why not just have a >> generic 'record' class? Tim> Hmm - possibilities. "record" definitely has greater connotations Tim> of heterogeneous elements than "tuple", which would put paid to the Tim> constant arguments that "a tuple is really just an immutable list". (What do you mean by "... put paid ..."? It doesn't parse for me.) Based on posts the current thread in c.l.py with the improbable subject "f---ing typechecking", lots of people refuse to believe tuples are anything other than immutable lists. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
Oleg Broytmann wrote: > > Given that they say "a camel is a horse designed by a committee" > > BTW, camels are very suited for their environments... The quote is actually "a camel is a *racehorse* designed by a committee". Camels are very good at surviving in the desert, but not so good at winning a horse race (not camel race). Which is the point of the saying. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
[EMAIL PROTECTED] wrote: > >> Hm, but why would they still have to be tuples? Why not just > have a >> generic 'record' class? > > Tim> Hmm - possibilities. "record" definitely has greater > connotations Tim> of heterogeneous elements than "tuple", which > would put paid to the Tim> constant arguments that "a tuple is > really just an immutable list". > > (What do you mean by "... put paid ..."? It doesn't parse for me.) > Based on posts the current thread in c.l.py with the improbable > subject "f---ing typechecking", lots of people refuse to believe > tuples are anything other than immutable lists. Sorry - "put paid to" means "to finish" ... http://www.phrases.org.uk/meanings/293200.html That thread is a perfect example why I think a "record" type should be standard in python, and "tuple" should be deprecated (and removed in 3.0). Instead, have mutable and immutable lists, and mutable and immutable records. You could add a mutable list and an immutable list (resulting always in a new mutable list I think). You could *not* add two records together (even if neither had named elements). Cheers, Tim Delaney ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
Greg Ewing wrote: > Oleg Broytmann wrote: > >>> Given that they say "a camel is a horse designed by a committee" >> >> BTW, camels are very suited for their environments... > > The quote is actually "a camel is a *racehorse* designed by a > committee". Camels are very good at surviving in the desert, but not > so good at winning a horse race (not camel race). Which is the point > of the saying. Speaking of which, have you ever seen a camel race? Those things go *fast* ... I think we're getting way too off-topic now ;) Tim Delaney ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax
Anthony Baxter wrote: > Unless the fans are perfectly balanced, small changes in gravity are > going to affect the rate at which they spin. So I guess the > position of the moon will affect it :-) A standard gravitational field could also be important to eliminate relativistic effects. So we need to standardise latitude/longitude/altitude, time of year and phase of moon. Better check atmospheric pressure and humidity, too, just to be on the safe side. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] generic async io (was: microthreading vs. async io)
Nick Maclaren wrote: > Threading > - > > An I/O operation passes a buffer, length, file and action and receives a > token back. You seem to be using the word "threading" in a completely different way than usual here, which may be causing some confusion. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Wrapping up 'dynamic attribute' discussion
On Friday 16 February 2007 09:08, Ben North wrote: > Instead of new syntax, a new "wrapper class" was proposed, > with the following specification / conceptual implementation > suggested by Martin Loewis: > > ...snip... > > This was considered a cleaner and more elegant solution to > the original problem. The decision was made that the present PEP > did not meet the burden of proof for the introduction of new > syntax, a view which had been put forward by some from the > beginning of the discussion. The wrapper class idea was left > open as a possibility for a future PEP. A good first step would be to contribute something like this to the Python Cookbook, if it isn't already there. -- Anthony Baxter <[EMAIL PROTECTED]> It's never too late to have a happy childhood. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
Greg Ewing wrote: > Oleg Broytmann wrote: > >>> Given that they say "a camel is a horse designed by a committee" >> BTW, camels are very suited for their environments... > > The quote is actually "a camel is a *racehorse* designed by a committee". > Camels are very good at surviving in the desert, but not so good at > winning a horse race (not camel race). Which is the point of the saying. > As far as I know Sir Alec Issigonis, the inventor of the Mini (the car, not the Mac Mini) said this, and he used "horse", not "racehorse". The point of the saying is that a camel has properties that are completely unnecessary in a horse, such as the ability to travel many days without water. He was saying that committees tend to over-specify and add redundant features rather than designing strictly for purpose. A bit like Python 3.0 ;-) regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
On Thu, Feb 15, 2007 at 05:41:51PM -0600, [EMAIL PROTECTED] wrote: > Tim> Hmm - possibilities. "record" definitely has greater connotations > Tim> of heterogeneous elements than "tuple", which would put paid to the > Tim> constant arguments that "a tuple is really just an immutable list". > > (What do you mean by "... put paid ..."? It doesn't parse for me.) "Put paid" usually means "to finish off"; Tim is saying this would finish the constant arguments that &c &c... --amk ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
[Raymond Hettinger]
>>> * Add a pure python named_tuple class to the collections module.
>>> I've been using the class for about a year and found that it greatly
>>> improves the usability of tuples as records.
>>> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261
[Delaney, Timothy]
> Hmm - possibilities. "record" definitely has greater connotations of
> heterogeneous elements than "tuple", which would put paid to the
> constant arguments that "a tuple is really just an immutable list".
No need to go so widely off-track. The idea is to have an efficient type that
is directly substitutable for tuples but is a bit more self-descriptive. I
like
to have the doctest result cast at NamedTuple('TestResults failed attempted).
The repr of that result looks like TestResult(failed=0, attempted=15) but is
still accessible as a tuple and passes easily into other functions that expect
a
tuple. This sort of thing would be handly for things like os.stat().
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261
Raymond
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
On 15/02/2007 20.59, Raymond Hettinger wrote: > * Add a pure python named_tuple class to the collections module. I've been > using the class for about a year and found that it greatly improves the > usability of tuples as records. > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 +1 from me too, I've been using a class with the same name and semantic (though an inferior implementation) for almost two years now, with great benefits. As suggested in the cookbook comment, please consider changing the semantic of the generated constructor so that it accepts a single iterable positional arguments (or keyword arguments). This matches tuple() (and other containers) in behaviour, and makes it easier to substitute existing uses with named tuples. -- Giovanni Bajo ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, 15 Feb 2007 13:55:31 -0800, Josiah Carlson <[EMAIL PROTECTED]> wrote: > >Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > [snip] >> >> Now if we can only figure out a way for everyone to benefit from this without >> tying too many brains up in knots. :) > >Whenever I need to deal with these kinds of things (in wxPython >specifically), I usually set up a wxTimer to signal >asyncore.poll(timeout=0), but I'm lazy, and rarely need significant >throughput in my GUI applications. And I guess you also don't mind that on OS X this is often noticably broken? :) > [snip] > >Protocol support is hit and miss. NNTP in Python could be better, but >that's not an asyncore issue (being that nntplib isn't implemented using >asyncore), that's an "NNTP in Python could be done better" issue. Is it >worth someone's time to patch it, or should they just use Twisted? Well, >if we start abandoning stdlib modules, "because they can always use >Twisted", then we may as well just ship Twisted with Python. > We could always replace the stdlib modules with thin compatibility layers based on the Twisted protocol implementations. It's trivial to turn an asynchronous API into a synchronous one. I think you are correct in marking this an unrelated issue, though. Jean-Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Wrapping up 'dynamic attribute' discussion
On 2/15/07, Anthony Baxter <[EMAIL PROTECTED]> wrote: > On Friday 16 February 2007 09:08, Ben North wrote: > > The wrapper class idea was left > > open as a possibility for a future PEP. > > A good first step would be to contribute something like this to the > Python Cookbook, if it isn't already there. I could not find such a class in the cookbook. (That's not to say there's not one there that I missed.) Because I think attrview() should happen, I submitted a recipe to the Python Cookbook. While it awaits editor approval, I have it posted at http://www.verylowsodium.com/attrview.py . One possibly controversial design choice here: since there is no guaranteed way to enumerate attributes in the face of __getattr__ and friends, my version of attrview() does not provide iteration or any other operation that assumes object attributes can be enumerated. Therefore, it technically does not implement a mapping. Once Python grows a __dir__ special method, it's possible I could be convinced this is the wrong choice, but I'm not sure of that. Greg F ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > > On Thu, 15 Feb 2007 13:55:31 -0800, Josiah Carlson <[EMAIL PROTECTED]> wrote: > > > >Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > > [snip] > >> > >> Now if we can only figure out a way for everyone to benefit from this > >> without > >> tying too many brains up in knots. :) > > > >Whenever I need to deal with these kinds of things (in wxPython > >specifically), I usually set up a wxTimer to signal > >asyncore.poll(timeout=0), but I'm lazy, and rarely need significant > >throughput in my GUI applications. > > And I guess you also don't mind that on OS X this is often noticably broken? > :) I don't own a Mac, and so far, of the perhaps dozen or so Mac users of the software that does this, I've heard no reports of it being broken. >From what I understand, wxTimers work on all supported platforms (which includes OSX), and if asyncore.poll() is broken on Macs, then someone should file a bug report. If it's asyncore's fault, assign it to me, otherwise someone with Mac experience needs to dig into it. > > [snip] > >Protocol support is hit and miss. NNTP in Python could be better, but > >that's not an asyncore issue (being that nntplib isn't implemented using > >asyncore), that's an "NNTP in Python could be done better" issue. Is it > >worth someone's time to patch it, or should they just use Twisted? Well, > >if we start abandoning stdlib modules, "because they can always use > >Twisted", then we may as well just ship Twisted with Python. > > We could always replace the stdlib modules with thin compatibility layers > based on the Twisted protocol implementations. It's trivial to turn an > asynchronous API into a synchronous one. I think you are correct in marking > this an unrelated issue, though. If the twisted folks (or anyone else) want to implement a "shim" that pretends to be nntplib, it's their business whether calling twisted.internet.monkeypatch.nntplib() does what the name suggests. :) That is to say, I don't believe anyone would be terribly distraught if there was an easy way to use Twisted without drinking the kool-aid. Then again, I do believe that it makes sense to patch the standard library whenever possible - if Twisted has better parsing of nntp, smtp, pop3, imap4, etc. responses, then perhaps we should get the original authors to sign a PSF contributor agreement, and we could translate whatever is better. - Josiah ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Larry Hastings schrieb: > Oof! I'm embarrassed to have forgotten that. But that's not a fatal > problem. It means that on Windows the PerfectReactor must service the > blocking GetMessage loop, and these other threads notify the > PerfectReactor of new events by sending a message. ... > I'd be interested to hear about other situations where threading would > cause a problem. My suspicion is that Windows is the hard one, and as > I've shown that one is solvable. As others have reported: if you have a gtk reactor and a wx reactor to support, in addition to a "regular" Win32 reactor (e.g. PythonWin), your approach still won't work. They all do GetMessage behind the scenes (sp?), yet you would run them in multiple threads, so they get their own messages. Plus they are not thread-safe. Speaking of which: Tcl has a "multiple interpreter" model where each interpreter is single-threaded. You can build Tcl either single- or multi-threaded; for the multi-threaded Tcl, you still can't share data across threads directly, but instead, you create an interpreter per thread. Then, all operations on that interpreter must occur in this thread, including the reactor calls. So if the PerfectReactor is to support Tcl/Tk, it needs to run the Tcl even loop in the Tcl thread, which is normally the main thread (in Tkinter, it's the thread where you originally created the tkapp object). OTOH, PerfectReactor may also need to run some Win32 event loop in the same thread. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py2.6 ideas
Raymond Hettinger schrieb:
> No need to go so widely off-track. The idea is to have an efficient type
> that
> is directly substitutable for tuples but is a bit more self-descriptive. I
> like
> to have the doctest result cast at NamedTuple('TestResults failed attempted).
> The repr of that result looks like TestResult(failed=0, attempted=15) but is
> still accessible as a tuple and passes easily into other functions that
> expect a
> tuple. This sort of thing would be handly for things like os.stat().
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261
I'd like to repeat Guido's question: Why does this still need to support
the tuple interface (i.e. indexed access)?
I'm not (anymore) sure that you are aware that the os.stat result
*already* has named fields, in addition to the indexed access. However,
the indexed access is deprecated, and only preserved for backwards
compatibility. So why would a new type be handy for os.stat?
And, if it's not for os.stat, what other uses does it have?
Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
"Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > Larry Hastings schrieb: > > Oof! I'm embarrassed to have forgotten that. But that's not a fatal > > problem. It means that on Windows the PerfectReactor must service the > > blocking GetMessage loop, and these other threads notify the > > PerfectReactor of new events by sending a message. > ... > > I'd be interested to hear about other situations where threading would > > cause a problem. My suspicion is that Windows is the hard one, and as > > I've shown that one is solvable. > > As others have reported: if you have a gtk reactor and a wx reactor > to support, in addition to a "regular" Win32 reactor (e.g. PythonWin), > your approach still won't work. They all do GetMessage behind the > scenes (sp?), yet you would run them in multiple threads, so they > get their own messages. Plus they are not thread-safe. At least on the wx side of things, there exist ways of scheduling calls to be executed inside its event dispatching loop (wx.CallAfter, wx.FutureCall, wx.PostEvent), which allows for worker threads to hand data back to the main thread (the creator of the wx.App instance, like Tk). I would be surprised if the other toolkits didn't have something similar, couldn't be manipulated into using Queues to communicate in some way, or couldn't be manipulated into directly sending a message on win32. - Josiah ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
