Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Wed, Jan 9, 2013 at 12:14 AM, Guido van Rossum wrote: > But which half? A socket is two independent streams, one in each > direction. Twisted uses half_close() for this concept but unless you > already know what this is for you are left wondering which half. Which > is why I like using 'write' in the name. FWIW, "half-closed" is, IMHO, a well-known term. It's not just a Twisted thing. Either name is better than "shutdown"! Dustin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
Well, if we're at the "bikeshedding about names" stage, that means that no serious issues with the proposal are left. So it's a sign of progress. On Wed, Jan 9, 2013 at 12:42 AM, Stephen J. Turnbull wrote: > Is this thread really ready to migrate to python-dev when we're still > bikeshedding method names? > > Yuriy Taraday writes: > > > > But which half? A socket is two independent streams, one in each > > > direction. Twisted uses half_close() for this concept but unless you > > > already know what this is for you are left wondering which half. Which > > > is why I like using 'write' in the name. > > > > Yes, 'write' part is good, I should mention it. I meant to say that I > won't > > need to explain that there were days when we had to handle a special > marker > > at the end of file. > > Mystery is good for students. > > Getting serious, "close_writer" occured to me as a possibility. > ___ > Python-ideas mailing list > python-id...@python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- Jasper ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Wed, Jan 9, 2013 at 8:55 PM, Yuriy Taraday wrote: > My interns told me that they remember EOF as special object only from high > school when they had to study Pascal. I guess, in 5 years students won't > understand how one can write an EOF. (and schools will finally replace > Pascal with Python) Python really doesn't try to avoid the concept of an End-of-file marker. $ python3 Python 3.2.3 (default, Jun 8 2012, 05:36:09) [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> quit Use quit() or Ctrl-D (i.e. EOF) to exit >>> import io >>> print(io.FileIO.read.__doc__) read(size: int) -> bytes. read at most size bytes, returned as bytes. Only makes one system call, so less data may be returned than requested In non-blocking mode, returns None if no data is available. On end-of-file, returns ''. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
Is this thread really ready to migrate to python-dev when we're still bikeshedding method names? Yuriy Taraday writes: > > But which half? A socket is two independent streams, one in each > > direction. Twisted uses half_close() for this concept but unless you > > already know what this is for you are left wondering which half. Which > > is why I like using 'write' in the name. > > Yes, 'write' part is good, I should mention it. I meant to say that I won't > need to explain that there were days when we had to handle a special marker > at the end of file. Mystery is good for students. Getting serious, "close_writer" occured to me as a possibility. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
As the maintainer of a pretty large, complex app written in Twisted, I think this is great. I look forward to a future of being able to select from a broad library of async tools, and being able to write tools that can be used outside of Twisted. Buildbot began, lo these many years ago, doing a lot of things in memory on on local disk, neither of which require asynchronous IO. So a lot of API methods did not originally return Deferreds. Those methods are then used by other methods, many of which also do not return Deferreds. Now, we want to use a database backend, and parallelize some of the operations, meaning that the methods need to return a Deferred. Unfortunately, that requires a complete tree traversal of all of the methods and methods that call them, rewriting them to take and return Deferreds. There's no "halfway" solution. This is a little easier with generators (@inlineCallbacks), since the syntax doesn't change much, but it's a significant change to the API (in fact, this is a large part of the reason for the big rewrite for Buildbot-0.9.x). I bring all this up to say, this PEP will introduce a new "kind" of method signature into standard Python, one which the caller must know, and the use of which changes the signature of the caller. That can cause sweeping changes, and debugging those changes can be tricky. Two things can help: First, `yield from somemeth()` should work fine even if `somemeth` is not a coroutine function, and authors of async tools should be encouraged to use this form to assist future-compatibility. Second, `somemeth()` without a yield should fail loudly if `somemeth` is a coroutine function. Otherwise, the effects can be pretty confusing. In http://code.google.com/p/uthreads, I accomplished the latter by taking advantage of garbage collection: if the generator is garbage collected before it's begun, then it's probably not been yielded. This is a bit gross, but good enough as a debugging technique. On the topic of debugging, I also took pains to make sure that tracebacks looked reasonable, filtering out scheduler code[1]. I haven't looked closely at Tulip to see if that's a problem. Most of the "noise" in the tracebacks came from the lack of 'yield from', so it may not be an issue at all. Dustin [1] http://code.google.com/p/uthreads/source/browse/trunk/uthreads/core.py#253 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Fri, Jan 4, 2013 at 2:38 PM, Dustin Mitchell wrote: > As the maintainer of a pretty large, complex app written in Twisted, I think > this is great. I look forward to a future of being able to select from a > broad library of async tools, and being able to write tools that can be used > outside of Twisted. Thanks. Me too. :-) > Buildbot began, lo these many years ago, doing a lot of things in memory on > on local disk, neither of which require asynchronous IO. So a lot of API > methods did not originally return Deferreds. Those methods are then used by > other methods, many of which also do not return Deferreds. Now, we want to > use a database backend, and parallelize some of the operations, meaning that > the methods need to return a Deferred. Unfortunately, that requires a > complete tree traversal of all of the methods and methods that call them, > rewriting them to take and return Deferreds. There's no "halfway" solution. > This is a little easier with generators (@inlineCallbacks), since the syntax > doesn't change much, but it's a significant change to the API (in fact, this > is a large part of the reason for the big rewrite for Buildbot-0.9.x). > > I bring all this up to say, this PEP will introduce a new "kind" of method > signature into standard Python, one which the caller must know, and the use > of which changes the signature of the caller. That can cause sweeping > changes, and debugging those changes can be tricky. Yes, and this is the biggest unproven point of the PEP. (The rest is all backed by a decade or more of experience.) > Two things can help: > > First, `yield from somemeth()` should work fine even if `somemeth` is not a > coroutine function, and authors of async tools should be encouraged to use > this form to assist future-compatibility. Second, `somemeth()` without a > yield should fail loudly if `somemeth` is a coroutine function. Otherwise, > the effects can be pretty confusing. That would be nice. But the way yield from and generators work, that's hard to accomplish without further changes to the language -- and I don't want to have to change the language again (at least not immediately -- maybe in a few releases, after we've learned what the real issues are). The best I can do for the first requirement is to define @coroutine in a way that if the decorated function isn't a generator, it is wrapped in one. For the second requirement, if you call somemeth() and ignore the result, nothing happens at all -- this is indeed infuriating but I see no way to change this.(*) If you use the result, well, Futures have different attributes than most other objects so hopefully you'll get a loud AttributeError or TypeError soon, but of course if you pass it into something else which uses it, it may still be difficult to track. Hopefully these error messages provide a hint: >>> f.foo Traceback (most recent call last): File "", line 1, in AttributeError: 'Future' object has no attribute 'foo' >>> f() Traceback (most recent call last): File "", line 1, in TypeError: 'Future' object is not callable >>> (*) There's a heavy gun we might use, but I would make this optional, as a heavy duty debugging mode only. @coroutine could wrap generators in a lightweight object with a __del__ method and an __iter__ method. If __del__ is called before __iter__ is ever called, it could raise an exception or log a warning. But this probably adds too much overhead to have it always enabled. > In http://code.google.com/p/uthreads, I accomplished the latter by taking > advantage of garbage collection: if the generator is garbage collected > before it's begun, then it's probably not been yielded. This is a bit > gross, but good enough as a debugging technique. Eh, yeah, what I said. :-) > On the topic of debugging, I also took pains to make sure that tracebacks > looked reasonable, filtering out scheduler code[1]. I haven't looked > closely at Tulip to see if that's a problem. Most of the "noise" in the > tracebacks came from the lack of 'yield from', so it may not be an issue at > all. One of the great advantages of using yield from is that the tracebacks automatically look nice. > Dustin > > [1] > http://code.google.com/p/uthreads/source/browse/trunk/uthreads/core.py#253 -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Fri, Dec 21, 2012 at 8:02 PM, Guido van Rossum wrote: ... snip ... In PEP 3156 conformant code you're supposed always to use 'yield > from'. The only time you see a bare yield is when it's part of the > implementation's internals. (However I think tulip actually will > handle a yield the same way as a yield from, except that it's slower > because it makes a roundtrip to the scheduler, a.k.a. trampoline.) > Would it be possible to fail on "yield"? Silently being slower when you forget to type a keyword is something I can imagine will creep up a lot by mistake, and I don't think it's a good idea to silently be slower when the only different is five more characters. > Will comment more as I keep reading I'm sure :) > > Please do! > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-ideas mailing list > python-id...@python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- Jasper ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Fri, Dec 21, 2012 at 5:17 PM, Jasper St. Pierre wrote: > On Fri, Dec 21, 2012 at 8:02 PM, Guido van Rossum wrote: > > ... snip ... > >> In PEP 3156 conformant code you're supposed always to use 'yield >> from'. The only time you see a bare yield is when it's part of the >> implementation's internals. (However I think tulip actually will >> handle a yield the same way as a yield from, except that it's slower >> because it makes a roundtrip to the scheduler, a.k.a. trampoline.) > > > Would it be possible to fail on "yield"? Silently being slower when you > forget to type a keyword is something I can imagine will creep up a lot by > mistake, and I don't think it's a good idea to silently be slower when the > only different is five more characters. That's also a possibility. If someone can figure out a patch that would be great. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Fri, Dec 21, 2012 at 2:26 PM, Jonathan Slenders wrote: > As far as I understand, "yield from" will always work, because a Future > object can act like an iterator, and you can delegate your own generator to > this iterator at the place of "yield from". > "yield" only works if the parameter behind yield is already a Future object. > Right Guido? Correct! Sounds like you got it now. That's the magic of yield from.. > In case of sleep, sleep could be implemented to return a Future object. It does; in tulip/futures.py: def sleep(when, result=None): future = Future() future._event_loop.call_later(when, future.set_result, result) return future -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Fri, Dec 21, 2012 at 1:04 PM, Laurens Van Houtven <_...@lvh.cc> wrote: > Looks reasonable to me :) Comments: > > create_transport "combines" a transport and a protocol. Is that process > reversible? that might seem like an exotic thing (and I guess it kind of > is), but I've wanted this e.g for websockets, and I guess there's a few > other cases where it could be useful :) If you really need this, it's probably best to start out doing this as a nonstandard extension of an implementation. The current *implementation* makes it simple enough, but I don't think it's worth complicating the PEP. Working code might convince me otherwise. > eof_received on protocols seems unusual. What's the rationale? Well how else would you indicate that the other end did a half-close (in Twisted terminology)? You can't call connection_lost() because you might still want to write more. E.g. this is how HTTP servers work if there's no Content-length or chunked encoding on a request body: they read until EOF, then do their thing and write the response. > I know we disagree that callbacks (of the line_received variety) are a good > idea for blocking IO (I think we should have universal protocol > implementations), but can we agree that they're what we want for tulip? If > so, I can try to figure out a way to get them to fit together :) I'm > assuming that this means you'd like protocols and transports in this PEP? Sorry, I have no idea what you're talking about. Can you clarify? I do know that the PEP is weakest in specifying how a coroutine can implement a transport. However my plans are clear: ild the old tulip code there's a BufferedReader; somehow the coroutine will receive a "stdin" and a "stdout" where the "stdin" is a BufferedReader, which has methods like read(), readline() etc. which return Futures and must be invoked using yield from; and "stdout" is a transport, which has write() and friends that don't return anything but just buffer stuff and start the I/O asynchronous (and may try to slow down the protocol by calling its pause() method). > A generic comment on yield from APIs that I'm sure has been discussed in > some e-mail I missed: is there an obvious way to know up front whether > something needs to be yielded or yield frommed? In twisted, which is what > I'm used to it's all deferreds; but here a future's yield from but sleep's > yield? In PEP 3156 conformant code you're supposed always to use 'yield from'. The only time you see a bare yield is when it's part of the implementation's internals. (However I think tulip actually will handle a yield the same way as a yield from, except that it's slower because it makes a roundtrip to the scheduler, a.k.a. trampoline.) > Will comment more as I keep reading I'm sure :) Please do! -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
As far as I understand, "yield from" will always work, because a Future object can act like an iterator, and you can delegate your own generator to this iterator at the place of "yield from". "yield" only works if the parameter behind yield is already a Future object. Right Guido? In case of sleep, sleep could be implemented to return a Future object. 2012/12/21 Laurens Van Houtven <_...@lvh.cc> > A generic comment on yield from APIs that I'm sure has been discussed in > some e-mail I missed: is there an obvious way to know up front whether > something needs to be yielded or yield frommed? In twisted, which is what > I'm used to it's all deferreds; but here a future's yield from but sleep's > yield? > > > > On Fri, Dec 21, 2012 at 8:09 PM, Guido van Rossum wrote: > >> On Fri, Dec 21, 2012 at 11:06 AM, Jesse Noller wrote: >> > I really do like tulip as the name. It's quite pretty. >> >> I chose it because Twisted and Tornado both start with T. But those >> have kind of dark associations; I wanted to offset that with something >> lighter. (OTOH we could use a black tulip as a logo. :-) >> >> Regardless, it's not the kind of name we tend to use for the stdlib. >> It'll probably end up being asynclib or something... >> >> -- >> --Guido van Rossum (python.org/~guido) >> ___ >> Python-ideas mailing list >> python-id...@python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> > > > > -- > cheers > lvh > > ___ > Python-ideas mailing list > python-id...@python.org > http://mail.python.org/mailman/listinfo/python-ideas > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
Looks reasonable to me :) Comments: create_transport "combines" a transport and a protocol. Is that process reversible? that might seem like an exotic thing (and I guess it kind of is), but I've wanted this e.g for websockets, and I guess there's a few other cases where it could be useful :) eof_received on protocols seems unusual. What's the rationale? I know we disagree that callbacks (of the line_received variety) are a good idea for blocking IO (I think we should have universal protocol implementations), but can we agree that they're what we want for tulip? If so, I can try to figure out a way to get them to fit together :) I'm assuming that this means you'd like protocols and transports in this PEP? A generic comment on yield from APIs that I'm sure has been discussed in some e-mail I missed: is there an obvious way to know up front whether something needs to be yielded or yield frommed? In twisted, which is what I'm used to it's all deferreds; but here a future's yield from but sleep's yield? Will comment more as I keep reading I'm sure :) On Fri, Dec 21, 2012 at 8:09 PM, Guido van Rossum wrote: > On Fri, Dec 21, 2012 at 11:06 AM, Jesse Noller wrote: > > I really do like tulip as the name. It's quite pretty. > > I chose it because Twisted and Tornado both start with T. But those > have kind of dark associations; I wanted to offset that with something > lighter. (OTOH we could use a black tulip as a logo. :-) > > Regardless, it's not the kind of name we tend to use for the stdlib. > It'll probably end up being asynclib or something... > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-ideas mailing list > python-id...@python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- cheers lvh ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
