Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 3 May 2015 at 16:24, Steven D'Aprano st...@pearwood.info wrote: On Fri, May 01, 2015 at 09:24:47PM +0100, Arnaud Delobelle wrote: I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion. In what way? I'm thinking that the only confusion would be if you wrote async for instead of for, or vice versa, and instead of getting an exception you got the (a)syncronous behaviour you didn't want. Yes. This is the same kind of confusion that this PEP is trying hard to get rid of in other parts (i.e. the confusion between 'yield' and 'yield from' in current coroutines). But I have no intuition for how likely it is that you could write an asyncronous for loop, leave out the async, and still have the code do something meaningful. Well if you've made you object both iterable and 'async iterable' then it's very likely that you're going to get something meaningful out of either kind of iteration. Just not the way you want it if you mistakenly left out (or added) the 'async' keyword in your loop. Other than that, I think it would be fine to have an object be both a syncronous and asyncronous iterator. You specify the behaviour you want by how you use it. We can already do that, e.g. unittest's assertRaises is both a test assertion and a context manager. The latter is fine, because there is no danger of mistaking one for the other, unlike iterators and 'async iterators'. But my argument is simply that there is no good reason to aim for the ability to have object conform to both protocols. So it shouldn't be a criterion when looking at the merits of a proposal. I may very well be wrong but I haven't yet seen a compelling case for an object implementing both protocols. Cheers, -- Arnaud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 6 May 2015 at 09:20, Greg Ewing greg.ew...@canterbury.ac.nz wrote: That doesn't cover any of the higher level abstractions like tasks or futures (at least not by those names or with those interfaces). Because a minimal event loop doesn't *need* those. It doesn't *need* them, but as abstractions they allow easier building of reusable higher-level libraries. You can write an event loop with nothing but coroutines, but to build reusable libraries on top of it, you need some common interfaces. In my little scheduler, a task is nothing more than a yield-frommable object sitting on a queue of things to be run. There is no need to wrap it in another object. And there's really no need for the concept of a future at all, except maybe at the boundary between generator-based async code and other things that are based on callbacks. Even then, a future is really just an object that can be passed to yield-from. There is no need for a concrete Future class, it's just a protocol. Agreed, you don't need a Future class, all you need is to agree what reusable code is allowed to do with the core objects you are passing around - that's how duck typing works. The objects *I* can see are futures (in a PEP 492 world, awaitables which may or may not be equivalent in terms of the operations you'd want to focus on) and the event loop itself. In your example, the event loop is implicit (as it's a singleton, you use global functions rather than methods on the loop object) but that's a minor detail. And I don't see where the PEP 492 additions would fit in (OK, replace yield from with await is part of it, but I don't see the rest). That's really all there is to it. The rest is concerned with catching certain kinds of mistakes, and providing convenient syntax for some patterns of using 'await'. So, things you can wait on have one operation - wait for a result. That's OK. You can create such things as coroutines, which is also fine. You may want to create such things explicitly (equivalent to generators vs __iter__) - maybe that's where __aiter__ comes in in PEP 492 and the Future class in asyncio. Again, all fine. You also need operations like schedule a thing to run, which is the event loop interface. Your sample has the following basic event loop methods that I can see: run, schedule, unschedule, and expire_timeslice (that last one may be an implementation detail, but the other 3 seem pretty basic). PEP 492 has nothing to say on the event loop side of things (something that became clear to me during this discussion). There's a lot of asyncio that doesn't seem to me to be IO-related. Specifically the future and task abstractions. I view those as relevant to coroutine programming in Python because they are referenced in any discussion of coroutines (you yield from a future, for example). Only because they've been elevated to prominence by asyncio and its documentation, which I regard as unfortunate. When Guido was designing asyncio, I tried very hard to dissuade him from giving futures such a central place in the model. I saw them as an unnecessary concept that would only clutter up people's thinking. Seeing all the confusion now, I'm more convinced than ever that I was right. :-( Futures seem to me to be (modulo a few details) what awaitables are in PEP 492. I can't see how you can meaningfully talk about event loops in a Python context without having *some* term for things you wait for. Maybe Future wasn't a good name, and maybe the parallel with concurrent.futures.Future wasn't helpful (I think both things were fine, but you may not) but we had to have *some* way of talking about them, and of constructing standalone awaitables. PEP 492 has new, and hopefully better, ways, but I think that awaitables *have* to be central to any model where you wait for things... By the way, it feels to me like I'm now arguing in favour of PEP 492 with a reasonable understanding of what it means. Assuming what I said above isn't complete rubbish, thanks to everyone who's helped me get to this point of understanding through this thread! (And if I haven't understood, that's my fault, and still thanks to everyone for their efforts :-)) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Paul Moore wrote: What about Greg Ewing's example? http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt That doesn't cover any of the higher level abstractions like tasks or futures (at least not by those names or with those interfaces). Because a minimal event loop doesn't *need* those. In my little scheduler, a task is nothing more than a yield-frommable object sitting on a queue of things to be run. There is no need to wrap it in another object. And there's really no need for the concept of a future at all, except maybe at the boundary between generator-based async code and other things that are based on callbacks. Even then, a future is really just an object that can be passed to yield-from. There is no need for a concrete Future class, it's just a protocol. And I don't see where the PEP 492 additions would fit in (OK, replace yield from with await is part of it, but I don't see the rest). That's really all there is to it. The rest is concerned with catching certain kinds of mistakes, and providing convenient syntax for some patterns of using 'await'. There's a lot of asyncio that doesn't seem to me to be IO-related. Specifically the future and task abstractions. I view those as relevant to coroutine programming in Python because they are referenced in any discussion of coroutines (you yield from a future, for example). Only because they've been elevated to prominence by asyncio and its documentation, which I regard as unfortunate. When Guido was designing asyncio, I tried very hard to dissuade him from giving futures such a central place in the model. I saw them as an unnecessary concept that would only clutter up people's thinking. Seeing all the confusion now, I'm more convinced than ever that I was right. :-( In some ways I wish there had been an asyncio library that covered the areas that are fundamentally about IO multiplexing. And a separate library (just async, maybe, although that's now a bad idea as it clashes with a keyword :-)) that covered generic event loop, task and synchronisation areas. As I said before, I don't think it's really possible to factor an event loop into those kinds of parts. You may be able to factor the *API* that way, but any given implementation has to address all the parts at once. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Hi Yury, On 05.05.2015 20:25, Yury Selivanov wrote: We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. gevent has a different answer, but greenlents/stackless is something that will never be merged in CPython and other implementations. I think monkeypatching and the gevent way is wrong. And explicit is better than implicit. I have to clear this. I meant how can we make this async stuff more accessible to the average sync user. Sometime even without writing or knowing how to write coroutines or other async stuff. Let me explain it with a deeper example (Some of it is related to Python 2 and twisted): I had the same problem for a server application using twisted. Provide easy interfaces to my users most not aware of async stuff. My solution was to write my own decorator similar to twisted's @inlineCallbacks. On top of it I added one more level to the decorator and distinguish if it was called from the main thread (also running the mainloop) and other threads. This object usable also as decorator is called task and has some utility methods. And returns a deferred. (in asyncio this would be a Future) Resulting in code like: @task def myAsyncFunction(webaddress): result = yield fetch(webaddress) # only to show sleep example yield task.sleep(0.1) task.Return(result) Usable in a sync context (extra script thread): def do(): content = myAsyncFunction(http://xyz;) or async context: @task def ado(): content = yield myAsyncFunction(http://xyz;) The task decorator has functionality to check if something is called from the main thread (- also a task and async) or it is called from another thread (- sync or script). So this async interface is usable from both worlds. If someone operates async he/she must only know the task decorator and when to yield. If he/she uses it in sync mode nothing special has to be done. To allow all this the server starts the async main loop in the main thread and executes the script in an extra script thread. The user has every time his own thread, also for rpc stuff. The only way to switch into the main loop is to decorate a function as @task, every task is a coroutine and executed in the main thread (also thread of main loop). Benefit of all this: - Easy to write a async task it is marked as one and special stuff belongs to the task object. (task.Return is needed because we are in Python 2) - The normal user executes his stuff in his own thread and he/she can program in sync mode. No problem it is an extra thread and the main loop does not block. - A network client or other stuff has to be written only once, most time this can be a @task in the async world. But this should not care the end user. We don't have to implement all twice once for async and once for the sync world. - Less overhead This is what I mean if I say we must address the bridging problem between the worlds. It think it is the wrong way to divide it in async and sync stuff and duplicate all networking libraries in the sync and async ones. For me the answer is to write one async netowrk library and use it in both, a sync script and in an async main loop. With an easy interface and not forcing the user to know this is an async library I have to do something special. And in future go one step further and use all this combined with PyParallel to solve the multiprocessing problem in Python. (Script in extra thread, mainloop in main thread, executed and managed via PyParallel avoiding the gil) But this is only a vision/dream of mine. And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. networking stuff, and in particular, web, is a huge part of current Python usage. Please don't underestimate that. I do not. But for most they want only use it as a client and the main concern for most is I want to get this web page and not I will implement a web client have to do this async and get it. Regards, Wolfgang ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Guido van Rossum wrote: the bytecode generated for await treats coroutine objects special, just like the bytecode generated for yield-from treats generator objects special. The special behavior they have in common is the presence of send() and throw() methods, I don't think that's quit accurate. Yield-from treats any object having send() and throw() methods the same way it treats a generator -- there's nothing special about the generator *type*. Presumably 'await' is the same. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Paul Moore wrote: It would probably be helpful to have a concrete example of a basic event loop that did *nothing* but schedule tasks. No IO waiting or similar, just scheduling. Take a look at the example I developed when working on the yield-from pep: http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt The first version of the event loop presented there does exactly that, just schedules tasks in a round- robin fashion. Then I gradually add more features to it. Actually, what *is* the minimal event loop interface that is needed for the various task/future mechanisms to work, independently of asyncio? I don't it's possible to answer that question, because there isn't a single answer. The minimal set of features that an event loop needs depends on what you want to achieve with it. Even the notion of just schedules tasks is ambiguous. What does schedule mean? Does it just mean round-robin switching between them, or should they be able to synchronise with each other in some way? Should it be possible for a task to suspend itself for an interval of real world time, or does that come under the heading of I/O (since you're waiting for an external event, i.e. the computer's clock reaching some time)? Etc. And what features of an event loop etc are needed for the PEP, if it's being used outside of asyncio?) I don't think *any* particular event loop features are needed. You can't pick out any of these features as being core. For example, it would be possible to have an event loop that handled socket I/O but *didn't* do round-robin scheduling -- it could just keep on running the same task, even if it yielded, until it blocked waiting for an external event. Such a scheduler would probably be quite adequate for many use cases. It seems to me that the idea of generator-based tasks managed by an event loop is more of a design pattern than something you can write a detailed API specification for. Another problem with the core idea is that you can't start with an event loop that just does scheduling and then add on other features such as I/O *from the outside*. There has to be some point at which everything comes together, which means choosing something like select() or poll() or I/O completion queues, and build that into the heart of your event loop. At that point it's no longer something with a simple core. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Paul Moore wrote: I can't see how you can meaningfully talk about event loops in a Python context without having *some* term for things you wait for. PEP 3152 was my attempt at showing how you could do that. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Tue, May 5, 2015 at 1:39 PM, Paul Moore p.f.mo...@gmail.com wrote: It would probably be helpful to have a concrete example of a basic event loop that did *nothing* but schedule tasks. No IO waiting or similar, just scheduling. I have a gut feeling that event loops are more than just asyncio, but without examples to point to it's hard to keep a focus on that fact. And even harder to isolate what is an event loop mechanism from what is asyncio specific. For example, asyncio.BaseEventLoop has a create_connection method. That's *obviously* not a fundamental aspect of a generic event loop, But call_soon (presumably) is. Having a documented basic event loop interface would probably help emphasise the idea than event loops don't have to be asyncio. (Actually, what *is* the minimal event loop interface that is needed for the various task/future mechanisms to work, independently of asyncio? And what features of an event loop etc are needed for the PEP, if it's being used outside of asyncio?) Twisted has a pretty good taxonomy of event loop methods, in the interfaces at the bottom of this page: http://twistedmatrix.com/documents/15.1.0/core/howto/reactor-basics.html and the comparison matrix at http://twistedmatrix.com/documents/15.1.0/core/howto/choosing-reactor.html The asyncio event loops implement most of these (not the exact interfaces, but the same functionality). Tornado implements FDSet, Time, and part of Threads in the IOLoop itself, with the rest of the functionality coming from separate classes. (You may wonder then why Twisted and asyncio put everything in the core event loop? It's necessary to abstract over certain platform differences, which is one big reason why Tornado has poor support for Windows). -Ben I guess the other canonical event loop use case is GUI system message dispatchers. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Perhaps, but arguing about the nitty-gritty details of something doesn't automatically lead to a clearer understanding of the higher level concept. Discussing how turning a steering wheel in a car might help you grasp how cars turn, but it isn't a requirement to get turn the wheel left to make the car go left. Fair point. If only I could avoid driving into walls :-) Of course, that's based on my perception of one of the goals of the PEP as being make coroutines and asyncio more accessible, If the actual goals are different, my conclusion is invalid. I think the goal is make coroutines easier to use and does not directly relate to asyncio. OK. But in that case, some examples using a non-asyncio toy just schedule tasks event loop might help. Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Yep. ... and so you can't use it with async/await? Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? I believe so; you need something to implement __await__. This is true in any language that implements co-routines. Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes something that the average user can't use. Which in turn further entrenches it as a specialist-only tool. You forgot to append ... yet to that statement. Just because something isn't available out of the box without some effort to support doesn't mean it will never happen, else there would be absolutely no Python 3 users out there. Fair point. Yuri mentioned aiohttp, as well. The one difference between this and Python 2/3, is that here you *have* to have two separate implementations. There's no equivalent of a shared source async and synchronous implementation of requests. So the typical please support Python 3 issue that motivates projects to move forward doesn't exist in the same way. It's not to say that there won't be async versions of important libraries, it's just hard to see how the dynamics will work. I can't see myself raising an issue on cx_Oracle saying please add asyncio support, and I don't know who else I would ask... Co-routine-based asynchronous programming is new to Python, so as a community we don't have it as something everyone learns over time. If you don't come from an area that supports it then it will be foreign to you and not make sense without someone giving you a good tutorial on it. But considering C#, Dart, and Ecmascript 6 (will) have co-routine support -- and those are just the languages I can name off the top of my head -- using the exact same keywords suggests to me that it isn't *that*
Re: [Python-Dev] PEP 492: async/await in Python; version 4
tds333 at gmail.com tds333 at gmail.com writes: Hi, still watching progress here. Read all posts and changes. Everything improved and I know it is a lot of work. Thx for doing this. But I still think this PEP goes to far. [...] We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. [...] Hi Wolfgang, You may want to see what I just posted on python-ideas. What I wrote about is related to several things you mention, and might provide a remedy. -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On May 5, 2015 2:14 PM, Guido van Rossum gu...@python.org wrote: In the PEP 492 world, these concepts map as follows: - Future translates to something with an __await__ method (and asyncio Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__); - asyncio coroutine maps to PEP 492 coroutine object (either defined with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter); - either of the above maps to awaitable. Err, aren't the first and third definitions above identical? Surely we want to say: an async def function is a convenient shorthand for creating a custom awaitable (exactly like how generators are a convenient shorthand for creating custom iterators), and a Future is-an awaitable that also adds some extra methods. -n ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Tue, May 5, 2015 at 3:01 PM, Nathaniel Smith n...@pobox.com wrote: On May 5, 2015 2:14 PM, Guido van Rossum gu...@python.org wrote: In the PEP 492 world, these concepts map as follows: - Future translates to something with an __await__ method (and asyncio Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__); - asyncio coroutine maps to PEP 492 coroutine object (either defined with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter); - either of the above maps to awaitable. Err, aren't the first and third definitions above identical? Surely we want to say: an async def function is a convenient shorthand for creating a custom awaitable (exactly like how generators are a convenient shorthand for creating custom iterators), and a Future is-an awaitable that also adds some extra methods. The current PEP 492 proposal does endow the object returned by calling an async function (let's call it a coroutine object) with an __await__ method. And there's a good reason for this -- the bytecode generated for await treats coroutine objects special, just like the bytecode generated for yield-from treats generator objects special. The special behavior they have in common is the presence of send() and throw() methods, which are used to allow send() and throw() calls on the outer generator to be passed into the inner generator with minimal fuss. (This is the reason why yield from X is *not* equivalent to for x in X: yield x.) @Yury: I have a feeling the PEP could use more clarity here -- perhaps the section Await Expression should explain what the interepreter does for each type of awaitable? -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Tue, May 5, 2015 at 3:14 PM Paul Moore p.f.mo...@gmail.com wrote: On 5 May 2015 at 19:25, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-05 7:27 AM, Wolfgang wrote: Even the discussion on python-dev suggests there is some time needed to finalize all this. I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not. But the terminology discussion appears to revolve around people finding the various concepts involved in asyncio (particularly the new PEP, but also to an extent the existing implementation) confusing. I can confirm, having tried to work through the asyncio docs, that the underlying concepts and how they are explained, are confusing to an outsider. That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of trust us, it makes sense if you need it, everyone else can ignore it. Watch David Beazley's talk from PyCon this year and you can watch him basically re-implement asyncio on stage in under 45 minutes. It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly). Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. I think this ties the concept of adding syntax to Python to make coroutine-based programming easier too much to asyncio; the latter is just an implementation of the former. This PEP doesn't require asyncio beyond the fact that will be what provides the initial event loop in the stdlib. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Perhaps, but arguing about the nitty-gritty details of something doesn't automatically lead to a clearer understanding of the higher level concept. Discussing how turning a steering wheel in a car might help you grasp how cars turn, but it isn't a requirement to get turn the wheel left to make the car go left. Of course, that's based on my perception of one of the goals of the PEP as being make coroutines and asyncio more accessible, If the actual goals are different, my conclusion is invalid. I think the goal is make coroutines easier to use and does not directly relate to asyncio. We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Yep. Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? I believe so; you need something to implement __await__. This is true in any language that implements co-routines. Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes something that the average user can't use. Which in turn further entrenches it as a specialist-only tool. You forgot to append ... yet to that statement. Just because something isn't available out of the box without some effort to support doesn't mean it will never happen, else there would be absolutely no Python 3 users out there. As another example, in Twisted I could use defer_to_thread to integrate Oracle database access into a twisted application (that's what the twisted database stuff did under the hood). Can I do that with asyncio? Will the syntax in the PEP help, hinder or be irrelevant to that? And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. networking stuff, and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Without async versions of requests and similar, how much of a chunk of the networking/web area will asyncio take? (Genuine question, I have no idea). And how much extra will this PEP add? Those may not be fair
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 5 May 2015 at 19:25, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-05 7:27 AM, Wolfgang wrote: Even the discussion on python-dev suggests there is some time needed to finalize all this. I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not. But the terminology discussion appears to revolve around people finding the various concepts involved in asyncio (particularly the new PEP, but also to an extent the existing implementation) confusing. I can confirm, having tried to work through the asyncio docs, that the underlying concepts and how they are explained, are confusing to an outsider. That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of trust us, it makes sense if you need it, everyone else can ignore it. Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Of course, that's based on my perception of one of the goals of the PEP as being make coroutines and asyncio more accessible, If the actual goals are different, my conclusion is invalid. We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes something that the average user can't use. Which in turn further entrenches it as a specialist-only tool. As another example, in Twisted I could use defer_to_thread to integrate Oracle database access into a twisted application (that's what the twisted database stuff did under the hood). Can I do that with asyncio? Will the syntax in the PEP help, hinder or be irrelevant to that? And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. networking stuff, and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Without async versions of requests and similar, how much of a chunk of the networking/web area will asyncio take? (Genuine question, I have no idea). And how much extra will this PEP add? Those may not be fair questions (and even if they are fair, the answers are probably unknowable), but as an outsider, I feel only the costs of the asyncio implementation (a new library that I don't understand, and now a relatively large amount of new syntax and special methods I have to ignore because they don't make sense to me). That's OK, but I think I am being reasonable to ask for some sense of the level of benefits others are getting to balance out the costs I incur. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Paul, On 2015-05-05 3:14 PM, Paul Moore wrote: On 5 May 2015 at 19:25, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-05 7:27 AM, Wolfgang wrote: Even the discussion on python-dev suggests there is some time needed to finalize all this. I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not. But the terminology discussion appears to revolve around people finding the various concepts involved in asyncio (particularly the new PEP, but also to an extent the existing implementation) confusing. I can confirm, having tried to work through the asyncio docs, that the underlying concepts and how they are explained, are confusing to an outsider. I agree. We have to improve asyncio docs in this area. That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of trust us, it makes sense if you need it, everyone else can ignore it. Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Of course, that's based on my perception of one of the goals of the PEP as being make coroutines and asyncio more accessible, If the actual goals are different, my conclusion is invalid. Again, PEP 492 is not only for asyncio. *Any* framework can use it, including Twisted. As for terminology, I view this discussion differently. It's not about the technical details (Python has asymmetric coroutines, that's it), but rather on how to disambiguate coroutines implemented with generators and yield-from, from new 'async def' coroutines. I can't see any fundamental problem with the PEP behind such discussions. We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes something that the average user can't use. Which in turn further entrenches it as a specialist-only tool. There is aiohttp library [1], which provides a client API similar to requests. And if you want to write high performance networking server in python3 you *will* choose asyncio (or gevent/twisted in python2). And PEP 492 is aimed to make this whole async stuff more accessible to an average user. As another example, in Twisted I could use defer_to_thread to integrate Oracle database access into a twisted application (that's what the twisted database stuff did under the hood). Can I do that with asyncio? Will the syntax in the PEP help, hinder or be irrelevant to that? You can use 'loop.run_in_executor' in asyncio. It returns a future that you can await on. You can also provide a nice facade for your Oracle-database code that provides a nice API but uses asyncio thread executor behind the scenes. And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. networking stuff, and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Without async versions of requests and similar, how much of a chunk of the networking/web area will asyncio take? (Genuine question, I have no idea). There are some things (like websockets) that are hard to implement correctly in existing frameworks like django and flask. And these kind of things are becoming more and more important. Languages like Go were designed specifically to allow writing efficient And how much extra will this PEP add? Those may not be fair questions (and even if they are fair, the answers are probably unknowable), but as an outsider, I feel only the costs of the asyncio implementation (a new library that I don't understand, and now a relatively
[Python-Dev] PEP 492: async/await in Python; version 4
Hi, still watching progress here. Read all posts and changes. Everything improved and I know it is a lot of work. Thx for doing this. But I still think this PEP goes to far. 1. To have native coroutines and await, __await__ is very good and useful. Also for beginners to see and mark coroutines are a different concept than generators. Also this is easy to learn. Easier as to explain why a generator is in this case a coroutine and so on. 2. I still don't like to sprinkle async everywhere. At all we don't need it for the first step. We can handle coroutines similar to generators, when there is a await in it then it is one. Same as for yield. Or to be more explicit, if it is marked as one with @coroutine it is enough. But then it makes also sense to do the same for generators with @generator. We should be consistent here. 3. async with is not needed, there are rare use cases for it and every can be written with a try: except: finally: Every async framework lived without it over years. No problem because for the seldom need try: ... was enough. 4. async for can be implemented with a while loop. For me this is even more explicit and clear. Every time I see the async for I try to find out what is done in async manner. For what I do an implicit await ? Also in most of my uses cases it was enough to produce Futures in a normal loop and yield (await) for them. Even for database interfaces. Most of the time they do prefetching under the hood and I don't have to care at all. 5. For async with/for a lot of new special methods are needed all only prefixed with a. Easy to confuse with a for abstract. Often used to prefix abstract classes. Still think __async_iter__, __async_enter__ is better for this. Yes more to write but not so easy to confuse with the sync __iter__ or __enter__, ... And matches more to I must use async to call them. I am not new to the async land, have done a lot of stuff with twisted and even looked at tornado. Also has tried to teach some people the asnc world. This is not easy and you learn most only by doing and using it. For my final conclusion, we should not rush all this into the language. Do it step by step and also help other Python implementations to support it. For now only a very low percentage are at Python 3.4. And if you compare the statistics in PyPi, you see most are still at Python 2.7 and using async frameworks like twisted/tornado. We do such a fundamental language change only because a few people need it for a niche in the whole Python universe? We confuse the rest with new stuff they never need? Even the discussion on python-dev suggests there is some time needed to finalize all this. We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. Don't get me wrong, I like async programming and I think it is very useful. But had to learn not everyone thinks so and most only want to solve there problems in an easy way and not get a new one called async. Now I shut up. Go to my normal mode and be quiet and read. ;-) Regards, Wolfgang ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
(Yury gave similar responses, so (a) I'll just respond here, and (b) it's encouraging that you both responded so quickly with the same message) On 5 May 2015 at 20:44, Brett Cannon br...@python.org wrote: That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of trust us, it makes sense if you need it, everyone else can ignore it. Watch David Beazley's talk from PyCon this year and you can watch him basically re-implement asyncio on stage in under 45 minutes. It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly). I'll watch that - it should be fun. But I have seen things like that before, and I've got an idea how to write an event loop. You're right that it's easy to lose track of the fundamentally simple idea in all the complex discussions. To me that feels like a peculiar failure of the abstraction, in that in some circumstances it makes things feel *more* complex than they are :-) Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. I think this ties the concept of adding syntax to Python to make coroutine-based programming easier too much to asyncio; the latter is just an implementation of the former. This PEP doesn't require asyncio beyond the fact that will be what provides the initial event loop in the stdlib. It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about. It would probably be helpful to have a concrete example of a basic event loop that did *nothing* but schedule tasks. No IO waiting or similar, just scheduling. I have a gut feeling that event loops are more than just asyncio, but without examples to point to it's hard to keep a focus on that fact. And even harder to isolate what is an event loop mechanism from what is asyncio specific. For example, asyncio.BaseEventLoop has a create_connection method. That's *obviously* not a fundamental aspect of a generic event loop, But call_soon (presumably) is. Having a documented basic event loop interface would probably help emphasise the idea than event loops don't have to be asyncio. (Actually, what *is* the minimal event loop interface that is needed for the various task/future mechanisms to work, independently of asyncio? And what features of an event loop etc are needed for the PEP, if it's being used outside of asyncio?) I guess the other canonical event loop use case is GUI system message dispatchers. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Perhaps, but arguing about the nitty-gritty details of something doesn't automatically lead to a clearer understanding of the higher level concept. Discussing how turning a steering wheel in a car might help you grasp how cars turn, but it isn't a requirement to get turn the wheel left to make the car go left. Fair point. If only I could avoid driving into walls :-) Of course, that's based on my perception of one of the goals of the PEP as being make coroutines and asyncio more accessible, If the actual goals are different, my conclusion is invalid. I think the goal is make coroutines easier to use and does not directly relate to asyncio. OK. But in that case, some examples using a non-asyncio toy just schedule tasks event loop might help. Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Yep. ... and so you can't use it with async/await? Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? I believe so; you need something to implement __await__. This is true in any language that implements co-routines. Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes something that the average user can't use. Which in turn further entrenches it as a specialist-only tool. You forgot to append ... yet to that statement. Just because something isn't available out of the box without some effort to support doesn't mean it will never happen, else there would be absolutely no Python 3 users out there. Fair point. Yuri mentioned aiohttp, as well. The one difference between this and Python 2/3, is that here you *have* to have two separate implementations. There's no equivalent of a shared source async and synchronous
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 5 May 2015 at 21:38, Guido van Rossum gu...@python.org wrote: Jumping in to correct one fact. On Tue, May 5, 2015 at 12:44 PM, Brett Cannon br...@python.org wrote: On Tue, May 5, 2015 at 3:14 PM Paul Moore p.f.mo...@gmail.com wrote: Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Yep. The run_in_executor call is not callback-based -- the confusion probably stems from the name of the function argument ('callback'). It actually returns a Future representing the result (or error) of an operation, where the operation is represented by the function argument. So if you have e.g. a function def factorial(n): return 1 if n = 0 else n*factorial(n-1) you can run it in an executor from your async(io) code like this: loop = asyncio.get_event_loop() result = yield from loop.run_in_executor(factorial, 100) (In a PEP 492 coroutine substitute await for yield from.) Thanks, that's an important correction. Given that, run_in_executor is the link to blocking calls that I was searching for. And yes, the callback terminology does make this far from obvious, unfortunately. As does the point at which it's introduced (before futures have been described) and the fact that it says this method is a coroutine rather than this method returns a Future[1]. Paul [1] I'm still struggling to understand the terminology, so if those two statements are equivalent, that's not yet obvious to me. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Jumping in to correct one fact. On Tue, May 5, 2015 at 12:44 PM, Brett Cannon br...@python.org wrote: On Tue, May 5, 2015 at 3:14 PM Paul Moore p.f.mo...@gmail.com wrote: Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Yep. The run_in_executor call is not callback-based -- the confusion probably stems from the name of the function argument ('callback'). It actually returns a Future representing the result (or error) of an operation, where the operation is represented by the function argument. So if you have e.g. a function def factorial(n): return 1 if n = 0 else n*factorial(n-1) you can run it in an executor from your async(io) code like this: loop = asyncio.get_event_loop() result = yield from loop.run_in_executor(factorial, 100) (In a PEP 492 coroutine substitute await for yield from.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Tue, May 5, 2015 at 1:39 PM, Paul Moore p.f.mo...@gmail.com wrote: It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about. What about Greg Ewing's example? http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-05-05 4:39 PM, Paul Moore wrote: Is there anyone who feels they could write a stripped down but working example of a valid Python event loop*without* the asyncio aspects? Or is that what David Beazley's talk does? Yes, in David's talk, where he starts to use 'yield from' you can simply use new coroutines. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Hi Wolfgang, On 2015-05-05 7:27 AM, Wolfgang wrote: Hi, [..] Even the discussion on python-dev suggests there is some time needed to finalize all this. I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not. We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. gevent has a different answer, but greenlents/stackless is something that will never be merged in CPython and other implementations. And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. networking stuff, and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Tue, May 5, 2015 at 1:44 PM, Paul Moore p.f.mo...@gmail.com wrote: [Guido] The run_in_executor call is not callback-based -- the confusion probably stems from the name of the function argument ('callback'). It actually returns a Future representing the result (or error) of an operation, where the operation is represented by the function argument. So if you have e.g. a function def factorial(n): return 1 if n = 0 else n*factorial(n-1) you can run it in an executor from your async(io) code like this: loop = asyncio.get_event_loop() result = yield from loop.run_in_executor(factorial, 100) (In a PEP 492 coroutine substitute await for yield from.) Thanks, that's an important correction. Given that, run_in_executor is the link to blocking calls that I was searching for. And yes, the callback terminology does make this far from obvious, unfortunately. As does the point at which it's introduced (before futures have been described) and the fact that it says this method is a coroutine rather than this method returns a Future[1]. Paul [1] I'm still struggling to understand the terminology, so if those two statements are equivalent, that's not yet obvious to me. I apologize for the confusing documentation. We need more help from qualified tech writers! Writing PEP 3156 was a huge undertaking for me; after that I was exhausted and did not want to take on writing the end user documentation as well, so it was left unfinished. :-( In PEP 3156 (asyncio package) there are really three separate concepts: - Future, which is a specific class (of which Task is a subclass); - coroutine, by which in this context is meant a generator object obtained by calling a generator function decorated with @asyncio.coroutine and written to conform to the asyncio protocol for coroutines (i.e. don't use bare yield, only use yield from, and the latter always with either a Future or a coroutine as argument); - either of the above, which is actually the most common requirement -- most asyncio functions that support one also support the other, and either is allowable as the argument to `yield from`. In the implementation we so often flipped between Future and coroutine that I imagine sometimes the implementation and docs differ; also, we don't have a good short name for either of the above so we end up using one or the other as a shorthand. *Unless* you want to attach callbacks, inspect the result or exception, or cancel it (all of which require a Future), your code shouldn't be concerned about the difference -- you should just use `res = yield from func(args)` and use try/except to catch exceptions if you care. And if you do need a Future, you can call the function asyncio.async() on it (which in PEP 492 is renamed to ensure_future()). In the PEP 492 world, these concepts map as follows: - Future translates to something with an __await__ method (and asyncio Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__); - asyncio coroutine maps to PEP 492 coroutine object (either defined with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter); - either of the above maps to awaitable. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-05-05 5:31 PM, Jim J. Jewett wrote: Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote: As for terminology, I view this discussion differently. It's not about the technical details (Python has asymmetric coroutines, that's it), but rather on how to disambiguate coroutines implemented with generators and yield-from, from new 'async def' coroutines. Not just How?, but Why?. Why do they*need* to be disambiguated? To clearly show how one interacts with the other, to explain how backwards compatibility is implemented, and to better illustrate some additional (and necessary) restrictions we put on 'async def' coroutines. Otherwise, the PEP would be completely unreadable :) Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Tue, May 5, 2015 at 2:29 PM, Paul Moore p.f.mo...@gmail.com wrote: On 5 May 2015 at 22:12, Guido van Rossum gu...@python.org wrote: I apologize for the confusing documentation. We need more help from qualified tech writers! Writing PEP 3156 was a huge undertaking for me; after that I was exhausted and did not want to take on writing the end user documentation as well, so it was left unfinished. :-( Fair enough. When I properly document one of my projects, *then* I'll think about complaining :-) These things happen. In PEP 3156 (asyncio package) there are really three separate concepts: - Future, which is a specific class (of which Task is a subclass); - coroutine, by which in this context is meant a generator object obtained by calling a generator function decorated with @asyncio.coroutine and written to conform to the asyncio protocol for coroutines (i.e. don't use bare yield, only use yield from, and the latter always with either a Future or a coroutine as argument); - either of the above, which is actually the most common requirement -- most asyncio functions that support one also support the other, and either is allowable as the argument to `yield from`. In the implementation we so often flipped between Future and coroutine that I imagine sometimes the implementation and docs differ; also, we don't have a good short name for either of the above so we end up using one or the other as a shorthand. OK, that makes a lot of sense. *Unless* you want to attach callbacks, inspect the result or exception, or cancel it (all of which require a Future), your code shouldn't be concerned about the difference -- you should just use `res = yield from func(args)` and use try/except to catch exceptions if you care. And if you do need a Future, you can call the function asyncio.async() on it (which in PEP 492 is renamed to ensure_future()). Again, makes sense. Although there are some bits of example code in the docs that call asyncio.async() on a coroutine and throw away the result (for example, https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-until-complete ). That confuses me. Are you saying that async() modifies its (coroutine) argument to make it a Future? Rather than wrapping a coroutine in a Future, which gets returned? No, it wraps a coroutine (i.e. a generator) in a Task, but leaves a Future alone. I'm stumped why that example calls async() and then throws the result away. I suspect it won't work without it (or else Victor wouldn't have added the call) but the reason seems, um, deep. I think wrapping it in a Task enters the generator in the event loop's queue of runnables -- otherwise the generator may well be garbage-collected without ever running. Such complexity doesn't belong in such a simple example though. In the PEP 492 world, these concepts map as follows: - Future translates to something with an __await__ method (and asyncio Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__); - asyncio coroutine maps to PEP 492 coroutine object (either defined with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter); - either of the above maps to awaitable. OK. Although future is a nicer term than something with an __await__ method and the plethora of flavours of coroutine is not great. But given that the only term we'll need in common cases is awaitable, it's still a net improvement. So in the PEP 492 world, there's no such thing as a Task outside of asyncio? Or, to put it another way, a Task is only relevant in an IO context (unless an alternative event loop library implemented a similar concept), and we should only be talking in terms of awaitables and futures (given concurrent.futures and asyncio, I doubt you're going to be able to stop people using Future for the generic term for something with an __await__ method at best, and quite possibly as equivalent to awaitable, unfortunately). I'm not sure. But it's true that Futures and Tasks in asyncio serve the purpose of linking the event loop (whose basic functioning is callback-based) to coroutines (implemented by generators). The basic idea is that when some I/O completes the event loop will call a callback function registered for that particular I/O operation; the callback then is actually a bound method of a Future or Task that causes the latter to be marked as complete (i.e. having a result) which in turn will call other callbacks (registered with the Future using add_done_callback()); in the case of a Task (i.e. a special kind of Future that wraps a generator/coroutine) this will resume the coroutine. (Actually it may resume an entire stack of coroutines that are blocked waiting for each other at yield-from; in my spare time I'm working on an explanation of the machinery underlying yield, yield
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 5 May 2015 at 21:57, Guido van Rossum gu...@python.org wrote: On Tue, May 5, 2015 at 1:39 PM, Paul Moore p.f.mo...@gmail.com wrote: It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about. What about Greg Ewing's example? http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt That doesn't cover any of the higher level abstractions like tasks or futures (at least not by those names or with those interfaces). And I don't see where the PEP 492 additions would fit in (OK, replace yield from with await is part of it, but I don't see the rest). We may be talking at cross purposes here. There's a lot of asyncio that doesn't seem to me to be IO-related. Specifically the future and task abstractions. I view those as relevant to coroutine programming in Python because they are referenced in any discussion of coroutines (you yield from a future, for example). If you see them as purely asyncio related (and not directly usable from outside of an asyncio context) then that may explain some of my confusion (but at the cost of reducing the coroutine concept to something pretty trivial in the absence of a library that independently implements these concepts). In some ways I wish there had been an asyncio library that covered the areas that are fundamentally about IO multiplexing. And a separate library (just async, maybe, although that's now a bad idea as it clashes with a keyword :-)) that covered generic event loop, task and synchronisation areas. But that's water under the bridge now. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote: As for terminology, I view this discussion differently. It's not about the technical details (Python has asymmetric coroutines, that's it), but rather on how to disambiguate coroutines implemented with generators and yield-from, from new 'async def' coroutines. Not just How?, but Why?. Why do they *need* to be disambiguated? With the benefit of having recently read all that discussion (as opposed to just the PEP), my answer is ... uh ... that generators vs async def is NOT an important distinction. What matters (as best I can tell) is: something using yield (or yield from) to mark execution context switches vs other kinds of callables, including those using yield to make an iterator I'm not quite sure that the actual proposal even really separates them effectively, in part because the terminology keeps suggesting other distinctions instead. (The glossary does help; just not enough.) -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 492: async/await in Python; version 4
Hi, still watching progress here. Read all posts and changes. Everything improved and I know it is a lot of work. Thx for doing this. But I still think this PEP goes to far. 1. To have native coroutines and await, __await__ is very good and useful. Also for beginners to see and mark coroutines are a different concept than generators. Also this is easy to learn. Easier as to explain why a generator is in this case a coroutine and so on. 2. I still don't like to sprinkle async everywhere. At all we don't need it for the first step. We can handle coroutines similar to generators, when there is a await in it then it is one. Same as for yield. Or to be more explicit, if it is marked as one with @coroutine it is enough. But then it makes also sense to do the same for generators with @generator. We should be consistent here. 3. async with is not needed, there are rare use cases for it and every can be written with a try: except: finally: Every async framework lived without it over years. No problem because for the seldom need try: ... was enough. 4. async for can be implemented with a while loop. For me this is even more explicit and clear. Every time I see the async for I try to find out what is done in async manner. For what I do an implicit await ? Also in most of my uses cases it was enough to produce Futures in a normal loop and yield (await) for them. Even for database interfaces. Most of the time they do prefetching under the hood and I don't have to care at all. 5. For async with/for a lot of new special methods are needed all only prefixed with a. Easy to confuse with a for abstract. Often used to prefix abstract classes. Still think __async_iter__, __async_enter__ is better for this. Yes more to write but not so easy to confuse with the sync __iter__ or __enter__, ... And matches more to I must use async to call them. I am not new to the async land, have done a lot of stuff with twisted and even looked at tornado. Also has tried to teach some people the asnc world. This is not easy and you learn most only by doing and using it. For my final conclusion, we should not rush all this into the language. Do it step by step and also help other Python implementations to support it. For now only a very low percentage are at Python 3.4. And if you compare the statistics in PyPi, you see most are still at Python 2.7 and using async frameworks like twisted/tornado. We do such a fundamental language change only because a few people need it for a niche in the whole Python universe? We confuse the rest with new stuff they never need? Even the discussion on python-dev suggests there is some time needed to finalize all this. We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. Don't get me wrong, I like async programming and I think it is very useful. But had to learn not everyone thinks so and most only want to solve there problems in an easy way and not get a new one called async. Now I shut up. Go to my normal mode and be quiet and read. ;-) Regards, Wolfgang ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Fri, May 01, 2015 at 09:24:47PM +0100, Arnaud Delobelle wrote: I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion. In what way? I'm thinking that the only confusion would be if you wrote async for instead of for, or vice versa, and instead of getting an exception you got the (a)syncronous behaviour you didn't want. But I have no intuition for how likely it is that you could write an asyncronous for loop, leave out the async, and still have the code do something meaningful. Other than that, I think it would be fine to have an object be both a syncronous and asyncronous iterator. You specify the behaviour you want by how you use it. We can already do that, e.g. unittest's assertRaises is both a test assertion and a context manager. Objects can have multiple roles, and it's not usually abused, or confusing. I'm not sure that async iterables will be any different. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 1 May 2015 at 21:27, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-01 4:24 PM, Arnaud Delobelle wrote: On 1 May 2015 at 20:24, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-01 3:19 PM, Ethan Furman wrote: [...] If we must have __aiter__, then we may as well also have __anext__; besides being more consistent, it also allows an object to be both a normol iterator and an asynch iterator. And this is a good point too. I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion. I doubt that it will be a popular thing. But disallowing this by merging two different protocols in one isn't a good idea either. I having been arguing for merging two different protocols. I'm saying that allowing an object to be both normal and async iterable is not an argument for having separate protocols because it's not a good thing. Cheers, -- Arnaud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 1 May 2015 at 20:24, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-01 3:19 PM, Ethan Furman wrote: [...] If we must have __aiter__, then we may as well also have __anext__; besides being more consistent, it also allows an object to be both a normol iterator and an asynch iterator. And this is a good point too. I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion. -- Arnaud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 5/1/2015 9:59 AM, Guido van Rossum wrote: I think coroutine is the name of a concept, not a specific implementation. Cheers, Cheers indeed! I agree that the *concept* is best called coroutine -- and we have used this term ever since PEP 342. But when we're talking specifics and trying to distinguish e.g. a function declared with 'async def' from a regular function or from a regular generator function, using 'async function' sounds right. And 'async method' if it's a method. Exactly. The async function/method is an implementation technique for a specific kind/subset of coroutine functionality. So the term coroutine, qualified by a description of its best usage and limitationsof async function, can be used in defining async function, thus appealing to what people know or have heard of and vaguely understand and can read more about in the literature. A glossary entry for coroutine in the docs seems appropriate, which could point out the 16† ways to implement coroutine-style functionalities in Python, and perhaps make recommendations for different types of usage. †OK, not 16 ways, but it is 3 now, or 4? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Yury Selivanov schrieb am 01.05.2015 um 20:52: I don't like the idea of combining __next__ and __anext__. Ok, fair enough. So, how would you use this new protocol manually then? Say, I already know that I won't need to await the next item that the iterator will return. For normal iterators, I could just call next() on it and continue the for-loop. How would I do it for AIterators? Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Fri, May 1, 2015 at 5:39 AM, Stefan Behnel stefan...@behnel.de wrote: Yury Selivanov schrieb am 30.04.2015 um 03:30: Asynchronous Iterators and async for -- An *asynchronous iterable* is able to call asynchronous code in its *iter* implementation, and *asynchronous iterator* can call asynchronous code in its *next* method. To support asynchronous iteration: 1. An object must implement an ``__aiter__`` method returning an *awaitable* resulting in an *asynchronous iterator object*. 2. An *asynchronous iterator object* must implement an ``__anext__`` method returning an *awaitable*. 3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration`` exception. What this section does not explain, AFAICT, nor the section on design considerations, is why the iterator protocol needs to be duplicated entirely. Can't we just assume (or even validate) that any 'regular' iterator returned from __aiter__() (as opposed to __iter__()) properly obeys to the new protocol? Why additionally duplicate __next__() and StopIteration? ISTM that all this really depends on is that __next__() returns an awaitable. Renaming the method doesn't help with that guarantee. This is an astute observation. I think its flaw (if any) is the situation where we want a single object to be both a regular iterator and an async iterator (say when migrating code to the new world). The __next__ method might want to return a result while __anext__ has to return an awaitable. The solution to that would be to have __aiter__ return an instance of a different class than __iter__, but that's not always convenient. Thus, aware of the choice, I would still prefer a separate __anext__ method. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 1 May 2015 at 16:31, Guido van Rossum gu...@python.org wrote: On Fri, May 1, 2015 at 5:50 AM, Stefan Behnel stefan...@behnel.de wrote: Yury Selivanov schrieb am 30.04.2015 um 03:30: 1. Terminology: - *native coroutine* term is used for async def functions. When I read native, I think of native (binary) code. So native coroutine sounds like it's implemented in some compiled low-level language. That might be the case (certainly in the CPython world), but it's not related to this PEP nor its set of examples. We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted. Well, it doesn't hurt to avoid obvious misleading terminology upfront. I think obvious[ly] misleading is too strong, nobody is trying to mislead anybody, we just have different associations with the same word. Given the feedback I'd call native coroutine suboptimal (even though I proposed it myself) and I am now in favor of using async function. But what if you have async methods? I know, a method is almost a function, but still, sounds slightly confusing. IMHO, these are really classical coroutines. If gevent calls them coroutines, I don't think asyncio has any less right to call them coroutines. You have to ask yourself this: a new programmer, when he sees mentions of coroutines, how likely is he to understand what he is dealing with? What about async function? The former is a well known concept, since decades ago, while the latter is something he probably (at least me) never heard of before. For me, an async function is just as likely to be an API that is asynchronous in the sense that it takes an extra callback parameter to be called when the asynchronous work is done. I think coroutine is the name of a concept, not a specific implementation. Cheers, -- Gustavo J. A. M. Carneiro Gambit Research The universe is always one step beyond logic. -- Frank Herbert ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Fri, May 1, 2015 at 5:50 AM, Stefan Behnel stefan...@behnel.de wrote: Yury Selivanov schrieb am 30.04.2015 um 03:30: 1. Terminology: - *native coroutine* term is used for async def functions. When I read native, I think of native (binary) code. So native coroutine sounds like it's implemented in some compiled low-level language. That might be the case (certainly in the CPython world), but it's not related to this PEP nor its set of examples. We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted. Well, it doesn't hurt to avoid obvious misleading terminology upfront. I think obvious[ly] misleading is too strong, nobody is trying to mislead anybody, we just have different associations with the same word. Given the feedback I'd call native coroutine suboptimal (even though I proposed it myself) and I am now in favor of using async function. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Guido van Rossum schrieb am 01.05.2015 um 17:28: On Fri, May 1, 2015 at 5:39 AM, Stefan Behnel wrote: Yury Selivanov schrieb am 30.04.2015 um 03:30: Asynchronous Iterators and async for -- An *asynchronous iterable* is able to call asynchronous code in its *iter* implementation, and *asynchronous iterator* can call asynchronous code in its *next* method. To support asynchronous iteration: 1. An object must implement an ``__aiter__`` method returning an *awaitable* resulting in an *asynchronous iterator object*. 2. An *asynchronous iterator object* must implement an ``__anext__`` method returning an *awaitable*. 3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration`` exception. What this section does not explain, AFAICT, nor the section on design considerations, is why the iterator protocol needs to be duplicated entirely. Can't we just assume (or even validate) that any 'regular' iterator returned from __aiter__() (as opposed to __iter__()) properly obeys to the new protocol? Why additionally duplicate __next__() and StopIteration? ISTM that all this really depends on is that __next__() returns an awaitable. Renaming the method doesn't help with that guarantee. This is an astute observation. I think its flaw (if any) is the situation where we want a single object to be both a regular iterator and an async iterator (say when migrating code to the new world). The __next__ method might want to return a result while __anext__ has to return an awaitable. The solution to that would be to have __aiter__ return an instance of a different class than __iter__, but that's not always convenient. My personal gut feeling is that this case would best be handled by a generic wrapper class. Both are well defined protocols, so I don't expect people to change all of their classes and instead return a wrapped object either from __iter__() or __aiter__(), depending on which they want to optimise for, or which will eventually turn out to be easier to wrap. But that's trying to predict the [Ff]uture, obviously. It just feels like unnecessary complexity for now. And we already have a type slot for __next__ (tp_iternext), but not for __anext__. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Fri, May 1, 2015 at 8:55 AM, Gustavo Carneiro gjcarne...@gmail.com wrote: On 1 May 2015 at 16:31, Guido van Rossum gu...@python.org wrote: On Fri, May 1, 2015 at 5:50 AM, Stefan Behnel stefan...@behnel.de wrote: Yury Selivanov schrieb am 30.04.2015 um 03:30: 1. Terminology: - *native coroutine* term is used for async def functions. When I read native, I think of native (binary) code. So native coroutine sounds like it's implemented in some compiled low-level language. That might be the case (certainly in the CPython world), but it's not related to this PEP nor its set of examples. We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted. Well, it doesn't hurt to avoid obvious misleading terminology upfront. I think obvious[ly] misleading is too strong, nobody is trying to mislead anybody, we just have different associations with the same word. Given the feedback I'd call native coroutine suboptimal (even though I proposed it myself) and I am now in favor of using async function. But what if you have async methods? I know, a method is almost a function, but still, sounds slightly confusing. IMHO, these are really classical coroutines. If gevent calls them coroutines, I don't think asyncio has any less right to call them coroutines. You have to ask yourself this: a new programmer, when he sees mentions of coroutines, how likely is he to understand what he is dealing with? What about async function? The former is a well known concept, since decades ago, while the latter is something he probably (at least me) never heard of before. For me, an async function is just as likely to be an API that is asynchronous in the sense that it takes an extra callback parameter to be called when the asynchronous work is done. I think coroutine is the name of a concept, not a specific implementation. Cheers, Cheers indeed! I agree that the *concept* is best called coroutine -- and we have used this term ever since PEP 342. But when we're talking specifics and trying to distinguish e.g. a function declared with 'async def' from a regular function or from a regular generator function, using 'async function' sounds right. And 'async method' if it's a method. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-05-01 4:24 PM, Arnaud Delobelle wrote: On 1 May 2015 at 20:24, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-01 3:19 PM, Ethan Furman wrote: [...] If we must have __aiter__, then we may as well also have __anext__; besides being more consistent, it also allows an object to be both a normol iterator and an asynch iterator. And this is a good point too. I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion. I doubt that it will be a popular thing. But disallowing this by merging two different protocols in one isn't a good idea either. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Let's say it this way: I want to know what I am looking at when I browse through the code -- an asynchronous iterator, or a normal iterator. I want an explicit difference between these protocols, because they are different. Moreover, the below code is a perfectly valid, infinite iterable: class SomeIterable: def __iter__(self): return self async def __next__(self): return 'spam' I'm strong -1 on this idea. Yury On 2015-05-01 3:03 PM, Stefan Behnel wrote: Yury Selivanov schrieb am 01.05.2015 um 20:52: I don't like the idea of combining __next__ and __anext__. In this case explicit is better than implicit. __next__ returning coroutines is a perfectly normal thing for a normal 'for' loop (it wouldn't to anything with them), whereas 'async for' will interpret that differently, and will try to await those coroutines. Sure, but the difference is that one would have called __aiter__() first and the other __iter__(). Normally, either of the two would not exist, so using the wrong loop on an object will just fail. However, after we passed that barrier, we already know that the object that was returned is supposed to obey to the expected protocol, so it doesn't matter whether we call __next__() or name it __anext__(), except that the second requires us to duplicate an existing protocol. This has nothing to do with implicit vs. explicit. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-05-01 3:19 PM, Ethan Furman wrote: Sure, but the difference is that one would have called __aiter__() first and the other __iter__(). Normally, either of the two would not exist, so using the wrong loop on an object will just fail. However, after we passed that barrier, we already know that the object that was returned is supposed to obey to the expected protocol, so it doesn't matter whether we call __next__() or name it __anext__(), except that the second requires us to duplicate an existing protocol. If we must have __aiter__, then we may as well also have __anext__; besides being more consistent, it also allows an object to be both a normol iterator and an asynch iterator. And this is a good point too. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Yury Selivanov schrieb am 01.05.2015 um 20:52: I don't like the idea of combining __next__ and __anext__. In this case explicit is better than implicit. __next__ returning coroutines is a perfectly normal thing for a normal 'for' loop (it wouldn't to anything with them), whereas 'async for' will interpret that differently, and will try to await those coroutines. Sure, but the difference is that one would have called __aiter__() first and the other __iter__(). Normally, either of the two would not exist, so using the wrong loop on an object will just fail. However, after we passed that barrier, we already know that the object that was returned is supposed to obey to the expected protocol, so it doesn't matter whether we call __next__() or name it __anext__(), except that the second requires us to duplicate an existing protocol. This has nothing to do with implicit vs. explicit. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Stefan, I don't like the idea of combining __next__ and __anext__. In this case explicit is better than implicit. __next__ returning coroutines is a perfectly normal thing for a normal 'for' loop (it wouldn't to anything with them), whereas 'async for' will interpret that differently, and will try to await those coroutines. Yury On 2015-05-01 1:10 PM, Stefan Behnel wrote: Guido van Rossum schrieb am 01.05.2015 um 17:28: On Fri, May 1, 2015 at 5:39 AM, Stefan Behnel wrote: Yury Selivanov schrieb am 30.04.2015 um 03:30: Asynchronous Iterators and async for -- An *asynchronous iterable* is able to call asynchronous code in its *iter* implementation, and *asynchronous iterator* can call asynchronous code in its *next* method. To support asynchronous iteration: 1. An object must implement an ``__aiter__`` method returning an *awaitable* resulting in an *asynchronous iterator object*. 2. An *asynchronous iterator object* must implement an ``__anext__`` method returning an *awaitable*. 3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration`` exception. What this section does not explain, AFAICT, nor the section on design considerations, is why the iterator protocol needs to be duplicated entirely. Can't we just assume (or even validate) that any 'regular' iterator returned from __aiter__() (as opposed to __iter__()) properly obeys to the new protocol? Why additionally duplicate __next__() and StopIteration? ISTM that all this really depends on is that __next__() returns an awaitable. Renaming the method doesn't help with that guarantee. This is an astute observation. I think its flaw (if any) is the situation where we want a single object to be both a regular iterator and an async iterator (say when migrating code to the new world). The __next__ method might want to return a result while __anext__ has to return an awaitable. The solution to that would be to have __aiter__ return an instance of a different class than __iter__, but that's not always convenient. My personal gut feeling is that this case would best be handled by a generic wrapper class. Both are well defined protocols, so I don't expect people to change all of their classes and instead return a wrapped object either from __iter__() or __aiter__(), depending on which they want to optimise for, or which will eventually turn out to be easier to wrap. But that's trying to predict the [Ff]uture, obviously. It just feels like unnecessary complexity for now. And we already have a type slot for __next__ (tp_iternext), but not for __anext__. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 05/01, Stefan Behnel wrote: Yury Selivanov schrieb am 01.05.2015 um 20:52: I don't like the idea of combining __next__ and __anext__. In this case explicit is better than implicit. __next__ returning coroutines is a perfectly normal thing for a normal 'for' loop (it wouldn't to anything with them), whereas 'async for' will interpret that differently, and will try to await those coroutines. Sure, but the difference is that one would have called __aiter__() first and the other __iter__(). Normally, either of the two would not exist, so using the wrong loop on an object will just fail. However, after we passed that barrier, we already know that the object that was returned is supposed to obey to the expected protocol, so it doesn't matter whether we call __next__() or name it __anext__(), except that the second requires us to duplicate an existing protocol. If we must have __aiter__, then we may as well also have __anext__; besides being more consistent, it also allows an object to be both a normol iterator and an asynch iterator. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Yury Selivanov schrieb am 30.04.2015 um 03:30: Asynchronous Iterators and async for -- An *asynchronous iterable* is able to call asynchronous code in its *iter* implementation, and *asynchronous iterator* can call asynchronous code in its *next* method. To support asynchronous iteration: 1. An object must implement an ``__aiter__`` method returning an *awaitable* resulting in an *asynchronous iterator object*. 2. An *asynchronous iterator object* must implement an ``__anext__`` method returning an *awaitable*. 3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration`` exception. What this section does not explain, AFAICT, nor the section on design considerations, is why the iterator protocol needs to be duplicated entirely. Can't we just assume (or even validate) that any 'regular' iterator returned from __aiter__() (as opposed to __iter__()) properly obeys to the new protocol? Why additionally duplicate __next__() and StopIteration? ISTM that all this really depends on is that __next__() returns an awaitable. Renaming the method doesn't help with that guarantee. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Yury Selivanov schrieb am 30.04.2015 um 03:30: 1. Terminology: - *native coroutine* term is used for async def functions. When I read native, I think of native (binary) code. So native coroutine sounds like it's implemented in some compiled low-level language. That might be the case (certainly in the CPython world), but it's not related to this PEP nor its set of examples. We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted. Well, it doesn't hurt to avoid obvious misleading terminology upfront. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Yury Selivanov wrote: 3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility. I don't think you can honestly claim full backwards compatibility as long as there are some combinations of old-style and new-style code that won't work together. You seem to be using your own personal definition of full here. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Thu, Apr 30, 2015 at 4:24 PM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Yury Selivanov wrote: Well, using next() and iter() on coroutines in asyncio code is something esoteric. I can't even imagine why you would want to do that. I'm talking about the fact that existing generator- based coroutines that aren't decorated with @coroutine won't be able to call new ones that use async def. This means that converting one body of code to the new style can force changes in other code that interacts with it. Maybe this is not considered a problem, but as long as it's true, I don't think it's accurate to claim full backwards compatibility. Greg, you seem to have an odd notion of full backwards compatibility. The term means that old code won't break. It doesn't imply that old and new code can always seamlessly interact (that would be an impossibly high bar for almost any change). That said, interoperability between old code and new code is an area of interest. But if the only thing that's standing between old code and new code is the @coroutine decorator, things are looking pretty good -- that decorator is already strongly required for coroutines intended for use with the asyncio package, and older versions of the asyncio package also define that decorator, so if there's old code out there that needs to be able to call the new coroutines (by whatever name, e.g. async functions :-), adding the @coroutine decorator to the old code doesn't look like too much of a burden. I assume there might be code out there that uses yield-from-based coroutines but does not use the asyncio package, but I doubt there is much code like that (I haven't seen much mention of yield-from outside its use in asyncio). So I think the interop problem is mostly limited to asyncio-using code that plays loose with the @coroutine decorator requirement and now wants to work with the new async functions. That's easy enough to address. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-04-30 7:24 PM, Greg Ewing wrote: Yury Selivanov wrote: Well, using next() and iter() on coroutines in asyncio code is something esoteric. I can't even imagine why you would want to do that. I'm talking about the fact that existing generator- based coroutines that aren't decorated with @coroutine won't be able to call new ones that use async def. Ah, alright. You quoted this: 3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility. I wrote full backwards compatibility for that particular point #3 -- existing @asyncio.coroutines will have __iter__ and __next__ working just fine. Sorry if this was misleading. This means that converting one body of code to the new style can force changes in other code that interacts with it. Maybe this is not considered a problem, but as long as it's true, I don't think it's accurate to claim full backwards compatibility. I covered this in point #4. I also touched this in https://www.python.org/dev/peps/pep-0492/#migration-strategy I'm still waiting for feedback on this from Guido. If he decides to go with RuntimeWarnings, then it's 100% backwards compatible. If we keep TypeErrors -- then *existing code will work on 3.5*, but something *might* break during adopting new syntax. I'll update the Backwards Compatibility section. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Yury Selivanov wrote: Well, using next() and iter() on coroutines in asyncio code is something esoteric. I can't even imagine why you would want to do that. I'm talking about the fact that existing generator- based coroutines that aren't decorated with @coroutine won't be able to call new ones that use async def. This means that converting one body of code to the new style can force changes in other code that interacts with it. Maybe this is not considered a problem, but as long as it's true, I don't think it's accurate to claim full backwards compatibility. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-04-30 5:16 AM, Greg Ewing wrote: Yury Selivanov wrote: 3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility. I don't think you can honestly claim full backwards compatibility as long as there are some combinations of old-style and new-style code that won't work together. You seem to be using your own personal definition of full here. Well, using next() and iter() on coroutines in asyncio code is something esoteric. I can't even imagine why you would want to do that. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 492: async/await in Python; version 4
Hi python-dev, New version of the PEP is attached. Summary of updates: 1. Terminology: - *native coroutine* term is used for async def functions. - *generator-based coroutine* term is used for PEP 380 coroutines used in asyncio. - *coroutine* is used when *native coroutine* or *generator based coroutine* can be used in the same context. I think that it's not really productive to discuss the terminology that we use in the PEP. Its only purpose is to disambiguate concepts used in the PEP. We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted. Although if you notice that somewhere in the PEP it is not crystal clear what coroutine means please give me a heads up! 2. Syntax of await expressions is now thoroghly defined in the PEP. See Await Expression, Updated operator precedence table, and Examples of await expressions sections. I like the current approach. I'm still not convinced that we should make 'await' the same grammatically as unary minus. I don't understand why we should allow parsing things like 'await -fut'; this should be a SyntaxError. I'm fine to modify the grammar to allow 'await await fut' syntax, though. And I'm open to discussion :) 3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility. 4. asyncio / Migration strategy. Existing code can be used with PEP 492 as is, everything will work as expected. However, since *plain generator*s (not decorated with @asyncio.coroutine) cannot 'yield from' native coroutines (async def functions), it might break some code *while adapting it to the new syntax*. I'm open to just throw a RuntimeWarning in this case in 3.5, and raise a TypeError in 3.6. Please see the Differences from generators section of the PEP. 5. inspect.isawaitable() function. And, all new functions are now listed in the New Standard Library Functions section. 6. Lot's of small updates and tweaks throughout the PEP. Thanks, Yury PEP: 492 Title: Coroutines with async and await syntax Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov yseliva...@sprymix.com Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Apr-2015 Python-Version: 3.5 Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015 Abstract This PEP introduces new syntax for coroutines, asynchronous ``with`` statements and ``for`` loops. The main motivation behind this proposal is to streamline writing and maintaining asynchronous code, as well as to simplify previously hard to implement code patterns. Rationale and Goals === Current Python supports implementing coroutines via generators (PEP 342), further enhanced by the ``yield from`` syntax introduced in PEP 380. This approach has a number of shortcomings: * it is easy to confuse coroutines with regular generators, since they share the same syntax; async libraries often attempt to alleviate this by using decorators (e.g. ``@asyncio.coroutine`` [1]_); * it is not possible to natively define a coroutine which has no ``yield`` or ``yield from`` statements, again requiring the use of decorators to fix potential refactoring issues; * support for asynchronous calls is limited to expressions where ``yield`` is allowed syntactically, limiting the usefulness of syntactic features, such as ``with`` and ``for`` statements. This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a specific library. This also enables linters and IDEs to improve static code analysis and refactoring. Native coroutines and the associated new syntax features make it possible to define context manager and iteration protocols in asynchronous terms. As shown later in this proposal, the new ``async with`` statement lets Python programs perform asynchronous calls when entering and exiting a runtime context, and the new ``async for`` statement makes it possible to perform asynchronous calls in iterators. Specification = This proposal introduces new syntax and semantics to enhance coroutine support in Python. This specification presumes knowledge of the implementation of coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax changes proposed here comes from the asyncio framework (PEP 3156) and the Cofunctions proposal (PEP 3152, now rejected in favor of this specification). From this point in this document we use the word *native coroutine* to refer to functions declared using the new syntax. *generator-based coroutine* is used where necessary to refer to coroutines that are based on generator syntax. *coroutine* is used in contexts where both definitions are applicable. New Coroutine Declaration Syntax The
Re: [Python-Dev] PEP 492: async/await in Python; version 4
One more thing to discuss: 7. StopAsyncIteration vs AsyncStopIteration. I don't have a strong opinion on this, I prefer the former because it reads better. There was no consensus on which one we should use. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com