Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-07 Thread Arnaud Delobelle
On 3 May 2015 at 16:24, Steven D'Aprano st...@pearwood.info wrote:
 On Fri, May 01, 2015 at 09:24:47PM +0100, Arnaud Delobelle wrote:

 I'm not convinced that allowing an object to be both a normal and an
 async iterator is a good thing.  It could be a recipe for confusion.

 In what way?

 I'm thinking that the only confusion would be if you wrote async for
 instead of for, or vice versa, and instead of getting an exception you
 got the (a)syncronous behaviour you didn't want.

Yes.  This is the same kind of confusion that this PEP is trying hard
to get rid of in other parts (i.e. the confusion between 'yield' and
'yield from' in current coroutines).

 But I have no intuition for how likely it is that you could write an
 asyncronous for loop, leave out the async, and still have the code do
 something meaningful.

Well if you've made you object both iterable and 'async iterable' then
it's very likely that you're going to get something meaningful out of
either kind of iteration.  Just not the way you want it if you
mistakenly left out (or added) the 'async' keyword in your loop.

 Other than that, I think it would be fine to have an object be both a
 syncronous and asyncronous iterator. You specify the behaviour you want
 by how you use it. We can already do that, e.g. unittest's assertRaises
 is both a test assertion and a context manager.

The latter is fine, because there is no danger of mistaking one for
the other, unlike iterators and 'async iterators'.

But my argument is simply that there is no good reason to aim for the
ability to have object conform to both protocols.  So it shouldn't be
a criterion when looking at the merits of a proposal.  I may very well
be wrong but I haven't yet seen a compelling case for an object
implementing both protocols.

Cheers,

-- 
Arnaud
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Paul Moore
On 6 May 2015 at 09:20, Greg Ewing greg.ew...@canterbury.ac.nz wrote:
 That doesn't cover any of the higher level abstractions like tasks or
 futures (at least not by those names or with those interfaces).

 Because a minimal event loop doesn't *need* those.

It doesn't *need* them, but as abstractions they allow easier building
of reusable higher-level libraries. You can write an event loop with
nothing but coroutines, but to build reusable libraries on top of it,
you need some common interfaces.

 In my little scheduler, a task is nothing more than
 a yield-frommable object sitting on a queue of things
 to be run. There is no need to wrap it in another
 object.

 And there's really no need for the concept of a
 future at all, except maybe at the boundary
 between generator-based async code and other things
 that are based on callbacks. Even then, a future
 is really just an object that can be passed to
 yield-from. There is no need for a concrete
 Future class, it's just a protocol.

Agreed, you don't need a Future class, all you need is to agree what
reusable code is allowed to do with the core objects you are passing
around - that's how duck typing works. The objects *I* can see are
futures (in a PEP 492 world, awaitables which may or may not be
equivalent in terms of the operations you'd want to focus on) and the
event loop itself.

In your example, the event loop is implicit (as it's a singleton, you
use global functions rather than methods on the loop object) but
that's a minor detail.

 And I
 don't see where the PEP 492 additions would fit in (OK, replace yield
 from with await is part of it, but I don't see the rest).

 That's really all there is to it. The rest is
 concerned with catching certain kinds of mistakes,
 and providing convenient syntax for some patterns
 of using 'await'.

So, things you can wait on have one operation - wait for a result.
That's OK. You can create such things as coroutines, which is also
fine. You may want to create such things explicitly (equivalent to
generators vs __iter__) - maybe that's where __aiter__ comes in in PEP
492 and the Future class in asyncio. Again, all fine.

You also need operations like schedule a thing to run, which is the
event loop interface. Your sample has the following basic event loop
methods that I can see: run, schedule, unschedule, and
expire_timeslice (that last one may be an implementation detail, but
the other 3 seem pretty basic). PEP 492 has nothing to say on the
event loop side of things (something that became clear to me during
this discussion).

 There's a lot of asyncio
 that doesn't seem to me to be IO-related. Specifically the future and
 task abstractions. I view those as relevant to coroutine programming
 in Python because they are referenced in any discussion of coroutines
 (you yield from a future, for example).

 Only because they've been elevated to prominence
 by asyncio and its documentation, which I regard
 as unfortunate.

 When Guido was designing asyncio, I tried very
 hard to dissuade him from giving futures such a
 central place in the model. I saw them as an
 unnecessary concept that would only clutter up
 people's thinking. Seeing all the confusion now,
 I'm more convinced than ever that I was right. :-(

Futures seem to me to be (modulo a few details) what awaitables are
in PEP 492. I can't see how you can meaningfully talk about event
loops in a Python context without having *some* term for things you
wait for. Maybe Future wasn't a good name, and maybe the parallel
with concurrent.futures.Future wasn't helpful (I think both things
were fine, but you may not) but we had to have *some* way of talking
about them, and of constructing standalone awaitables. PEP 492 has
new, and hopefully better, ways, but I think that awaitables *have* to
be central to any model where you wait for things...

By the way, it feels to me like I'm now arguing in favour of PEP 492
with a reasonable understanding of what it means. Assuming what I
said above isn't complete rubbish, thanks to everyone who's helped me
get to this point of understanding through this thread! (And if I
haven't understood, that's my fault, and still thanks to everyone for
their efforts :-))

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Greg Ewing

Paul Moore wrote:


What about Greg Ewing's example?
http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt


That doesn't cover any of the higher level abstractions like tasks or
futures (at least not by those names or with those interfaces).


Because a minimal event loop doesn't *need* those.

In my little scheduler, a task is nothing more than
a yield-frommable object sitting on a queue of things
to be run. There is no need to wrap it in another
object.

And there's really no need for the concept of a
future at all, except maybe at the boundary
between generator-based async code and other things
that are based on callbacks. Even then, a future
is really just an object that can be passed to
yield-from. There is no need for a concrete
Future class, it's just a protocol.


And I
don't see where the PEP 492 additions would fit in (OK, replace yield
from with await is part of it, but I don't see the rest).


That's really all there is to it. The rest is
concerned with catching certain kinds of mistakes,
and providing convenient syntax for some patterns
of using 'await'.


There's a lot of asyncio
that doesn't seem to me to be IO-related. Specifically the future and
task abstractions. I view those as relevant to coroutine programming
in Python because they are referenced in any discussion of coroutines
(you yield from a future, for example).


Only because they've been elevated to prominence
by asyncio and its documentation, which I regard
as unfortunate.

When Guido was designing asyncio, I tried very
hard to dissuade him from giving futures such a
central place in the model. I saw them as an
unnecessary concept that would only clutter up
people's thinking. Seeing all the confusion now,
I'm more convinced than ever that I was right. :-(


In some ways I wish there had been an asyncio library that covered
the areas that are fundamentally about IO multiplexing. And a separate
library (just async, maybe, although that's now a bad idea as it
clashes with a keyword :-)) that covered generic event loop, task and
synchronisation areas.


As I said before, I don't think it's really
possible to factor an event loop into those kinds
of parts. You may be able to factor the *API* that
way, but any given implementation has to address
all the parts at once.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Wolfgang Langner
Hi Yury,


On 05.05.2015 20:25, Yury Selivanov wrote:


 We forget to address the major problems here. How can someone in a
 sync script use this async stuff easy. How can async and sync stuff
 cooperate and we don't need to rewrite the world for async stuff.
 How can a normal user access the power of async stuff without rewriting
 all his code. So he can use a simple asyc request library in his code.
 How can a normal user learn and use all this in an easy way.
 
 asyncio and twisted answered these questions ;) The answer is
 that you have to write async implementations.
 
 gevent has a different answer, but greenlents/stackless is
 something that will never be merged in CPython and other
 implementations.
 

I think monkeypatching and the gevent way is wrong. And explicit is better
than implicit.

I have to clear this. I meant how can we make this async stuff more accessible
to the average sync user. Sometime even without writing or knowing how
to write coroutines or other async stuff.

Let me explain it with a deeper example (Some of it is related to Python 2 and
twisted):

I had the same problem for a server application using twisted. Provide
easy interfaces to my users most not aware of async stuff.

My solution was to write my own decorator similar to twisted's
@inlineCallbacks. On top of it I added one more level to the decorator
and distinguish if it was called from the main thread (also running the
mainloop) and other threads. This object usable also as decorator
is called task and has some utility methods. And returns a deferred.
(in asyncio this would be a Future)

Resulting in code like:

@task
def myAsyncFunction(webaddress):
  result = yield fetch(webaddress)
  # only to show sleep example
  yield task.sleep(0.1)
  task.Return(result)

Usable in a sync context (extra script thread):

def do():
  content = myAsyncFunction(http://xyz;)

or async context:

@task
def ado():
  content = yield myAsyncFunction(http://xyz;)


The task decorator has functionality to check if something is
called from the main thread (- also a task and async)
or it is called from another thread (- sync or script).

So this async interface is usable from both worlds. If someone
operates async he/she must only know the task decorator and when to yield.
If he/she uses it in sync mode nothing special has to be done.

To allow all this the server starts the async main loop in the main thread
and executes the script in an extra script thread. The user has every time his
own thread, also for rpc stuff. The only way to switch into the main loop
is to decorate a function as @task, every task is a coroutine and executed
in the main thread (also thread of main loop).

Benefit of all this:

- Easy to write a async task it is marked as one and special stuff belongs
  to the task object. (task.Return is needed because we are in Python 2)
- The normal user executes his stuff in his own thread and he/she
  can program in sync mode. No problem it is an extra thread and the main loop
  does not block.
- A network client or other stuff has to be written only once, most time this
  can be a @task in the async world. But this should not care the end user.
  We don't have to implement all twice once for async and once for the sync
  world. - Less overhead

This is what I mean if I say we must address the bridging problem between the
worlds.
It think it is the wrong way to divide it in async and sync stuff and
duplicate all networking libraries in the sync and async ones.

For me the answer is to write one async netowrk library and use it in both,
a sync script and in an async main loop. With an easy interface and not
forcing the user to know this is an async library I have to do something 
special.

And in future go one step further and use all this combined with PyParallel
to solve the multiprocessing problem in Python.
(Script in extra thread, mainloop in main thread, executed and managed via
PyParallel avoiding the gil)
But this is only a vision/dream of mine.



 And for all this we still can't tell them oh the async stuff solves
 the multiprocessing problem of Python learn it and switch to version
 3.5. It does not and it is only most useful for networking stuff
 nothing more.
 
 networking stuff, and in particular, web, is a huge
 part of current Python usage.  Please don't underestimate
 that.

I do not. But for most they want only use it as a client
and the main concern for most is I want to get this web page
and not I will implement a web client have to do this async and get it.


Regards,

Wolfgang

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Greg Ewing

Guido van Rossum wrote:
the bytecode generated for 
await treats coroutine objects special, just like the bytecode generated 
for yield-from treats generator objects special. The special behavior 
they have in common is the presence of send() and throw() methods,


I don't think that's quit accurate. Yield-from treats
any object having send() and throw() methods the same
way it treats a generator -- there's nothing special
about the generator *type*. Presumably 'await' is the
same.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Greg Ewing

Paul Moore wrote:

It would probably be helpful to have a concrete example of a basic
event loop that did *nothing* but schedule tasks. No IO waiting or
similar, just scheduling.


Take a look at the example I developed when working
on the yield-from pep:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt

The first version of the event loop presented there
does exactly that, just schedules tasks in a round-
robin fashion. Then I gradually add more features
to it.


Actually, what *is* the minimal event loop
interface that is needed for the various task/future mechanisms to
work, independently of asyncio?


I don't it's possible to answer that question,
because there isn't a single answer. The minimal
set of features that an event loop needs depends
on what you want to achieve with it.

Even the notion of just schedules tasks is
ambiguous. What does schedule mean? Does it
just mean round-robin switching between them, or
should they be able to synchronise with each
other in some way? Should it be possible for a
task to suspend itself for an interval of real
world time, or does that come under the heading
of I/O (since you're waiting for an external
event, i.e. the computer's clock reaching some
time)? Etc.

 And what features of an event loop etc
 are needed for the PEP, if it's being used outside of asyncio?)

I don't think *any* particular event loop
features are needed.

You can't pick out any of these features as
being core. For example, it would be possible
to have an event loop that handled socket I/O
but *didn't* do round-robin scheduling -- it
could just keep on running the same task, even
if it yielded, until it blocked waiting for
an external event. Such a scheduler would
probably be quite adequate for many use cases.

It seems to me that the idea of generator-based
tasks managed by an event loop is more of a
design pattern than something you can write a
detailed API specification for.

Another problem with the core idea is that
you can't start with an event loop that just does
scheduling and then add on other features such
as I/O *from the outside*. There has to be some
point at which everything comes together, which
means choosing something like select() or
poll() or I/O completion queues, and build that
into the heart of your event loop. At that point
it's no longer something with a simple core.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Greg Ewing

Paul Moore wrote:

I can't see how you can meaningfully talk about event
loops in a Python context without having *some* term for things you
wait for.


PEP 3152 was my attempt at showing how you could do that.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Ben Darnell
On Tue, May 5, 2015 at 1:39 PM, Paul Moore p.f.mo...@gmail.com wrote:


 It would probably be helpful to have a concrete example of a basic
 event loop that did *nothing* but schedule tasks. No IO waiting or
 similar, just scheduling. I have a gut feeling that event loops are
 more than just asyncio, but without examples to point to it's hard to
 keep a focus on that fact. And even harder to isolate what is an
 event loop mechanism from what is asyncio specific. For example,
 asyncio.BaseEventLoop has a create_connection method. That's
 *obviously* not a fundamental aspect of a generic event loop, But
 call_soon (presumably) is. Having a documented basic event loop
 interface would probably help emphasise the idea than event loops
 don't have to be asyncio. (Actually, what *is* the minimal event loop
 interface that is needed for the various task/future mechanisms to
 work, independently of asyncio? And what features of an event loop etc
 are needed for the PEP, if it's being used outside of asyncio?)


Twisted has a pretty good taxonomy of event loop methods, in the interfaces
at the bottom of this page:
http://twistedmatrix.com/documents/15.1.0/core/howto/reactor-basics.html
and the comparison matrix at
http://twistedmatrix.com/documents/15.1.0/core/howto/choosing-reactor.html

The asyncio event loops implement most of these (not the exact interfaces,
but the same functionality). Tornado implements FDSet, Time, and part of
Threads in the IOLoop itself, with the rest of the functionality coming
from separate classes. (You may wonder then why Twisted and asyncio put
everything in the core event loop? It's necessary to abstract over certain
platform differences, which is one big reason why Tornado has poor support
for Windows).

-Ben




 I guess the other canonical event loop use case is GUI system message
 dispatchers.

  You can argue that the syntax is needed to help
  make async more accessible - but if that's the case then the
  terminology debates and confusion are clear evidence that it's not
  succeeding in that goal.
 
  Perhaps, but arguing about the nitty-gritty details of something doesn't
  automatically lead to a clearer understanding of the higher level
 concept.
  Discussing how turning a steering wheel in a car might help you grasp how
  cars turn, but it isn't a requirement to get turn the wheel left to make
  the car go left.

 Fair point. If only I could avoid driving into walls :-)

  Of course, that's based on my perception of
  one of the goals of the PEP as being make coroutines and asyncio more
  accessible, If the actual goals are different, my conclusion is
  invalid.
 
  I think the goal is make coroutines easier to use and does not directly
  relate to asyncio.

 OK. But in that case, some examples using a non-asyncio toy just
 schedule tasks event loop might help.

  Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
  but that seems to be callback-based rather than coroutine-based?
 
  Yep.

 ... and so you can't use it with async/await?

  Many people use requests for their web access. There are good reasons
  for this. Are you saying that until someone steps up and writes an
  async implementation of requests, I have to make a choice - requests
  or asyncio?
 
  I believe so; you need something to implement __await__. This is true in
 any
  language that implements co-routines.
 
  Unfortunately, I can't see myself choosing asyncio in that
  situation. Which again means that asyncio becomes something that the
  average user can't use. Which in turn further entrenches it as a
  specialist-only tool.
 
  You forgot to append ... yet to that statement. Just because something
  isn't available out of the box without some effort to support doesn't
 mean
  it will never happen, else there would be absolutely no Python 3 users
 out
  there.

 Fair point. Yuri mentioned aiohttp, as well. The one difference
 between this and Python 2/3, is that here you *have* to have two
 separate implementations. There's no equivalent of a shared source
 async and synchronous implementation of requests. So the typical
 please support Python 3 issue that motivates projects to move
 forward doesn't exist in the same way. It's not to say that there
 won't be async versions of important libraries, it's just hard to see
 how the dynamics will work. I can't see myself raising an issue on
 cx_Oracle saying please add asyncio support, and I don't know who
 else I would ask...

  Co-routine-based asynchronous programming is new to Python, so as a
  community we don't have it as something everyone learns over time. If you
  don't come from an area that supports it then it will be foreign to you
 and
  not make sense without someone giving you a good tutorial on it. But
  considering C#, Dart, and Ecmascript 6 (will) have co-routine support --
 and
  those are just the languages I can name off the top of my head -- using
 the
  exact same keywords suggests to me that it isn't *that* 

Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Koos Zevenhoven
tds333 at gmail.com tds333 at gmail.com writes:

 
 Hi,
 
 still watching progress here. Read all posts and changes.
 
 Everything improved and I know it is a lot of work. Thx for doing this.
 
 But I still think this PEP goes to far.
 
 [...]
 
 We forget to address the major problems here. How can someone in a
 sync script use this async stuff easy. How can async and sync stuff
 cooperate and we don't need to rewrite the world for async stuff.
 How can a normal user access the power of async stuff without rewriting
 all his code. So he can use a simple asyc request library in his code.
 How can a normal user learn and use all this in an easy way.
 
 [...]

Hi Wolfgang,

You may want to see what I just posted on python-ideas. What I wrote about 
is related to several things you mention, and might provide a remedy.

-- Koos



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Nathaniel Smith
On May 5, 2015 2:14 PM, Guido van Rossum gu...@python.org wrote:

 In the PEP 492 world, these concepts map as follows:

 - Future translates to something with an __await__ method (and asyncio
Futures are trivially made compliant by defining Future.__await__ as an
alias for Future.__iter__);

 - asyncio coroutine maps to PEP 492 coroutine object (either defined
with `async def` or a generator decorated with @types.coroutine -- note
that @asyncio.coroutine incorporates the latter);

 - either of the above maps to awaitable.

Err, aren't the first and third definitions above identical?

Surely we want to say: an async def function is a convenient shorthand for
creating a custom awaitable (exactly like how generators are a convenient
shorthand for creating custom iterators), and a Future is-an awaitable that
also adds some extra methods.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Guido van Rossum
On Tue, May 5, 2015 at 3:01 PM, Nathaniel Smith n...@pobox.com wrote:

On May 5, 2015 2:14 PM, Guido van Rossum gu...@python.org wrote:
 
  In the PEP 492 world, these concepts map as follows:
 
  - Future translates to something with an __await__ method (and asyncio
 Futures are trivially made compliant by defining Future.__await__ as an
 alias for Future.__iter__);
 
  - asyncio coroutine maps to PEP 492 coroutine object (either defined
 with `async def` or a generator decorated with @types.coroutine -- note
 that @asyncio.coroutine incorporates the latter);
 
  - either of the above maps to awaitable.

 Err, aren't the first and third definitions above identical?

 Surely we want to say: an async def function is a convenient shorthand for
 creating a custom awaitable (exactly like how generators are a convenient
 shorthand for creating custom iterators), and a Future is-an awaitable that
 also adds some extra methods.


The current PEP 492 proposal does endow the object returned by calling an
async function (let's call it a coroutine object) with an __await__ method.
And there's a good reason for this -- the bytecode generated for await
treats coroutine objects special, just like the bytecode generated for
yield-from treats generator objects special. The special behavior they have
in common is the presence of send() and throw() methods, which are used to
allow send() and throw() calls on the outer generator to be passed into the
inner generator with minimal fuss. (This is the reason why yield from X
is *not* equivalent to for x in X: yield x.)

@Yury: I have a feeling the PEP could use more clarity here -- perhaps the
section Await Expression should explain what the interepreter does for
each type of awaitable?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Brett Cannon
On Tue, May 5, 2015 at 3:14 PM Paul Moore p.f.mo...@gmail.com wrote:

 On 5 May 2015 at 19:25, Yury Selivanov yselivanov...@gmail.com wrote:
  On 2015-05-05 7:27 AM, Wolfgang wrote:
  Even the discussion on python-dev suggests there is some time needed
  to finalize all this.
 
  I'd say that:
 
  80% of the recent discussion of the PEP is about terminology.
  10% is about whether we should have __future__ import or not.

 But the terminology discussion appears to revolve around people
 finding the various concepts involved in asyncio (particularly the new
 PEP, but also to an extent the existing implementation) confusing. I
 can confirm, having tried to work through the asyncio docs, that the
 underlying concepts and how they are explained, are confusing to an
 outsider.

 That's not to say that everything needs to be beginner-friendly, but
 it *does* mean that it's hard for the wider Python community to
 meaningfully comment, or evaluate or sanity-check the design. We're
 left with a sense of trust us, it makes sense if you need it,
 everyone else can ignore it.


Watch David Beazley's talk from PyCon this year and you can watch him
basically re-implement asyncio on stage in under 45 minutes. It's not as
complicated as it seems when you realize there is an event loop driving
everything (which people have been leaving out of the conversation since it
doesn't tie into the syntax directly).



 Personally, I feel as if PEP 492 is looking a little premature - maybe
 the focus should be on making asyncio more accessible first, and
 *then* adding syntax.


I think this ties the concept of adding syntax to Python to make
coroutine-based programming easier too much to asyncio; the latter is just
an implementation of the former. This PEP doesn't require asyncio beyond
the fact that will be what provides the initial event loop in the stdlib.


 You can argue that the syntax is needed to help
 make async more accessible - but if that's the case then the
 terminology debates and confusion are clear evidence that it's not
 succeeding in that goal.


Perhaps, but arguing about the nitty-gritty details of something doesn't
automatically lead to a clearer understanding of the higher level concept.
Discussing how turning a steering wheel in a car might help you grasp how
cars turn, but it isn't a requirement to get turn the wheel left to make
the car go left.


 Of course, that's based on my perception of
 one of the goals of the PEP as being make coroutines and asyncio more
 accessible, If the actual goals are different, my conclusion is
 invalid.


I think the goal is make coroutines easier to use and does not directly
relate to asyncio.



  We forget to address the major problems here. How can someone in a
  sync script use this async stuff easy. How can async and sync stuff
  cooperate and we don't need to rewrite the world for async stuff.
  How can a normal user access the power of async stuff without rewriting
  all his code. So he can use a simple asyc request library in his code.
  How can a normal user learn and use all this in an easy way.
 
  asyncio and twisted answered these questions ;) The answer is
  that you have to write async implementations.

 Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
 but that seems to be callback-based rather than coroutine-based?


Yep.



 Many people use requests for their web access. There are good reasons
 for this. Are you saying that until someone steps up and writes an
 async implementation of requests, I have to make a choice - requests
 or asyncio?


I believe so; you need something to implement __await__. This is true in
any language that implements co-routines.

Unfortunately, I can't see myself choosing asyncio in that
 situation. Which again means that asyncio becomes something that the
 average user can't use. Which in turn further entrenches it as a
 specialist-only tool.


You forgot to append ... yet to that statement. Just because something
isn't available out of the box without some effort to support doesn't mean
it will never happen, else there would be absolutely no Python 3 users out
there.



 As another example, in Twisted I could use defer_to_thread to
 integrate Oracle database access into a twisted application (that's
 what the twisted database stuff did under the hood). Can I do that
 with asyncio? Will the syntax in the PEP help, hinder or be irrelevant
 to that?

  And for all this we still can't tell them oh the async stuff solves
  the multiprocessing problem of Python learn it and switch to version
  3.5. It does not and it is only most useful for networking stuff
  nothing more.
 
  networking stuff, and in particular, web, is a huge
  part of current Python usage.  Please don't underestimate
  that.

 Without async versions of requests and similar, how much of a chunk of
 the networking/web area will asyncio take? (Genuine question, I have
 no idea). And how much extra will this PEP add? Those may not be fair
 

Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Paul Moore
On 5 May 2015 at 19:25, Yury Selivanov yselivanov...@gmail.com wrote:
 On 2015-05-05 7:27 AM, Wolfgang wrote:
 Even the discussion on python-dev suggests there is some time needed
 to finalize all this.

 I'd say that:

 80% of the recent discussion of the PEP is about terminology.
 10% is about whether we should have __future__ import or not.

But the terminology discussion appears to revolve around people
finding the various concepts involved in asyncio (particularly the new
PEP, but also to an extent the existing implementation) confusing. I
can confirm, having tried to work through the asyncio docs, that the
underlying concepts and how they are explained, are confusing to an
outsider.

That's not to say that everything needs to be beginner-friendly, but
it *does* mean that it's hard for the wider Python community to
meaningfully comment, or evaluate or sanity-check the design. We're
left with a sense of trust us, it makes sense if you need it,
everyone else can ignore it.

Personally, I feel as if PEP 492 is looking a little premature - maybe
the focus should be on making asyncio more accessible first, and
*then* adding syntax. You can argue that the syntax is needed to help
make async more accessible - but if that's the case then the
terminology debates and confusion are clear evidence that it's not
succeeding in that goal. Of course, that's based on my perception of
one of the goals of the PEP as being make coroutines and asyncio more
accessible, If the actual goals are different, my conclusion is
invalid.

 We forget to address the major problems here. How can someone in a
 sync script use this async stuff easy. How can async and sync stuff
 cooperate and we don't need to rewrite the world for async stuff.
 How can a normal user access the power of async stuff without rewriting
 all his code. So he can use a simple asyc request library in his code.
 How can a normal user learn and use all this in an easy way.

 asyncio and twisted answered these questions ;) The answer is
 that you have to write async implementations.

Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
but that seems to be callback-based rather than coroutine-based?

Many people use requests for their web access. There are good reasons
for this. Are you saying that until someone steps up and writes an
async implementation of requests, I have to make a choice - requests
or asyncio? Unfortunately, I can't see myself choosing asyncio in that
situation. Which again means that asyncio becomes something that the
average user can't use. Which in turn further entrenches it as a
specialist-only tool.

As another example, in Twisted I could use defer_to_thread to
integrate Oracle database access into a twisted application (that's
what the twisted database stuff did under the hood). Can I do that
with asyncio? Will the syntax in the PEP help, hinder or be irrelevant
to that?

 And for all this we still can't tell them oh the async stuff solves
 the multiprocessing problem of Python learn it and switch to version
 3.5. It does not and it is only most useful for networking stuff
 nothing more.

 networking stuff, and in particular, web, is a huge
 part of current Python usage.  Please don't underestimate
 that.

Without async versions of requests and similar, how much of a chunk of
the networking/web area will asyncio take? (Genuine question, I have
no idea). And how much extra will this PEP add? Those may not be fair
questions (and even if they are fair, the answers are probably
unknowable), but as an outsider, I feel only the costs of the asyncio
implementation (a new library that I don't understand, and now a
relatively large amount of new syntax and special methods I have to
ignore because they don't make sense to me). That's OK, but I think I
am being reasonable to ask for some sense of the level of benefits
others are getting to balance out the costs I incur.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Yury Selivanov

Paul,

On 2015-05-05 3:14 PM, Paul Moore wrote:

On 5 May 2015 at 19:25, Yury Selivanov yselivanov...@gmail.com wrote:

On 2015-05-05 7:27 AM, Wolfgang wrote:

Even the discussion on python-dev suggests there is some time needed
to finalize all this.

I'd say that:

80% of the recent discussion of the PEP is about terminology.
10% is about whether we should have __future__ import or not.

But the terminology discussion appears to revolve around people
finding the various concepts involved in asyncio (particularly the new
PEP, but also to an extent the existing implementation) confusing. I
can confirm, having tried to work through the asyncio docs, that the
underlying concepts and how they are explained, are confusing to an
outsider.


I agree. We have to improve asyncio docs in this area.



That's not to say that everything needs to be beginner-friendly, but
it *does* mean that it's hard for the wider Python community to
meaningfully comment, or evaluate or sanity-check the design. We're
left with a sense of trust us, it makes sense if you need it,
everyone else can ignore it.

Personally, I feel as if PEP 492 is looking a little premature - maybe
the focus should be on making asyncio more accessible first, and
*then* adding syntax. You can argue that the syntax is needed to help
make async more accessible - but if that's the case then the
terminology debates and confusion are clear evidence that it's not
succeeding in that goal. Of course, that's based on my perception of
one of the goals of the PEP as being make coroutines and asyncio more
accessible, If the actual goals are different, my conclusion is
invalid.


Again, PEP 492 is not only for asyncio. *Any* framework can
use it, including Twisted.

As for terminology, I view this discussion differently.  It's
not about the technical details (Python has asymmetric
coroutines, that's it), but rather on how to disambiguate
coroutines implemented with generators and yield-from, from
new 'async def' coroutines.  I can't see any fundamental
problem with the PEP behind such discussions.




We forget to address the major problems here. How can someone in a
sync script use this async stuff easy. How can async and sync stuff
cooperate and we don't need to rewrite the world for async stuff.
How can a normal user access the power of async stuff without rewriting
all his code. So he can use a simple asyc request library in his code.
How can a normal user learn and use all this in an easy way.

asyncio and twisted answered these questions ;) The answer is
that you have to write async implementations.

Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
but that seems to be callback-based rather than coroutine-based?

Many people use requests for their web access. There are good reasons
for this. Are you saying that until someone steps up and writes an
async implementation of requests, I have to make a choice - requests
or asyncio? Unfortunately, I can't see myself choosing asyncio in that
situation. Which again means that asyncio becomes something that the
average user can't use. Which in turn further entrenches it as a
specialist-only tool.


There is aiohttp library [1], which provides a client API
similar to requests.

And if you want to write high performance networking server
in python3 you *will* choose asyncio (or gevent/twisted
in python2).

And PEP 492 is aimed to make this whole async stuff more
accessible to an average user.



As another example, in Twisted I could use defer_to_thread to
integrate Oracle database access into a twisted application (that's
what the twisted database stuff did under the hood). Can I do that
with asyncio? Will the syntax in the PEP help, hinder or be irrelevant
to that?


You can use 'loop.run_in_executor' in asyncio. It returns a
future that you can await on.  You can also provide a nice
facade for your Oracle-database code that provides a nice
API but uses asyncio thread executor behind the scenes.




And for all this we still can't tell them oh the async stuff solves
the multiprocessing problem of Python learn it and switch to version
3.5. It does not and it is only most useful for networking stuff
nothing more.

networking stuff, and in particular, web, is a huge
part of current Python usage.  Please don't underestimate
that.

Without async versions of requests and similar, how much of a chunk of
the networking/web area will asyncio take? (Genuine question, I have
no idea).


There are some things (like websockets) that are hard
to implement correctly in existing frameworks like django
and flask.  And these kind of things are becoming more
and more important.  Languages like Go were designed
specifically to allow writing efficient


And how much extra will this PEP add? Those may not be fair
questions (and even if they are fair, the answers are probably
unknowable), but as an outsider, I feel only the costs of the asyncio
implementation (a new library that I don't understand, and now a
relatively 

Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Paul Moore
(Yury gave similar responses, so (a) I'll just respond here, and (b)
it's encouraging that you both responded so quickly with the same
message)

On 5 May 2015 at 20:44, Brett Cannon br...@python.org wrote:
 That's not to say that everything needs to be beginner-friendly, but
 it *does* mean that it's hard for the wider Python community to
 meaningfully comment, or evaluate or sanity-check the design. We're
 left with a sense of trust us, it makes sense if you need it,
 everyone else can ignore it.

 Watch David Beazley's talk from PyCon this year and you can watch him
 basically re-implement asyncio on stage in under 45 minutes. It's not as
 complicated as it seems when you realize there is an event loop driving
 everything (which people have been leaving out of the conversation since it
 doesn't tie into the syntax directly).

I'll watch that - it should be fun. But I have seen things like that
before, and I've got an idea how to write an event loop. You're right
that it's easy to lose track of the fundamentally simple idea in all
the complex discussions. To me that feels like a peculiar failure of
the abstraction, in that in some circumstances it makes things feel
*more* complex than they are :-)

 Personally, I feel as if PEP 492 is looking a little premature - maybe
 the focus should be on making asyncio more accessible first, and
 *then* adding syntax.

 I think this ties the concept of adding syntax to Python to make
 coroutine-based programming easier too much to asyncio; the latter is just
 an implementation of the former. This PEP doesn't require asyncio beyond the
 fact that will be what provides the initial event loop in the stdlib.

It's very hard to separate coroutines from asyncio, because there's no
other example (not even a toy one) to reason about.

It would probably be helpful to have a concrete example of a basic
event loop that did *nothing* but schedule tasks. No IO waiting or
similar, just scheduling. I have a gut feeling that event loops are
more than just asyncio, but without examples to point to it's hard to
keep a focus on that fact. And even harder to isolate what is an
event loop mechanism from what is asyncio specific. For example,
asyncio.BaseEventLoop has a create_connection method. That's
*obviously* not a fundamental aspect of a generic event loop, But
call_soon (presumably) is. Having a documented basic event loop
interface would probably help emphasise the idea than event loops
don't have to be asyncio. (Actually, what *is* the minimal event loop
interface that is needed for the various task/future mechanisms to
work, independently of asyncio? And what features of an event loop etc
are needed for the PEP, if it's being used outside of asyncio?)

I guess the other canonical event loop use case is GUI system message
dispatchers.

 You can argue that the syntax is needed to help
 make async more accessible - but if that's the case then the
 terminology debates and confusion are clear evidence that it's not
 succeeding in that goal.

 Perhaps, but arguing about the nitty-gritty details of something doesn't
 automatically lead to a clearer understanding of the higher level concept.
 Discussing how turning a steering wheel in a car might help you grasp how
 cars turn, but it isn't a requirement to get turn the wheel left to make
 the car go left.

Fair point. If only I could avoid driving into walls :-)

 Of course, that's based on my perception of
 one of the goals of the PEP as being make coroutines and asyncio more
 accessible, If the actual goals are different, my conclusion is
 invalid.

 I think the goal is make coroutines easier to use and does not directly
 relate to asyncio.

OK. But in that case, some examples using a non-asyncio toy just
schedule tasks event loop might help.

 Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
 but that seems to be callback-based rather than coroutine-based?

 Yep.

... and so you can't use it with async/await?

 Many people use requests for their web access. There are good reasons
 for this. Are you saying that until someone steps up and writes an
 async implementation of requests, I have to make a choice - requests
 or asyncio?

 I believe so; you need something to implement __await__. This is true in any
 language that implements co-routines.

 Unfortunately, I can't see myself choosing asyncio in that
 situation. Which again means that asyncio becomes something that the
 average user can't use. Which in turn further entrenches it as a
 specialist-only tool.

 You forgot to append ... yet to that statement. Just because something
 isn't available out of the box without some effort to support doesn't mean
 it will never happen, else there would be absolutely no Python 3 users out
 there.

Fair point. Yuri mentioned aiohttp, as well. The one difference
between this and Python 2/3, is that here you *have* to have two
separate implementations. There's no equivalent of a shared source
async and synchronous 

Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Paul Moore
On 5 May 2015 at 21:38, Guido van Rossum gu...@python.org wrote:
 Jumping in to correct one fact.

 On Tue, May 5, 2015 at 12:44 PM, Brett Cannon br...@python.org wrote:


 On Tue, May 5, 2015 at 3:14 PM Paul Moore p.f.mo...@gmail.com wrote:

 Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
 but that seems to be callback-based rather than coroutine-based?


 Yep.


 The run_in_executor call is not callback-based -- the confusion probably
 stems from the name of the function argument ('callback'). It actually
 returns a Future representing the result (or error) of an operation, where
 the operation is represented by the function argument. So if you have e.g. a
 function

 def factorial(n):
 return 1 if n = 0 else n*factorial(n-1)

 you can run it in an executor from your async(io) code like this:

 loop = asyncio.get_event_loop()
 result = yield from loop.run_in_executor(factorial, 100)

 (In a PEP 492 coroutine substitute await for yield from.)

Thanks, that's an important correction. Given that, run_in_executor is
the link to blocking calls that I was searching for. And yes, the
callback terminology does make this far from obvious, unfortunately.
As does the point at which it's introduced (before futures have been
described) and the fact that it says this method is a coroutine
rather than this method returns a Future[1].

Paul

[1] I'm still struggling to understand the terminology, so if those
two statements are equivalent, that's not yet obvious to me.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Guido van Rossum
Jumping in to correct one fact.

On Tue, May 5, 2015 at 12:44 PM, Brett Cannon br...@python.org wrote:


 On Tue, May 5, 2015 at 3:14 PM Paul Moore p.f.mo...@gmail.com wrote:

 Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
 but that seems to be callback-based rather than coroutine-based?


 Yep.


The run_in_executor call is not callback-based -- the confusion probably
stems from the name of the function argument ('callback'). It actually
returns a Future representing the result (or error) of an operation, where
the operation is represented by the function argument. So if you have e.g.
a function

def factorial(n):
return 1 if n = 0 else n*factorial(n-1)

you can run it in an executor from your async(io) code like this:

loop = asyncio.get_event_loop()
result = yield from loop.run_in_executor(factorial, 100)

(In a PEP 492 coroutine substitute await for yield from.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Guido van Rossum
On Tue, May 5, 2015 at 1:39 PM, Paul Moore p.f.mo...@gmail.com wrote:

 It's very hard to separate coroutines from asyncio, because there's no
 other example (not even a toy one) to reason about.


What about Greg Ewing's example?
http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Yury Selivanov

On 2015-05-05 4:39 PM, Paul Moore wrote:

Is there anyone who feels they could write a stripped down but working
example of a valid Python event loop*without*  the asyncio aspects? Or
is that what David Beazley's talk does?

Yes, in David's talk, where he starts to use 'yield from' you
can simply use new coroutines.


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Yury Selivanov

Hi Wolfgang,


On 2015-05-05 7:27 AM, Wolfgang wrote:

Hi,

[..]

Even the discussion on python-dev suggests there is some time needed
to finalize all this.


I'd say that:

80% of the recent discussion of the PEP is about terminology.
10% is about whether we should have __future__ import or not.



We forget to address the major problems here. How can someone in a
sync script use this async stuff easy. How can async and sync stuff
cooperate and we don't need to rewrite the world for async stuff.
How can a normal user access the power of async stuff without rewriting
all his code. So he can use a simple asyc request library in his code.
How can a normal user learn and use all this in an easy way.


asyncio and twisted answered these questions ;) The answer is
that you have to write async implementations.

gevent has a different answer, but greenlents/stackless is
something that will never be merged in CPython and other
implementations.



And for all this we still can't tell them oh the async stuff solves
the multiprocessing problem of Python learn it and switch to version
3.5. It does not and it is only most useful for networking stuff
nothing more.


networking stuff, and in particular, web, is a huge
part of current Python usage.  Please don't underestimate
that.


Thanks,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Guido van Rossum
On Tue, May 5, 2015 at 1:44 PM, Paul Moore p.f.mo...@gmail.com wrote:

 [Guido]
  The run_in_executor call is not callback-based -- the confusion probably
  stems from the name of the function argument ('callback'). It actually
  returns a Future representing the result (or error) of an operation,
 where
  the operation is represented by the function argument. So if you have
 e.g. a
  function
 
  def factorial(n):
  return 1 if n = 0 else n*factorial(n-1)
 
  you can run it in an executor from your async(io) code like this:
 
  loop = asyncio.get_event_loop()
  result = yield from loop.run_in_executor(factorial, 100)
 
  (In a PEP 492 coroutine substitute await for yield from.)

 Thanks, that's an important correction. Given that, run_in_executor is
 the link to blocking calls that I was searching for. And yes, the
 callback terminology does make this far from obvious, unfortunately.
 As does the point at which it's introduced (before futures have been
 described) and the fact that it says this method is a coroutine
 rather than this method returns a Future[1].

 Paul

 [1] I'm still struggling to understand the terminology, so if those
 two statements are equivalent, that's not yet obvious to me.


I apologize for the confusing documentation. We need more help from
qualified tech writers! Writing PEP 3156 was a huge undertaking for me;
after that I was exhausted and did not want to take on writing the end user
documentation as well, so it was left unfinished. :-(

In PEP 3156 (asyncio package) there are really three separate concepts:

- Future, which is a specific class (of which Task is a subclass);

- coroutine, by which in this context is meant a generator object obtained
by calling a generator function decorated with @asyncio.coroutine and
written to conform to the asyncio protocol for coroutines (i.e. don't use
bare yield, only use yield from, and the latter always with either a Future
or a coroutine as argument);

- either of the above, which is actually the most common requirement --
most asyncio functions that support one also support the other, and either
is allowable as the argument to `yield from`.

In the implementation we so often flipped between Future and coroutine that
I imagine sometimes the implementation and docs differ; also, we don't have
a good short name for either of the above so we end up using one or the
other as a shorthand.

*Unless* you want to attach callbacks, inspect the result or exception, or
cancel it (all of which require a Future), your code shouldn't be concerned
about the difference -- you should just use `res = yield from func(args)`
and use try/except to catch exceptions if you care. And if you do need a
Future, you can call the function asyncio.async() on it (which in PEP 492
is renamed to ensure_future()).

In the PEP 492 world, these concepts map as follows:

- Future translates to something with an __await__ method (and asyncio
Futures are trivially made compliant by defining Future.__await__ as an
alias for Future.__iter__);

- asyncio coroutine maps to PEP 492 coroutine object (either defined
with `async def` or a generator decorated with @types.coroutine -- note
that @asyncio.coroutine incorporates the latter);

- either of the above maps to awaitable.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Yury Selivanov



On 2015-05-05 5:31 PM, Jim J. Jewett wrote:

Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote:


As for terminology, I view this discussion differently.  It's
not about the technical details (Python has asymmetric
coroutines, that's it), but rather on how to disambiguate
coroutines implemented with generators and yield-from, from
new 'async def' coroutines.

Not just How?, but Why?.

Why do they*need*  to be disambiguated?


To clearly show how one interacts with the other, to explain
how backwards compatibility is implemented, and to better
illustrate some additional (and necessary) restrictions
we put on 'async def' coroutines.  Otherwise, the PEP would
be completely unreadable :)


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Guido van Rossum
On Tue, May 5, 2015 at 2:29 PM, Paul Moore p.f.mo...@gmail.com wrote:

 On 5 May 2015 at 22:12, Guido van Rossum gu...@python.org wrote:
  I apologize for the confusing documentation. We need more help from
  qualified tech writers! Writing PEP 3156 was a huge undertaking for me;
  after that I was exhausted and did not want to take on writing the end
 user
  documentation as well, so it was left unfinished. :-(

 Fair enough. When I properly document one of my projects, *then* I'll
 think about complaining :-) These things happen.

  In PEP 3156 (asyncio package) there are really three separate concepts:
 
  - Future, which is a specific class (of which Task is a subclass);
 
  - coroutine, by which in this context is meant a generator object
 obtained
  by calling a generator function decorated with @asyncio.coroutine and
  written to conform to the asyncio protocol for coroutines (i.e. don't use
  bare yield, only use yield from, and the latter always with either a
 Future
  or a coroutine as argument);
 
  - either of the above, which is actually the most common requirement --
 most
  asyncio functions that support one also support the other, and either is
  allowable as the argument to `yield from`.
 
  In the implementation we so often flipped between Future and coroutine
 that
  I imagine sometimes the implementation and docs differ; also, we don't
 have
  a good short name for either of the above so we end up using one or the
  other as a shorthand.

 OK, that makes a lot of sense.

  *Unless* you want to attach callbacks, inspect the result or exception,
 or
  cancel it (all of which require a Future), your code shouldn't be
 concerned
  about the difference -- you should just use `res = yield from func(args)`
  and use try/except to catch exceptions if you care. And if you do need a
  Future, you can call the function asyncio.async() on it (which in PEP
 492 is
  renamed to ensure_future()).

 Again, makes sense. Although there are some bits of example code in
 the docs that call asyncio.async() on a coroutine and throw away the
 result (for example,

 https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-until-complete
 ).
 That confuses me. Are you saying that async() modifies its (coroutine)
 argument to make it a Future? Rather than wrapping a coroutine in a
 Future, which gets returned?


No, it wraps a coroutine (i.e. a generator) in a Task, but leaves a Future
alone. I'm stumped why that example calls async() and then throws the
result away. I suspect it won't work without it (or else Victor wouldn't
have added the call) but the reason seems, um, deep. I think wrapping it in
a Task enters the generator in the event loop's queue of runnables --
otherwise the generator may well be garbage-collected without ever running.

Such complexity doesn't belong in such a simple example though.


  In the PEP 492 world, these concepts map as follows:
 
  - Future translates to something with an __await__ method (and asyncio
  Futures are trivially made compliant by defining Future.__await__ as an
  alias for Future.__iter__);
 
  - asyncio coroutine maps to PEP 492 coroutine object (either defined
  with `async def` or a generator decorated with @types.coroutine -- note
 that
  @asyncio.coroutine incorporates the latter);
 
  - either of the above maps to awaitable.

 OK. Although future is a nicer term than something with an
 __await__ method and the plethora of flavours of coroutine is not
 great. But given that the only term we'll need in common cases is
 awaitable, it's still a net improvement.

 So in the PEP 492 world, there's no such thing as a Task outside of
 asyncio? Or, to put it another way, a Task is only relevant in an IO
 context (unless an alternative event loop library implemented a
 similar concept), and we should only be talking in terms of awaitables
 and futures (given concurrent.futures and asyncio, I doubt you're
 going to be able to stop people using Future for the generic term
 for something with an __await__ method at best, and quite possibly
 as equivalent to awaitable, unfortunately).


I'm not sure. But it's true that Futures and Tasks in asyncio serve the
purpose of linking the event loop (whose basic functioning is
callback-based) to coroutines (implemented by generators). The basic idea
is that when some I/O completes the event loop will call a callback
function registered for that particular I/O operation; the callback then is
actually a bound method of a Future or Task that causes the latter to be
marked as complete (i.e. having a result) which in turn will call other
callbacks (registered with the Future using add_done_callback()); in the
case of a Task (i.e. a special kind of Future that wraps a
generator/coroutine) this will resume the coroutine. (Actually it may
resume an entire stack of coroutines that are blocked waiting for each
other at yield-from; in my spare time I'm working on an explanation of the
machinery underlying yield, yield 

Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Paul Moore
On 5 May 2015 at 21:57, Guido van Rossum gu...@python.org wrote:
 On Tue, May 5, 2015 at 1:39 PM, Paul Moore p.f.mo...@gmail.com wrote:

 It's very hard to separate coroutines from asyncio, because there's no
 other example (not even a toy one) to reason about.

 What about Greg Ewing's example?
 http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt

That doesn't cover any of the higher level abstractions like tasks or
futures (at least not by those names or with those interfaces). And I
don't see where the PEP 492 additions would fit in (OK, replace yield
from with await is part of it, but I don't see the rest).

We may be talking at cross purposes here. There's a lot of asyncio
that doesn't seem to me to be IO-related. Specifically the future and
task abstractions. I view those as relevant to coroutine programming
in Python because they are referenced in any discussion of coroutines
(you yield from a future, for example). If you see them as purely
asyncio related (and not directly usable from outside of an asyncio
context) then that may explain some of my confusion (but at the cost
of reducing the coroutine concept to something pretty trivial in the
absence of a library that independently implements these concepts).

In some ways I wish there had been an asyncio library that covered
the areas that are fundamentally about IO multiplexing. And a separate
library (just async, maybe, although that's now a bad idea as it
clashes with a keyword :-)) that covered generic event loop, task and
synchronisation areas. But that's water under the bridge now.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-05 Thread Jim J. Jewett

Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote:

 As for terminology, I view this discussion differently.  It's
 not about the technical details (Python has asymmetric
 coroutines, that's it), but rather on how to disambiguate
 coroutines implemented with generators and yield-from, from
 new 'async def' coroutines.

Not just How?, but Why?.

Why do they *need* to be disambiguated?

With the benefit of having recently read all that discussion
(as opposed to just the PEP), my answer is ... uh ... that
generators vs async def is NOT an important distinction.
What matters (as best I can tell) is:

something using yield (or yield from) to mark execution context switches

  vs

other kinds of callables, including those using yield to make an iterator


I'm not quite sure that the actual proposal even really separates them
effectively, in part because the terminology keeps suggesting other
distinctions instead.  (The glossary does help; just not enough.)


-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them.  -jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-03 Thread Steven D'Aprano
On Fri, May 01, 2015 at 09:24:47PM +0100, Arnaud Delobelle wrote:

 I'm not convinced that allowing an object to be both a normal and an
 async iterator is a good thing.  It could be a recipe for confusion.

In what way?

I'm thinking that the only confusion would be if you wrote async for 
instead of for, or vice versa, and instead of getting an exception you 
got the (a)syncronous behaviour you didn't want.

But I have no intuition for how likely it is that you could write an 
asyncronous for loop, leave out the async, and still have the code do 
something meaningful.

Other than that, I think it would be fine to have an object be both a 
syncronous and asyncronous iterator. You specify the behaviour you want 
by how you use it. We can already do that, e.g. unittest's assertRaises 
is both a test assertion and a context manager.

Objects can have multiple roles, and it's not usually abused, or 
confusing. I'm not sure that async iterables will be any different.



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-02 Thread Arnaud Delobelle
On 1 May 2015 at 21:27, Yury Selivanov yselivanov...@gmail.com wrote:
 On 2015-05-01 4:24 PM, Arnaud Delobelle wrote:

 On 1 May 2015 at 20:24, Yury Selivanov yselivanov...@gmail.com wrote:

 On 2015-05-01 3:19 PM, Ethan Furman wrote:

 [...]

 If we must have __aiter__, then we may as well also have __anext__;
 besides
 being more consistent, it also allows an object to be both a normol
 iterator
 and an asynch iterator.


 And this is a good point too.

 I'm not convinced that allowing an object to be both a normal and an
 async iterator is a good thing.  It could be a recipe for confusion.



 I doubt that it will be a popular thing.  But disallowing this
 by merging two different protocols in one isn't a good idea
 either.

I having been arguing for merging two different protocols.  I'm saying
that allowing an object to be both normal and async iterable is not an
argument for having separate protocols because it's not a good thing.

Cheers,

-- 
Arnaud
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-02 Thread Arnaud Delobelle
On 1 May 2015 at 20:24, Yury Selivanov yselivanov...@gmail.com wrote:
 On 2015-05-01 3:19 PM, Ethan Furman wrote:
[...]
 If we must have __aiter__, then we may as well also have __anext__;
 besides
 being more consistent, it also allows an object to be both a normol
 iterator
 and an asynch iterator.


 And this is a good point too.

I'm not convinced that allowing an object to be both a normal and an
async iterator is a good thing.  It could be a recipe for confusion.

-- 
Arnaud
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Glenn Linderman

On 5/1/2015 9:59 AM, Guido van Rossum wrote:


I think coroutine is the name of a concept, not a specific
implementation.

Cheers,

 Cheers indeed! I agree that the *concept* is best called coroutine -- 
and we have used this term ever since PEP 342. But when we're talking 
specifics and trying to distinguish e.g. a function declared with 
'async def' from a regular function or from a regular generator 
function, using 'async function' sounds right. And 'async method' if 
it's a method.


Exactly. The async function/method is an implementation technique for a 
specific kind/subset of coroutine functionality.  So the term coroutine, 
qualified by a description of its best usage and limitationsof async 
function, can be used in defining async function, thus appealing to what 
people know or have heard of and vaguely understand and can read more 
about in the literature.


A glossary entry for coroutine in the docs seems appropriate, which 
could point out the 16† ways to implement coroutine-style 
functionalities in Python, and perhaps make recommendations for 
different types of usage.


†OK, not 16 ways, but it is 3 now, or 4?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Stefan Behnel
Yury Selivanov schrieb am 01.05.2015 um 20:52:
 I don't like the idea of combining __next__ and __anext__.

Ok, fair enough. So, how would you use this new protocol manually then?
Say, I already know that I won't need to await the next item that the
iterator will return. For normal iterators, I could just call next() on it
and continue the for-loop. How would I do it for AIterators?

Stefan


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Guido van Rossum
On Fri, May 1, 2015 at 5:39 AM, Stefan Behnel stefan...@behnel.de wrote:

 Yury Selivanov schrieb am 30.04.2015 um 03:30:
  Asynchronous Iterators and async for
  --
 
  An *asynchronous iterable* is able to call asynchronous code in its
  *iter* implementation, and *asynchronous iterator* can call
  asynchronous code in its *next* method.  To support asynchronous
  iteration:
 
  1. An object must implement an  ``__aiter__`` method returning an
 *awaitable* resulting in an *asynchronous iterator object*.
 
  2. An *asynchronous iterator object* must implement an ``__anext__``
 method returning an *awaitable*.
 
  3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration``
 exception.

 What this section does not explain, AFAICT, nor the section on design
 considerations, is why the iterator protocol needs to be duplicated
 entirely. Can't we just assume (or even validate) that any 'regular'
 iterator returned from __aiter__() (as opposed to __iter__()) properly
 obeys to the new protocol? Why additionally duplicate __next__() and
 StopIteration?

 ISTM that all this really depends on is that __next__() returns an
 awaitable. Renaming the method doesn't help with that guarantee.


This is an astute observation. I think its flaw (if any) is the situation
where we want a single object to be both a regular iterator and an async
iterator (say when migrating code to the new world). The __next__ method
might want to return a result while __anext__ has to return an awaitable.
The solution to that would be to have __aiter__ return an instance of a
different class than __iter__, but that's not always convenient.

Thus, aware of the choice, I would still prefer a separate __anext__ method.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Gustavo Carneiro
On 1 May 2015 at 16:31, Guido van Rossum gu...@python.org wrote:

 On Fri, May 1, 2015 at 5:50 AM, Stefan Behnel stefan...@behnel.de wrote:

 Yury Selivanov schrieb am 30.04.2015 um 03:30:
  1. Terminology:
  - *native coroutine* term is used for async def functions.

 When I read native, I think of native (binary) code. So native
 coroutine sounds like it's implemented in some compiled low-level
 language. That might be the case (certainly in the CPython world), but
 it's
 not related to this PEP nor its set of examples.


  We should discuss how we will name new 'async def' coroutines in
  Python Documentation if the PEP is accepted.

 Well, it doesn't hurt to avoid obvious misleading terminology upfront.


 I think obvious[ly] misleading is too strong, nobody is trying to
 mislead anybody, we just have different associations with the same word.
 Given the feedback I'd call native coroutine suboptimal (even though I
 proposed it myself) and I am now in favor of using async function.


But what if you have async methods?  I know, a method is almost a function,
but still, sounds slightly confusing.

IMHO, these are really classical coroutines.  If gevent calls them
coroutines, I don't think asyncio has any less right to call them
coroutines.

You have to ask yourself this: a new programmer, when he sees mentions of
coroutines, how likely is he to understand what he is dealing with?  What
about async function?  The former is a well known concept, since decades
ago, while the latter is something he probably (at least me) never heard of
before.

For me, an async function is just as likely to be an API that is
asynchronous in the sense that it takes an extra callback parameter to be
called when the asynchronous work is done.

I think coroutine is the name of a concept, not a specific implementation.

Cheers,

-- 
Gustavo J. A. M. Carneiro
Gambit Research
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Guido van Rossum
On Fri, May 1, 2015 at 5:50 AM, Stefan Behnel stefan...@behnel.de wrote:

 Yury Selivanov schrieb am 30.04.2015 um 03:30:
  1. Terminology:
  - *native coroutine* term is used for async def functions.

 When I read native, I think of native (binary) code. So native
 coroutine sounds like it's implemented in some compiled low-level
 language. That might be the case (certainly in the CPython world), but it's
 not related to this PEP nor its set of examples.


  We should discuss how we will name new 'async def' coroutines in
  Python Documentation if the PEP is accepted.

 Well, it doesn't hurt to avoid obvious misleading terminology upfront.


I think obvious[ly] misleading is too strong, nobody is trying to mislead
anybody, we just have different associations with the same word. Given the
feedback I'd call native coroutine suboptimal (even though I proposed it
myself) and I am now in favor of using async function.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Stefan Behnel
Guido van Rossum schrieb am 01.05.2015 um 17:28:
 On Fri, May 1, 2015 at 5:39 AM, Stefan Behnel wrote:
 
 Yury Selivanov schrieb am 30.04.2015 um 03:30:
 Asynchronous Iterators and async for
 --

 An *asynchronous iterable* is able to call asynchronous code in its
 *iter* implementation, and *asynchronous iterator* can call
 asynchronous code in its *next* method.  To support asynchronous
 iteration:

 1. An object must implement an  ``__aiter__`` method returning an
*awaitable* resulting in an *asynchronous iterator object*.

 2. An *asynchronous iterator object* must implement an ``__anext__``
method returning an *awaitable*.

 3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration``
exception.

 What this section does not explain, AFAICT, nor the section on design
 considerations, is why the iterator protocol needs to be duplicated
 entirely. Can't we just assume (or even validate) that any 'regular'
 iterator returned from __aiter__() (as opposed to __iter__()) properly
 obeys to the new protocol? Why additionally duplicate __next__() and
 StopIteration?

 ISTM that all this really depends on is that __next__() returns an
 awaitable. Renaming the method doesn't help with that guarantee.
 
 
 This is an astute observation. I think its flaw (if any) is the situation
 where we want a single object to be both a regular iterator and an async
 iterator (say when migrating code to the new world). The __next__ method
 might want to return a result while __anext__ has to return an awaitable.
 The solution to that would be to have __aiter__ return an instance of a
 different class than __iter__, but that's not always convenient.

My personal gut feeling is that this case would best be handled by a
generic wrapper class. Both are well defined protocols, so I don't expect
people to change all of their classes and instead return a wrapped object
either from __iter__() or __aiter__(), depending on which they want to
optimise for, or which will eventually turn out to be easier to wrap.

But that's trying to predict the [Ff]uture, obviously. It just feels like
unnecessary complexity for now. And we already have a type slot for
__next__ (tp_iternext), but not for __anext__.

Stefan


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Guido van Rossum
On Fri, May 1, 2015 at 8:55 AM, Gustavo Carneiro gjcarne...@gmail.com
wrote:




 On 1 May 2015 at 16:31, Guido van Rossum gu...@python.org wrote:

 On Fri, May 1, 2015 at 5:50 AM, Stefan Behnel stefan...@behnel.de
 wrote:

 Yury Selivanov schrieb am 30.04.2015 um 03:30:
  1. Terminology:
  - *native coroutine* term is used for async def functions.

 When I read native, I think of native (binary) code. So native
 coroutine sounds like it's implemented in some compiled low-level
 language. That might be the case (certainly in the CPython world), but
 it's
 not related to this PEP nor its set of examples.


  We should discuss how we will name new 'async def' coroutines in
  Python Documentation if the PEP is accepted.

 Well, it doesn't hurt to avoid obvious misleading terminology upfront.


 I think obvious[ly] misleading is too strong, nobody is trying to
 mislead anybody, we just have different associations with the same word.
 Given the feedback I'd call native coroutine suboptimal (even though I
 proposed it myself) and I am now in favor of using async function.


 But what if you have async methods?  I know, a method is almost a
 function, but still, sounds slightly confusing.

 IMHO, these are really classical coroutines.  If gevent calls them
 coroutines, I don't think asyncio has any less right to call them
 coroutines.

 You have to ask yourself this: a new programmer, when he sees mentions of
 coroutines, how likely is he to understand what he is dealing with?  What
 about async function?  The former is a well known concept, since decades
 ago, while the latter is something he probably (at least me) never heard of
 before.

 For me, an async function is just as likely to be an API that is
 asynchronous in the sense that it takes an extra callback parameter to be
 called when the asynchronous work is done.

 I think coroutine is the name of a concept, not a specific implementation.

 Cheers,

  Cheers indeed! I agree that the *concept* is best called coroutine -- and
we have used this term ever since PEP 342. But when we're talking specifics
and trying to distinguish e.g. a function declared with 'async def' from a
regular function or from a regular generator function, using 'async
function' sounds right. And 'async method' if it's a method.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Yury Selivanov

On 2015-05-01 4:24 PM, Arnaud Delobelle wrote:

On 1 May 2015 at 20:24, Yury Selivanov yselivanov...@gmail.com wrote:

On 2015-05-01 3:19 PM, Ethan Furman wrote:

[...]

If we must have __aiter__, then we may as well also have __anext__;
besides
being more consistent, it also allows an object to be both a normol
iterator
and an asynch iterator.


And this is a good point too.

I'm not convinced that allowing an object to be both a normal and an
async iterator is a good thing.  It could be a recipe for confusion.




I doubt that it will be a popular thing.  But disallowing this
by merging two different protocols in one isn't a good idea
either.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Yury Selivanov

Let's say it this way: I want to know what I am looking at
when I browse through the code -- an asynchronous iterator,
or a normal iterator.  I want an explicit difference between
these protocols, because they are different.

Moreover, the below code is a perfectly valid, infinite
iterable:

class SomeIterable:
 def __iter__(self):
 return self
 async def __next__(self):
 return 'spam'

I'm strong -1 on this idea.

Yury

On 2015-05-01 3:03 PM, Stefan Behnel wrote:

Yury Selivanov schrieb am 01.05.2015 um 20:52:

I don't like the idea of combining __next__ and __anext__.
In this case explicit is better than implicit.  __next__
returning coroutines is a perfectly normal thing for a
normal 'for' loop (it wouldn't to anything with them),
whereas 'async for' will interpret that differently, and
will try to await those coroutines.

Sure, but the difference is that one would have called __aiter__() first
and the other __iter__(). Normally, either of the two would not exist, so
using the wrong loop on an object will just fail. However, after we passed
that barrier, we already know that the object that was returned is supposed
to obey to the expected protocol, so it doesn't matter whether we call
__next__() or name it __anext__(), except that the second requires us to
duplicate an existing protocol.

This has nothing to do with implicit vs. explicit.

Stefan


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Yury Selivanov

On 2015-05-01 3:19 PM, Ethan Furman wrote:

Sure, but the difference is that one would have called __aiter__() first
and the other __iter__(). Normally, either of the two would not exist, so
using the wrong loop on an object will just fail. However, after we passed
that barrier, we already know that the object that was returned is supposed
to obey to the expected protocol, so it doesn't matter whether we call
__next__() or name it __anext__(), except that the second requires us to
duplicate an existing protocol.

If we must have __aiter__, then we may as well also have __anext__; besides
being more consistent, it also allows an object to be both a normol iterator
and an asynch iterator.


And this is a good point too.

Thanks,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Stefan Behnel
Yury Selivanov schrieb am 01.05.2015 um 20:52:
 I don't like the idea of combining __next__ and __anext__.
 In this case explicit is better than implicit.  __next__
 returning coroutines is a perfectly normal thing for a
 normal 'for' loop (it wouldn't to anything with them),
 whereas 'async for' will interpret that differently, and
 will try to await those coroutines.

Sure, but the difference is that one would have called __aiter__() first
and the other __iter__(). Normally, either of the two would not exist, so
using the wrong loop on an object will just fail. However, after we passed
that barrier, we already know that the object that was returned is supposed
to obey to the expected protocol, so it doesn't matter whether we call
__next__() or name it __anext__(), except that the second requires us to
duplicate an existing protocol.

This has nothing to do with implicit vs. explicit.

Stefan


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Yury Selivanov

Stefan,

I don't like the idea of combining __next__ and __anext__.
In this case explicit is better than implicit.  __next__
returning coroutines is a perfectly normal thing for a
normal 'for' loop (it wouldn't to anything with them),
whereas 'async for' will interpret that differently, and
will try to await those coroutines.

Yury

On 2015-05-01 1:10 PM, Stefan Behnel wrote:

Guido van Rossum schrieb am 01.05.2015 um 17:28:

On Fri, May 1, 2015 at 5:39 AM, Stefan Behnel wrote:


Yury Selivanov schrieb am 30.04.2015 um 03:30:

Asynchronous Iterators and async for
--

An *asynchronous iterable* is able to call asynchronous code in its
*iter* implementation, and *asynchronous iterator* can call
asynchronous code in its *next* method.  To support asynchronous
iteration:

1. An object must implement an  ``__aiter__`` method returning an
*awaitable* resulting in an *asynchronous iterator object*.

2. An *asynchronous iterator object* must implement an ``__anext__``
method returning an *awaitable*.

3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration``
exception.

What this section does not explain, AFAICT, nor the section on design
considerations, is why the iterator protocol needs to be duplicated
entirely. Can't we just assume (or even validate) that any 'regular'
iterator returned from __aiter__() (as opposed to __iter__()) properly
obeys to the new protocol? Why additionally duplicate __next__() and
StopIteration?

ISTM that all this really depends on is that __next__() returns an
awaitable. Renaming the method doesn't help with that guarantee.


This is an astute observation. I think its flaw (if any) is the situation
where we want a single object to be both a regular iterator and an async
iterator (say when migrating code to the new world). The __next__ method
might want to return a result while __anext__ has to return an awaitable.
The solution to that would be to have __aiter__ return an instance of a
different class than __iter__, but that's not always convenient.

My personal gut feeling is that this case would best be handled by a
generic wrapper class. Both are well defined protocols, so I don't expect
people to change all of their classes and instead return a wrapped object
either from __iter__() or __aiter__(), depending on which they want to
optimise for, or which will eventually turn out to be easier to wrap.

But that's trying to predict the [Ff]uture, obviously. It just feels like
unnecessary complexity for now. And we already have a type slot for
__next__ (tp_iternext), but not for __anext__.

Stefan


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Ethan Furman
On 05/01, Stefan Behnel wrote:
 Yury Selivanov schrieb am 01.05.2015 um 20:52:

 I don't like the idea of combining __next__ and __anext__.
 In this case explicit is better than implicit.  __next__
 returning coroutines is a perfectly normal thing for a
 normal 'for' loop (it wouldn't to anything with them),
 whereas 'async for' will interpret that differently, and
 will try to await those coroutines.
 
 Sure, but the difference is that one would have called __aiter__() first
 and the other __iter__(). Normally, either of the two would not exist, so
 using the wrong loop on an object will just fail. However, after we passed
 that barrier, we already know that the object that was returned is supposed
 to obey to the expected protocol, so it doesn't matter whether we call
 __next__() or name it __anext__(), except that the second requires us to
 duplicate an existing protocol.

If we must have __aiter__, then we may as well also have __anext__; besides
being more consistent, it also allows an object to be both a normol iterator
and an asynch iterator.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Stefan Behnel
Yury Selivanov schrieb am 30.04.2015 um 03:30:
 Asynchronous Iterators and async for
 --
 
 An *asynchronous iterable* is able to call asynchronous code in its
 *iter* implementation, and *asynchronous iterator* can call
 asynchronous code in its *next* method.  To support asynchronous
 iteration:
 
 1. An object must implement an  ``__aiter__`` method returning an
*awaitable* resulting in an *asynchronous iterator object*.
 
 2. An *asynchronous iterator object* must implement an ``__anext__``
method returning an *awaitable*.
 
 3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration``
exception.

What this section does not explain, AFAICT, nor the section on design
considerations, is why the iterator protocol needs to be duplicated
entirely. Can't we just assume (or even validate) that any 'regular'
iterator returned from __aiter__() (as opposed to __iter__()) properly
obeys to the new protocol? Why additionally duplicate __next__() and
StopIteration?

ISTM that all this really depends on is that __next__() returns an
awaitable. Renaming the method doesn't help with that guarantee.

Stefan


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-01 Thread Stefan Behnel
Yury Selivanov schrieb am 30.04.2015 um 03:30:
 1. Terminology:
 - *native coroutine* term is used for async def functions.

When I read native, I think of native (binary) code. So native
coroutine sounds like it's implemented in some compiled low-level
language. That might be the case (certainly in the CPython world), but it's
not related to this PEP nor its set of examples.


 We should discuss how we will name new 'async def' coroutines in
 Python Documentation if the PEP is accepted.

Well, it doesn't hurt to avoid obvious misleading terminology upfront.

Stefan


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-04-30 Thread Greg Ewing

Yury Selivanov wrote:


3. CO_NATIVE_COROUTINE flag. This enables us to disable
__iter__ and __next__ on native coroutines while maintaining
full backwards compatibility.


I don't think you can honestly claim full backwards
compatibility as long as there are some combinations
of old-style and new-style code that won't work
together. You seem to be using your own personal
definition of full here.

--
Greg

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-04-30 Thread Guido van Rossum
On Thu, Apr 30, 2015 at 4:24 PM, Greg Ewing greg.ew...@canterbury.ac.nz
wrote:

 Yury Selivanov wrote:

 Well, using next() and iter() on coroutines in asyncio
 code is something esoteric.  I can't even imagine
 why you would want to do that.


 I'm talking about the fact that existing generator-
 based coroutines that aren't decorated with
 @coroutine won't be able to call new ones that use
 async def.

 This means that converting one body of code to the
 new style can force changes in other code that
 interacts with it.

 Maybe this is not considered a problem, but as long
 as it's true, I don't think it's accurate to claim
 full backwards compatibility.


Greg, you seem to have an odd notion of full backwards compatibility. The
term means that old code won't break. It doesn't imply that old and new
code can always seamlessly interact (that would be an impossibly high bar
for almost any change).

That said, interoperability between old code and new code is an area of
interest. But if the only thing that's standing between old code and new
code is the @coroutine decorator, things are looking pretty good -- that
decorator is already strongly required for coroutines intended for use with
the asyncio package, and older versions of the asyncio package also define
that decorator, so if there's old code out there that needs to be able to
call the new coroutines (by whatever name, e.g. async functions :-), adding
the @coroutine decorator to the old code doesn't look like too much of a
burden.

I assume there might be code out there that uses yield-from-based
coroutines but does not use the asyncio package, but I doubt there is much
code like that (I haven't seen much mention of yield-from outside its use
in asyncio). So I think the interop problem is mostly limited to
asyncio-using code that plays loose with the @coroutine decorator
requirement and now wants to work with the new async functions. That's easy
enough to address.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-04-30 Thread Yury Selivanov

On 2015-04-30 7:24 PM, Greg Ewing wrote:

Yury Selivanov wrote:

Well, using next() and iter() on coroutines in asyncio
code is something esoteric.  I can't even imagine
why you would want to do that.


I'm talking about the fact that existing generator-
based coroutines that aren't decorated with
@coroutine won't be able to call new ones that use
async def.



Ah, alright.

You quoted this:

3. CO_NATIVE_COROUTINE flag. This enables us to
disable __iter__ and __next__ on native coroutines
while maintaining full backwards compatibility.

I wrote full backwards compatibility for that
particular point #3 -- existing @asyncio.coroutines
will have __iter__ and __next__ working just fine.

Sorry if this was misleading.


This means that converting one body of code to the
new style can force changes in other code that
interacts with it.

Maybe this is not considered a problem, but as long
as it's true, I don't think it's accurate to claim
full backwards compatibility.



I covered this in point #4.  I also touched this in
https://www.python.org/dev/peps/pep-0492/#migration-strategy


I'm still waiting for feedback on this from Guido.  If
he decides to go with RuntimeWarnings, then it's 100%
backwards compatible.  If we keep TypeErrors --
then *existing code will work on 3.5*, but something
*might* break during adopting new syntax.  I'll update
the Backwards Compatibility section.


Thanks,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-04-30 Thread Greg Ewing

Yury Selivanov wrote:

Well, using next() and iter() on coroutines in asyncio
code is something esoteric.  I can't even imagine
why you would want to do that.


I'm talking about the fact that existing generator-
based coroutines that aren't decorated with
@coroutine won't be able to call new ones that use
async def.

This means that converting one body of code to the
new style can force changes in other code that
interacts with it.

Maybe this is not considered a problem, but as long
as it's true, I don't think it's accurate to claim
full backwards compatibility.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-04-30 Thread Yury Selivanov

On 2015-04-30 5:16 AM, Greg Ewing wrote:

Yury Selivanov wrote:


3. CO_NATIVE_COROUTINE flag. This enables us to disable
__iter__ and __next__ on native coroutines while maintaining
full backwards compatibility.


I don't think you can honestly claim full backwards
compatibility as long as there are some combinations
of old-style and new-style code that won't work
together. You seem to be using your own personal
definition of full here.



Well, using next() and iter() on coroutines in asyncio
code is something esoteric.  I can't even imagine
why you would want to do that.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-04-29 Thread Yury Selivanov

One more thing to discuss:

7. StopAsyncIteration vs AsyncStopIteration.

I don't have a strong opinion on this, I prefer
the former because it reads better.

There was no consensus on which one we should use.

Thanks,

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com