Re: [Python-ideas] gevent-like Coroutines in Python

[Mark E. Haase]

Python's coroutines are designed to make suspension points visible, which
enhances "local reasoning" about code. This concept has been written up
very well over here: https://glyph.twistedmatrix.com/2014/02/unyielding.html

On Tue, Oct 30, 2018 at 8:37 AM Ron Reiter  wrote:

> You are right that they are different, I was actually assuming that
> developers by default don't try to parallelize and would rather go ahead
> and write code to yield one function at a time, which is fine. The need to
> separate "await" from the invocation is something which is rarely used. Not
> sure what you mean about "threading" as it is still more efficient and
> lightweight to parallelize workers on an event loop rather than using
> blocking threads.
>
> As I said - I don't think we should downgrade Python's current ability to
> do so, my suggestion is to create something like the "codef" proposal,
> which will also await on every function invocation - for readability.
>
> - Ron
>
>
> [image: Facebook]  [image: Twitter]
>  [image: LinkedIn]
> 
>
>
> On Tue, Oct 30, 2018 at 12:41 PM Chris Angelico  wrote:
>
>> On Tue, Oct 30, 2018 at 6:01 PM Ron Reiter  wrote:
>> >
>> > ... most developers would always mean they prefer to do this:
>> >
>> > result = [await fun(x) for fun in funcs]
>> >
>> > versus:
>> >
>> > result = [fun(x) for fun in funcs]
>> > await asyncio.gather(*result)
>> >
>> > Moreso, having it become the default makes statements like this:
>> >
>> > result = [await fun(x) for fun in funcs if await smth]
>> >
>> > Look like this:
>> >
>> > result = [fun(x) for fun in funcs if smth]
>> >
>> > Therefore, my suggestion is to create a new "async" definition which
>> basically turns every function invocation into an "await" if a generator is
>> returned.
>> >
>>
>> I'm not sure what you're driving at here. From your first example, I
>> gather that (pun intended) you're expecting the 'result' list to
>> contain all the results from the different function calls, running
>> them all in parallel; but your second example and described suggestion
>> seem to imply that the waitings would continue to be sequential.
>>
>> Unless you're asking for straight-up magic ("do everything in parallel
>> unless they need to be serialized"), there still needs to be a clear
>> way to differentiate between "wait for this right now and give me a
>> result before this function continues" and "gather all these jobs
>> together, get me the results, and then move on once you have them
>> all". It might perhaps be nice to have an easier/more obvious syntax
>> for gather(), but it definitely needs to have some form of spelling.
>>
>> If you're not asking for them to be run in parallel, you're asking for
>> an implicit way for a function call to block its caller, and for the
>> calling function to act sequentially. Python already has that - it's
>> called threading :)
>>
>> ChrisA
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] gevent-like Coroutines in Python

[Chris Angelico]

On Tue, Oct 30, 2018 at 11:36 PM Ron Reiter  wrote:
>
> You are right that they are different, I was actually assuming that 
> developers by default don't try to parallelize and would rather go ahead and 
> write code to yield one function at a time, which is fine. The need to 
> separate "await" from the invocation is something which is rarely used. Not 
> sure what you mean about "threading" as it is still more efficient and 
> lightweight to parallelize workers on an event loop rather than using 
> blocking threads.
>

Okay, so it's actually nothing to do with asyncio.gather(). Sure.

So what you're looking for is JUST the removal of the "await"
keywords. As Greg already said, Guido considers the explicit await
markers as a feature, not a bug; these are the exact points where an
intrathread context switch can occur.

As to the efficiency of parallelizing on an event loop rather than
using threads, that's a tradeoff; threads aren't going anywhere just
because asyncio is here. When you want the extreme simplicity of "just
do this stuff, okay?", the easiest way to get it is to just use
threads, and pay a bit of overhead. You'll often find that the
overhead isn't actually all that significant until you get to extremes
of throughput - most Python apps are not trying to run tens of
thousands of concurrent TCP sockets, for instance.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] gevent-like Coroutines in Python

[Ron Reiter]

You are right that they are different, I was actually assuming that
developers by default don't try to parallelize and would rather go ahead
and write code to yield one function at a time, which is fine. The need to
separate "await" from the invocation is something which is rarely used. Not
sure what you mean about "threading" as it is still more efficient and
lightweight to parallelize workers on an event loop rather than using
blocking threads.

As I said - I don't think we should downgrade Python's current ability to
do so, my suggestion is to create something like the "codef" proposal,
which will also await on every function invocation - for readability.

- Ron


[image: Facebook]  [image: Twitter]
 [image: LinkedIn]



On Tue, Oct 30, 2018 at 12:41 PM Chris Angelico  wrote:

> On Tue, Oct 30, 2018 at 6:01 PM Ron Reiter  wrote:
> >
> > ... most developers would always mean they prefer to do this:
> >
> > result = [await fun(x) for fun in funcs]
> >
> > versus:
> >
> > result = [fun(x) for fun in funcs]
> > await asyncio.gather(*result)
> >
> > Moreso, having it become the default makes statements like this:
> >
> > result = [await fun(x) for fun in funcs if await smth]
> >
> > Look like this:
> >
> > result = [fun(x) for fun in funcs if smth]
> >
> > Therefore, my suggestion is to create a new "async" definition which
> basically turns every function invocation into an "await" if a generator is
> returned.
> >
>
> I'm not sure what you're driving at here. From your first example, I
> gather that (pun intended) you're expecting the 'result' list to
> contain all the results from the different function calls, running
> them all in parallel; but your second example and described suggestion
> seem to imply that the waitings would continue to be sequential.
>
> Unless you're asking for straight-up magic ("do everything in parallel
> unless they need to be serialized"), there still needs to be a clear
> way to differentiate between "wait for this right now and give me a
> result before this function continues" and "gather all these jobs
> together, get me the results, and then move on once you have them
> all". It might perhaps be nice to have an easier/more obvious syntax
> for gather(), but it definitely needs to have some form of spelling.
>
> If you're not asking for them to be run in parallel, you're asking for
> an implicit way for a function call to block its caller, and for the
> calling function to act sequentially. Python already has that - it's
> called threading :)
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] gevent-like Coroutines in Python

[Chris Angelico]

On Tue, Oct 30, 2018 at 6:01 PM Ron Reiter  wrote:
>
> ... most developers would always mean they prefer to do this:
>
> result = [await fun(x) for fun in funcs]
>
> versus:
>
> result = [fun(x) for fun in funcs]
> await asyncio.gather(*result)
>
> Moreso, having it become the default makes statements like this:
>
> result = [await fun(x) for fun in funcs if await smth]
>
> Look like this:
>
> result = [fun(x) for fun in funcs if smth]
>
> Therefore, my suggestion is to create a new "async" definition which 
> basically turns every function invocation into an "await" if a generator is 
> returned.
>

I'm not sure what you're driving at here. From your first example, I
gather that (pun intended) you're expecting the 'result' list to
contain all the results from the different function calls, running
them all in parallel; but your second example and described suggestion
seem to imply that the waitings would continue to be sequential.

Unless you're asking for straight-up magic ("do everything in parallel
unless they need to be serialized"), there still needs to be a clear
way to differentiate between "wait for this right now and give me a
result before this function continues" and "gather all these jobs
together, get me the results, and then move on once you have them
all". It might perhaps be nice to have an easier/more obvious syntax
for gather(), but it definitely needs to have some form of spelling.

If you're not asking for them to be run in parallel, you're asking for
an implicit way for a function call to block its caller, and for the
calling function to act sequentially. Python already has that - it's
called threading :)

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] gevent-like Coroutines in Python

[Greg Ewing]

Ron Reiter wrote:
I feel like having the await syntax trigger by default on any awaitable 
invocation in a coroutine context makes much more sense.


Guido seems to regard the requirement to use 'await' as a
feature, not a bug. He says he likes to be able to see where
all the potential suspension points are.

Therefore, my suggestion is to create a new "async" definition which 
basically turns every function invocation into an "await" if a generator 
is returned. Instead of "async def" I propose the alternative "coroutine 
def" syntax. However, a better solution may be to imply the word "async" 
in every function definition given some sort of trigger (however I 
assume that this won't be the preferable approach as it is not something 
that can be implemented at the parsing level).


I don't think it's feasible to automatically infer both 'async'
*and* 'await', even at run time. An async function needs to be
treated differently from the moment it starts running, so there
must be something that statically identifies it as such.

Anyway, if you want to pursue these ideas further, you should
take a look at PEP 3152, which was my attempt at a nicer syntax
for generator-based coroutines, before async/await came along.
I think it would have been better in some ways, but it was
rejected.

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] gevent-like Coroutines in Python

[Ron Reiter]

One of Golang's advantages is that goroutines act like gevent's coroutines
instead of relying on an async/await syntax. In my opinion, it makes Golang
code much more readable.

I feel like having the await syntax trigger by default on any awaitable
invocation in a coroutine context makes much more sense. I consider
async/await syntax to be too complicated for the average developer since it
opens up too many abilities whereas most developers would always mean they
prefer to do this:

result = [await fun(x) for fun in funcs]

versus:

result = [fun(x) for fun in funcs]
await asyncio.gather(*result)

Moreso, having it become the default makes statements like this:

result = [await fun(x) for fun in funcs if await smth]

Look like this:

result = [fun(x) for fun in funcs if smth]

Therefore, my suggestion is to create a new "async" definition which
basically turns every function invocation into an "await" if a generator is
returned. Instead of "async def" I propose the alternative "coroutine def"
syntax. However, a better solution may be to imply the word "async" in
every function definition given some sort of trigger (however I assume that
this won't be the preferable approach as it is not something that can be
implemented at the parsing level).

For me, I believe that Python should aspire to be both concise and
straightforward, which means no boilerplate code just because it's more
"correct" but rather assume there's some logical default behavior going on
in the back - and to me this logical behavior is that every invocation can
trigger an await and go back to the main loop. Our assumption that a
function invocation has to tie back to instant execution unless an "await"
statement has been placed should be challenged in favor of readability,
conciseness, and to always aim to appeal to novice developers (which I
believe is the reason Python is on such a rise these days).

I do have to admit I have not thought through what is the best syntax or
the implications of this because I would like to see what is the general
opinion on this idea first.

- Ron
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/