Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

I need to cut this debate short (too much to do already) but I'd like to
press that I wish async/await to be available for general tinkering (like
writing elegant parsers), not just for full fledged event loops.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 27 November 2017 at 06:29, Nathaniel Smith  wrote:
> - In async/await, it's not obvious how to write leaf functions:
> 'await' is equivalent to 'yield from', but there's no equivalent to
> 'yield'. You have to jump through some hoops by writing a class with a
> custom __await__ method or using @types.coroutine. Of course it's
> doable, and it's no big deal if you're writing a proper async library,
> but it's awkward for quick ad hoc usage.
>
> For a concrete example of 'ad hoc coroutines' where I think 'yield
> from' is appropriate, here's wsproto's old 'yield from'-based
> incremental websocket protocol parser:
>
> 
> https://github.com/python-hyper/wsproto/blob/4b7db502cc0568ab2354798552148dadd563a4e3/wsproto/frame_protocol.py#L142

sys.set_coroutine_wrapper itself is another case where you genuinely
*can't* rely on async/await in the wrapper implementation.

The current example shown at
https://docs.python.org/3/library/sys.html#sys.set_coroutine_wrapper
is of a case that will *fail*, since it would otherwise result in
infinite recursion when the coroutine wrapper attempts to call the
coroutine wrapper.

Cheers,
Nick.

P.S. Making that point reminded me that I still haven't got around to
updating those docs to also include examples of how to do it *right*:
https://bugs.python.org/issue30578

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Yury Selivanov]

On Sun, Nov 26, 2017 at 6:51 PM, Guido van Rossum  wrote:
> On Sun, Nov 26, 2017 at 12:29 PM, Nathaniel Smith  wrote:
[..]
>> - async/await has associated thread-global state like
>> sys.set_coroutine_wrapper and sys.set_asyncgen_hooks. Generally async
>> libraries assume that they own these, and arbitrarily weird things may
>> happen if you have multiple async/await coroutine runners in same
>> thread with no coordination between them.
>
>
> The existence of these is indeed a bit unfortunate for this use case. I'm
> CC'ing Yury to ask him if he can think of a different way to deal with the
> problems that these are supposed to solve. For each, the reason they exist
> is itself an edge case -- debugging for the former, finalization for the
> latter. A better solution for these problem may also be important for
> situations where multiple event loops exist (in the same thread, e.g.
> running alternately). Maybe a context manager could be used to manage this
> state better?

Yeah, both of them are for solving edge cases:

- sys.set_coroutine_wrapper() is a debug API: asyncio uses it to
slightly enhance the warning about non-awaited coroutines by showing
where they were *created*.  Capturing traceback when we create every
coroutine is expensive, hence we only do that in asyncio debug mode.

- sys.set_asyncgen_hooks() is more important, we use it to let event
loops finalize partially iterated and then abandoned asynchronous
generators.

The rule of thumb is to get the previous coro-wrapper/asyncgen-hook,
set your own (both are thread specific), and after you're done --
restore the saved old versions back.  This should work fine in all use
cases (even a trio event loop nested in an asyncio event loop).  But
nested coroutine loops is a sign of bad design, the nested loop will
completely block the execution of the outer event loop, so situations
like that should be avoided.  Not to mention that debugging becomes
much harder when you have complex nested coroutine runners.

If someone wants to run some async/await code without an event loop
for educational purposes or to implement some pattern, they likely
don't even need to use any of the above APIs.

For me the status quo is OK: asyncio/twisted/curio/trio use these APIs
internally, and for them it shouldn't be a problem to use a couple
low-level functions from sys.

That said I have a couple ideas:

- We can simplify these low-level APIs by combining
set_coroutine_wrapper() and set_asyncgen_hooks() into one function --
set_async_hooks(coroutine_wrapper=, firstiter=, etc). It could return
a context-manager, so the following would be possible:

   with sys.set_async_hooks(...):
   # run async code or an event loop

- Another option is to design an API that would let you to stack your
coro-wrapper/asyncgen-hooks, so that many coroutine runners can
control coroutines/generators simultaneously.  This sounds very
complex to me though, and I haven't seen any compelling real-world use
case that would require this kind of design.

sys.set_coroutine_wrapper() is marked as an experimental debug API;
sys.set_asyncgen_hooks() is still provisional.  So we can in theory
change/improve both of them.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Sun, Nov 26, 2017 at 12:29 PM, Nathaniel Smith  wrote:

> On Sat, Nov 25, 2017 at 3:37 PM, Guido van Rossum 
> wrote:
> > On Sat, Nov 25, 2017 at 1:05 PM, David Mertz  wrote:
> >>
> >> FWIW, on a side point. I use 'yield' and 'yield from' ALL THE TIME in
> real
> >> code. Probably 80% of those would be fine with yield statements, but a
> >> significant fraction use `gen.send()`.
> >>
> >> On the other hand, I have yet once to use 'await', or 'async' outside of
> >> pedagogical contexts. There are a whole lot of generators, including
> ones
> >> utilizing state injection, that are useful without the scaffolding of an
> >> event loop, in synchronous code.
> >
> >
> > Maybe you didn't realize async/await don't need an event loop? Driving an
> > async/await-based coroutine is just as simple as driving a
> yield-from-based
> > one (`await` does exactly the same thing as `yield from`).
>
> Technically anything you can write with yield/yield from could also be
> written using async/await and vice-versa, but I think it's actually
> nice to have both in the language.
>

Perhaps. You seem somewhat biased towards the devil you know, but you also
bring up some good points.


> The distinction I'd make is that yield/yield from is what you should
> use for ad hoc coroutines where the person writing the code that has
> 'yield from's in it is expected to understand the details of the
> coroutine runner, while async/await is what you should use when the
> coroutine running is handled by a library like asyncio, and the person
> writing code with 'await's in it is expected to treat coroutine stuff
> as an opaque implementation detail. (NB I'm using "coroutine" in the
> CS sense here, where generators and async functions are both
> "coroutines".)
>
> I think of this as being sort of half-way between a style guideline
> and a technical guideline. It's like the guideline that lists should
> be homogenously-typed and variable length, while tuples are
> heterogenously-typed and fixed length: there's nothing in the language
> that outright *enforces* this, but it's a helpful convention *and*
> things tend to work better if you go along with it.
>

Hm. That would disappoint me. We carefully tried to design async/await to
*not* require an event loop. (I'll get to the global state below.)


> Here are some technical issues you'll run into if you try to use
> async/await for ad hoc coroutines:
>
> - If you don't iterate an async function, you get a "coroutine never
> awaited" warning. This may or may not be what you want.
>

It should indicate a bug, and the equivalent bug is silent when you're
using yield-from, so I see this as a positive. If you find yourself
designing an API where abandoning an async function is a valid action you
should probably think twice.


> - async/await has associated thread-global state like
> sys.set_coroutine_wrapper and sys.set_asyncgen_hooks. Generally async
> libraries assume that they own these, and arbitrarily weird things may
> happen if you have multiple async/await coroutine runners in same
> thread with no coordination between them.
>

The existence of these is indeed a bit unfortunate for this use case. I'm
CC'ing Yury to ask him if he can think of a different way to deal with the
problems that these are supposed to solve. For each, the reason they exist
is itself an edge case -- debugging for the former, finalization for the
latter. A better solution for these problem may also be important for
situations where multiple event loops exist (in the same thread, e.g.
running alternately). Maybe a context manager could be used to manage this
state better?


> - In async/await, it's not obvious how to write leaf functions:
> 'await' is equivalent to 'yield from', but there's no equivalent to
> 'yield'. You have to jump through some hoops by writing a class with a
> custom __await__ method or using @types.coroutine. Of course it's
> doable, and it's no big deal if you're writing a proper async library,
> but it's awkward for quick ad hoc usage.
>

Ah, yes, you need the equivalent of a Future. Maybe we should have a simple
one in the stdlib that's not tied to asyncio.


> For a concrete example of 'ad hoc coroutines' where I think 'yield
> from' is appropriate, here's wsproto's old 'yield from'-based
> incremental websocket protocol parser:
>
> https://github.com/python-hyper/wsproto/blob/
> 4b7db502cc0568ab2354798552148dadd563a4e3/wsproto/frame_protocol.py#L142
>

Ah yes, these kinds of parsers are interesting use cases for coroutines.
There are many more potential use cases than protocol parsers -- e.g. the
Python REPL could really use one if you want to replicate it completely in
pure Python.


> The flow here is: received_frames is the public API: it gives you an
> iterator over all completed frames. When it stops you're expected to
> add more data to the buffer and then call it again. Internally,
> received_frames acts as a coroutine runner 

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nathaniel Smith]

On Sat, Nov 25, 2017 at 3:37 PM, Guido van Rossum  wrote:
> On Sat, Nov 25, 2017 at 1:05 PM, David Mertz  wrote:
>>
>> FWIW, on a side point. I use 'yield' and 'yield from' ALL THE TIME in real
>> code. Probably 80% of those would be fine with yield statements, but a
>> significant fraction use `gen.send()`.
>>
>> On the other hand, I have yet once to use 'await', or 'async' outside of
>> pedagogical contexts. There are a whole lot of generators, including ones
>> utilizing state injection, that are useful without the scaffolding of an
>> event loop, in synchronous code.
>
>
> Maybe you didn't realize async/await don't need an event loop? Driving an
> async/await-based coroutine is just as simple as driving a yield-from-based
> one (`await` does exactly the same thing as `yield from`).

Technically anything you can write with yield/yield from could also be
written using async/await and vice-versa, but I think it's actually
nice to have both in the language.

The distinction I'd make is that yield/yield from is what you should
use for ad hoc coroutines where the person writing the code that has
'yield from's in it is expected to understand the details of the
coroutine runner, while async/await is what you should use when the
coroutine running is handled by a library like asyncio, and the person
writing code with 'await's in it is expected to treat coroutine stuff
as an opaque implementation detail. (NB I'm using "coroutine" in the
CS sense here, where generators and async functions are both
"coroutines".)

I think of this as being sort of half-way between a style guideline
and a technical guideline. It's like the guideline that lists should
be homogenously-typed and variable length, while tuples are
heterogenously-typed and fixed length: there's nothing in the language
that outright *enforces* this, but it's a helpful convention *and*
things tend to work better if you go along with it.

Here are some technical issues you'll run into if you try to use
async/await for ad hoc coroutines:

- If you don't iterate an async function, you get a "coroutine never
awaited" warning. This may or may not be what you want.

- async/await has associated thread-global state like
sys.set_coroutine_wrapper and sys.set_asyncgen_hooks. Generally async
libraries assume that they own these, and arbitrarily weird things may
happen if you have multiple async/await coroutine runners in same
thread with no coordination between them.

- In async/await, it's not obvious how to write leaf functions:
'await' is equivalent to 'yield from', but there's no equivalent to
'yield'. You have to jump through some hoops by writing a class with a
custom __await__ method or using @types.coroutine. Of course it's
doable, and it's no big deal if you're writing a proper async library,
but it's awkward for quick ad hoc usage.

For a concrete example of 'ad hoc coroutines' where I think 'yield
from' is appropriate, here's wsproto's old 'yield from'-based
incremental websocket protocol parser:


https://github.com/python-hyper/wsproto/blob/4b7db502cc0568ab2354798552148dadd563a4e3/wsproto/frame_protocol.py#L142

The flow here is: received_frames is the public API: it gives you an
iterator over all completed frames. When it stops you're expected to
add more data to the buffer and then call it again. Internally,
received_frames acts as a coroutine runner for parse_more_gen, which
is the main parser that calls various helper methods to parse
different parts of the websocket frame. These calls eventually bottom
out in _consume_exactly or _consume_at_most, which use 'yield' to
"block" until enough data is available in the internal buffer.
Basically this is the classic trick of using coroutines to write an
incremental state machine parser as ordinary-looking code where the
state is encoded in local variables on the stack.

Using coroutines here isn't just a cute trick; I'm pretty confident
that there is absolutely no other way to write a readable incremental
websocket parser in Python. This is the 3rd rewrite of wsproto's
parser, and I think I've read the code for all the other Python
libraries that do this too. The websocket framing format is branchy
enough that trying to write out the state machine explicitly will
absolutely tie you in knots. (Of course we then rewrote wsproto's
parser a 4th time for py2 compatibility; the current version's not
*terrible* but the 'yield from' version was simpler and more
maintainable.)

For wsproto's use case, I think using 'await' would be noticeably
worse than 'yield from'. It'd make the code more opaque to readers
(people know generators but no-one shows up already knowing what
@types.coroutine does), the "coroutine never awaited" warnings would
be obnoxious (it's totally fine to instantiate a parser and then throw
it away without using it!), and the global state issues would make us
very nervous (wsproto is absolutely designed to be used alongside a
library like asyncio or trio). 

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Sat, Nov 25, 2017 at 4:57 PM, Nick Coghlan  wrote:

> On 26 November 2017 at 02:59, Guido van Rossum  wrote:
> >
> > I'd be happy to stop with the conclusion that we're going to rip out some
> > confusing syntax rather than trying to generate code for it -- IMO we've
> > proved to ourselves that this stuff is too complicated to be useful.
>
> I'll also note that even if we go through a period of deprecating and
> then prohibiting the syntax entirely, we'll still have the option of
> bringing support for "yield" in comprehensions back later with
> deliberately designed semantics (as happened for "await" in
> https://www.python.org/dev/peps/pep-0530/#await-in-comprehensions), as
> opposed to the accident-of-implementation semantics they have now.
>
> It may also turn out that as more asynchronous code is able to switch
> to being 3.6+ only, allowing "await" and prohibiting "yield" will
> prove to be sufficient for all practical purposes (as even the "yield
> from" based spelling is Python-3-only, so it's only code that still
> has to support 3.3, 3.4, 3.5, without needing to support 2.7, that
> could use "yield from + yield" comprehensions, but wouldn't have the
> option of just switching to async+await instead).
>

And I'll also note that some style guides recommend using comprehensions
only for simple cases. Example:
https://google.github.io/styleguide/pyguide.html#List_Comprehensions --
expand by clicking on the arrow; the "Con" comment is "Complicated list
comprehensions or generator expressions can be hard to read."

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ethan Furman]

On 11/25/2017 04:20 PM, David Mertz wrote:

On Sat, Nov 25, 2017 at 3:37 PM, Guido van Rossum wrote:



Maybe you didn't realize async/await don't need an event loop? Driving an 
async/await-based coroutine is just as
simple as driving a yield-from-based one (`await` does exactly the same thing 
as `yield from`).


I realize I *can*, but it seems far from straightforward.  I guess this is 
really a python-list question or something,
but what is the async/await spelling of something toy like:

In [1]: def fib():
...: a, b = 1, 1
...: while True:
...: yield a
...: a, b = b, a+b
...:

In [2]: from itertools import takewhile

In [3]: list(takewhile(lambda x: x<200, fib()))
Out[3]: [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144]



Maybe the rest of the discussion should be about deprecation vs. SyntaxError in 
Python 3.7.


I vote SyntaxError, of course. :-)


Given the recent thread about the difficulty of noticing DeprecationWarnings, I 
also vote SyntaxError.

On the other hand, if we have a change in 3.7 about the visibility of DeprecationWarnings, this would make an excellent 
test case.


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 26 November 2017 at 02:59, Guido van Rossum  wrote:
>
> I'd be happy to stop with the conclusion that we're going to rip out some
> confusing syntax rather than trying to generate code for it -- IMO we've
> proved to ourselves that this stuff is too complicated to be useful.

I'll also note that even if we go through a period of deprecating and
then prohibiting the syntax entirely, we'll still have the option of
bringing support for "yield" in comprehensions back later with
deliberately designed semantics (as happened for "await" in
https://www.python.org/dev/peps/pep-0530/#await-in-comprehensions), as
opposed to the accident-of-implementation semantics they have now.

It may also turn out that as more asynchronous code is able to switch
to being 3.6+ only, allowing "await" and prohibiting "yield" will
prove to be sufficient for all practical purposes (as even the "yield
from" based spelling is Python-3-only, so it's only code that still
has to support 3.3, 3.4, 3.5, without needing to support 2.7, that
could use "yield from + yield" comprehensions, but wouldn't have the
option of just switching to async+await instead).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[David Mertz]

On Sat, Nov 25, 2017 at 3:37 PM, Guido van Rossum  wrote:

> Maybe you didn't realize async/await don't need an event loop? Driving an
> async/await-based coroutine is just as simple as driving a yield-from-based
> one (`await` does exactly the same thing as `yield from`).
>

I realize I *can*, but it seems far from straightforward.  I guess this is
really a python-list question or something, but what is the async/await
spelling of something toy like:

In [1]: def fib():
   ...: a, b = 1, 1
   ...: while True:
   ...: yield a
   ...: a, b = b, a+b
   ...:

In [2]: from itertools import takewhile

In [3]: list(takewhile(lambda x: x<200, fib()))
Out[3]: [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144]



> Maybe the rest of the discussion should be about deprecation vs.
> SyntaxError in Python 3.7.
>

I vote SyntaxError, of course. :-)

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Sat, Nov 25, 2017 at 3:24 PM, Greg Ewing 
wrote:

> Nick Coghlan wrote:
>
> def example():
>> comp1 = yield from [(yield x) for x in ('1st', '2nd')]
>> comp2 = yield from [(yield x) for x in ('3rd', '4th')]
>> return comp1, comp2
>>
>
> If the implicit "yield from" idea seems too magical, then the other
>> direction we could go is to make the immediate "yield from" mandatory,
>>
>
> Whether it's "too magical" or not depends on your stance with
> regard to the implicit function scope of a comprehension. There
> seem to be two irreconcilable schools of thought on that:
>
> (1) It's an implementation detail that happens to be used to
> stop the loop variable from leaking. Most of the time you can
> ignore it and use the nested-loop-expansion mental model.
> Most people seem to think this way in practice.
>
> (2) It's an important part of the semantics of comprehensions
> that needs to be taken into account at all times. This seems
> to be Guido's position.
>
> If you're in school (1), then the current behaviour of yield
> in a comprehension is the thing that's weird and magical.
> Moreover, it's bad magic, because it does something you
> almost certainly don't want. Adding an implicit yield-from
> would make things *less* magical and more understandable.
>
> However, it will break things for people in school (2), who
> are aware of the arcane details and know enough to put the
> yield-from in themselves when needed.
>
> Just because it breaks their code doesn't necessarily mean
> they will find it more magical, though. Generators are
> already sufficiently magical that they're probably smart
> enough to figure out what's going on.
>
> Personally I'm somewhat uncomfortable with having a rule
> that "yield" in a comprehension creates a subgenerator,
> because the syntactic clues that a given "yield" is in a
> comprehension are fairly subtle.
>
> For the same reason, I'm uncomfortable with the nested
> function scope of a comprehension being an official part
> of the semantics. All other function scopes are introduced
> by a very clear piece of syntax, but with comprehensions
> you have to notice a combination of things that don't
> have anything to do with function scopes by themselves.
>
> So I think I've just talked myself into the opinion that
> anything that would allow you to tell whether comprehensions
> have an implicit function scope or not should be banned.
>

That's not an unreasonable position, and one that would leave a door open
to a general design for inner scopes that are not implemented using
functions or code objects (something to ponder for 3.8 perhaps?). In terms
of the current thread the only thing it would disallow seems to be the
current behavior of yield in comprehensions (and perhaps genexprs?).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Serhiy Storchaka wrote:


Ivan explained that this function should be rough equivalent to

   def f():
   t = [(yield i) for i in range(3)]
   return (x for x in t)

This is a *rough* equivalent. There are differences in details.


The details would seem to be overwhelmingly important, though.

I take it you're saying the semantics should be "like the
above except that the returned iterator is lazy". But that
seems impossible, because f() can't return anything until it
finishes having all its values sent to it.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Sat, Nov 25, 2017 at 1:05 PM, David Mertz  wrote:

> FWIW, on a side point. I use 'yield' and 'yield from' ALL THE TIME in real
> code. Probably 80% of those would be fine with yield statements, but a
> significant fraction use `gen.send()`.
>
> On the other hand, I have yet once to use 'await', or 'async' outside of
> pedagogical contexts. There are a whole lot of generators, including ones
> utilizing state injection, that are useful without the scaffolding of an
> event loop, in synchronous code.
>

Maybe you didn't realize async/await don't need an event loop? Driving an
async/await-based coroutine is just as simple as driving a yield-from-based
one (`await` does exactly the same thing as `yield from`).

But I won't argue with the usefulness of `yield [from]` in expressions.
That is not the topic of this thread.


> Of course, I never use them in comprehensions or generator expressions.
> And even after reading every post in this thread, the behavior (either
> existing or desired by some) such constructs have is murky and difficult
> for me to reason about. I strongly support deprecation or even just
> immediate SyntaxError in 3.7.
>

Maybe the rest of the discussion should be about deprecation vs.
SyntaxError in Python 3.7.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Nick Coghlan wrote:


def example():
comp1 = yield from [(yield x) for x in ('1st', '2nd')]
comp2 = yield from [(yield x) for x in ('3rd', '4th')]
return comp1, comp2



If the implicit "yield from" idea seems too magical, then the other
direction we could go is to make the immediate "yield from" mandatory,


Whether it's "too magical" or not depends on your stance with
regard to the implicit function scope of a comprehension. There
seem to be two irreconcilable schools of thought on that:

(1) It's an implementation detail that happens to be used to
stop the loop variable from leaking. Most of the time you can
ignore it and use the nested-loop-expansion mental model.
Most people seem to think this way in practice.

(2) It's an important part of the semantics of comprehensions
that needs to be taken into account at all times. This seems
to be Guido's position.

If you're in school (1), then the current behaviour of yield
in a comprehension is the thing that's weird and magical.
Moreover, it's bad magic, because it does something you
almost certainly don't want. Adding an implicit yield-from
would make things *less* magical and more understandable.

However, it will break things for people in school (2), who
are aware of the arcane details and know enough to put the
yield-from in themselves when needed.

Just because it breaks their code doesn't necessarily mean
they will find it more magical, though. Generators are
already sufficiently magical that they're probably smart
enough to figure out what's going on.

Personally I'm somewhat uncomfortable with having a rule
that "yield" in a comprehension creates a subgenerator,
because the syntactic clues that a given "yield" is in a
comprehension are fairly subtle.

For the same reason, I'm uncomfortable with the nested
function scope of a comprehension being an official part
of the semantics. All other function scopes are introduced
by a very clear piece of syntax, but with comprehensions
you have to notice a combination of things that don't
have anything to do with function scopes by themselves.

So I think I've just talked myself into the opinion that
anything that would allow you to tell whether comprehensions
have an implicit function scope or not should be banned.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Yury Selivanov]

On Sat, Nov 25, 2017 at 3:27 PM, Guido van Rossum  wrote:
> On Sat, Nov 25, 2017 at 9:21 AM, Yury Selivanov 
> wrote:
>>
>> So we are keeping asynchronous generator expressions as long as they are
>> defined in an 'async def' coroutine?
>
>
> I would be happy to declare that `await` is out of scope for this thread. It
> seems that it is always well-defined and sensible what it does in
> comprehensions and in genexprs. (Although I can't help noticing that PEP 530
> does not appear to propose `await` in generator expressions -- it proposes
> `async for` in comprehensions and in genexprs, and `await` in comprehensions
> only -- but they appear to be accepted nevertheless.)

Great!

As for PEP 530, after reading this discussion I realized how many
things in it are underspecified.  I'll be working on PEP 550 successor
next week and will also try to update PEP 530 to make it clearer.

> So we're back to the original issue, which is that `yield` inside a
> comprehension accidentally makes it become a generator rather than a list,
> set or dict. I believe that this can be fixed. But I don't believe we should
> fix it. I believe we should ban `yield` from comprehensions and from
> genexprs.

+1 from me.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[David Mertz]

FWIW, on a side point. I use 'yield' and 'yield from' ALL THE TIME in real
code. Probably 80% of those would be fine with yield statements, but a
significant fraction use `gen.send()`.

On the other hand, I have yet once to use 'await', or 'async' outside of
pedagogical contexts. There are a whole lot of generators, including ones
utilizing state injection, that are useful without the scaffolding of an
event loop, in synchronous code.

Of course, I never use them in comprehensions or generator expressions. And
even after reading every post in this thread, the behavior (either existing
or desired by some) such constructs have is murky and difficult for me to
reason about. I strongly support deprecation or even just immediate
SyntaxError in 3.7.

On Nov 25, 2017 12:38 PM, "Guido van Rossum"  wrote:

> On Sat, Nov 25, 2017 at 12:17 PM Brett Cannon  wrote:
>>
> On Fri, Nov 24, 2017, 19:32 Guido van Rossum,  wrote:
>>>
 On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum 
 wrote:

> The more I hear about this topic, the more I think that `await`,
> `yield` and `yield from` should all be banned from occurring in all
> comprehensions and generator expressions. That's not much different from
> disallowing `return` or `break`.
>

 From the responses it seems that I tried to simplify things too far.
 Let's say that `await` in comprehensions is fine, as long as that
 comprehension is contained in an `async def`. While we *could* save `yield
 [from]` in comprehensions, I still see it as mostly a source of confusion,
 and the fact that the presence of `yield [from]` *implicitly* makes the
 surrounding `def` a generator makes things worse. It just requires too many
 mental contortions to figure out what it does.

 I still propose to rule out all of the above from generator
 expressions, because those can escape from the surrounding scope.

>>>
>>> +1 from me
>>>
>>
> On Sat, Nov 25, 2017 at 9:21 AM, Yury Selivanov 
> wrote:
>
>> So we are keeping asynchronous generator expressions as long as they are
>> defined in an 'async def' coroutine?
>>
>
> I would be happy to declare that `await` is out of scope for this thread.
> It seems that it is always well-defined and sensible what it does in
> comprehensions and in genexprs. (Although I can't help noticing that PEP
> 530 does not appear to propose `await` in generator expressions -- it
> proposes `async for` in comprehensions and in genexprs, and `await` in
> comprehensions only -- but they appear to be accepted nevertheless.)
>
> So we're back to the original issue, which is that `yield` inside a
> comprehension accidentally makes it become a generator rather than a list,
> set or dict. I believe that this can be fixed. But I don't believe we
> should fix it. I believe we should ban `yield` from comprehensions and from
> genexprs. We don't need it, and it's confused most everyone. And the ban
> should extend to `yield from` in those same contexts. I think we have a
> hope for consensus on this.
>
> (I also think that if we had invented `await` earlier we wouldn't have
> gone down the path of `yield` expressions -- but historically it appears we
> wouldn't have invented `await` at all if we hadn't first tried `yield` and
> then `yield from` to build coroutines, so I don't think this so bad after
> all. :-)
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> mertz%40gnosis.cx
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

>
> On Sat, Nov 25, 2017 at 12:17 PM Brett Cannon  wrote:
>
On Fri, Nov 24, 2017, 19:32 Guido van Rossum,  wrote:
>>
>>> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum 
>>> wrote:
>>>
 The more I hear about this topic, the more I think that `await`,
 `yield` and `yield from` should all be banned from occurring in all
 comprehensions and generator expressions. That's not much different from
 disallowing `return` or `break`.

>>>
>>> From the responses it seems that I tried to simplify things too far.
>>> Let's say that `await` in comprehensions is fine, as long as that
>>> comprehension is contained in an `async def`. While we *could* save `yield
>>> [from]` in comprehensions, I still see it as mostly a source of confusion,
>>> and the fact that the presence of `yield [from]` *implicitly* makes the
>>> surrounding `def` a generator makes things worse. It just requires too many
>>> mental contortions to figure out what it does.
>>>
>>> I still propose to rule out all of the above from generator expressions,
>>> because those can escape from the surrounding scope.
>>>
>>
>> +1 from me
>>
>
On Sat, Nov 25, 2017 at 9:21 AM, Yury Selivanov 
wrote:

> So we are keeping asynchronous generator expressions as long as they are
> defined in an 'async def' coroutine?
>

I would be happy to declare that `await` is out of scope for this thread.
It seems that it is always well-defined and sensible what it does in
comprehensions and in genexprs. (Although I can't help noticing that PEP
530 does not appear to propose `await` in generator expressions -- it
proposes `async for` in comprehensions and in genexprs, and `await` in
comprehensions only -- but they appear to be accepted nevertheless.)

So we're back to the original issue, which is that `yield` inside a
comprehension accidentally makes it become a generator rather than a list,
set or dict. I believe that this can be fixed. But I don't believe we
should fix it. I believe we should ban `yield` from comprehensions and from
genexprs. We don't need it, and it's confused most everyone. And the ban
should extend to `yield from` in those same contexts. I think we have a
hope for consensus on this.

(I also think that if we had invented `await` earlier we wouldn't have gone
down the path of `yield` expressions -- but historically it appears we
wouldn't have invented `await` at all if we hadn't first tried `yield` and
then `yield from` to build coroutines, so I don't think this so bad after
all. :-)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Yury Selivanov]

So we are keeping asynchronous generator expressions as long as they are
defined in an 'async def' coroutine?

Yury

On Sat, Nov 25, 2017 at 12:17 PM Brett Cannon  wrote:

>
>
> On Fri, Nov 24, 2017, 19:32 Guido van Rossum,  wrote:
>
>> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum 
>> wrote:
>>
>>> The more I hear about this topic, the more I think that `await`, `yield`
>>> and `yield from` should all be banned from occurring in all comprehensions
>>> and generator expressions. That's not much different from disallowing
>>> `return` or `break`.
>>>
>>
>> From the responses it seems that I tried to simplify things too far.
>> Let's say that `await` in comprehensions is fine, as long as that
>> comprehension is contained in an `async def`. While we *could* save `yield
>> [from]` in comprehensions, I still see it as mostly a source of confusion,
>> and the fact that the presence of `yield [from]` *implicitly* makes the
>> surrounding `def` a generator makes things worse. It just requires too many
>> mental contortions to figure out what it does.
>>
>> I still propose to rule out all of the above from generator expressions,
>> because those can escape from the surrounding scope.
>>
>
> +1 from me.
>
> -Brett
>
>
>> --
>> --Guido van Rossum (python.org/~guido)
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Brett Cannon]

On Fri, Nov 24, 2017, 19:32 Guido van Rossum,  wrote:

> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum 
> wrote:
>
>> The more I hear about this topic, the more I think that `await`, `yield`
>> and `yield from` should all be banned from occurring in all comprehensions
>> and generator expressions. That's not much different from disallowing
>> `return` or `break`.
>>
>
> From the responses it seems that I tried to simplify things too far. Let's
> say that `await` in comprehensions is fine, as long as that comprehension
> is contained in an `async def`. While we *could* save `yield [from]` in
> comprehensions, I still see it as mostly a source of confusion, and the
> fact that the presence of `yield [from]` *implicitly* makes the surrounding
> `def` a generator makes things worse. It just requires too many mental
> contortions to figure out what it does.
>
> I still propose to rule out all of the above from generator expressions,
> because those can escape from the surrounding scope.
>

+1 from me.

-Brett


> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Sat, Nov 25, 2017 at 8:07 AM, Ivan Levkivskyi 
wrote:

> On 25 November 2017 at 16:57, Guido van Rossum  wrote:
>
>> On Sat, Nov 25, 2017 at 6:55 AM, Ivan Levkivskyi 
>> wrote:
>>
>>> On 25 November 2017 at 04:30, Guido van Rossum  wrote:
>>>
 On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum 
 wrote:

> The more I hear about this topic, the more I think that `await`,
> `yield` and `yield from` should all be banned from occurring in all
> comprehensions and generator expressions. That's not much different from
> disallowing `return` or `break`.
>

 From the responses it seems that I tried to simplify things too far.
 Let's say that `await` in comprehensions is fine, as long as that
 comprehension is contained in an `async def`. While we *could* save `yield
 [from]` in comprehensions, I still see it as mostly a source of confusion,
 and the fact that the presence of `yield [from]` *implicitly* makes the
 surrounding `def` a generator makes things worse. It just requires too many
 mental contortions to figure out what it does.

>>>
>>> [...]
>>> If the first example will be allowed, then one will be surprised why it
>>> can't be rewritten as
>>>
>>> def pack_two():
>>> return [(yield) for _ in range(2)]
>>>
>>
>> And yet Nick's example shows that that is not equivalent!
>>
>> [...]
>>
>> In this example each thing that looks syntactically like a list
>> comprehension becomes actually a generator expression at at runtime! And so
>> does your example, so instead of a list of two items, it returns a
>> generator that will produce two values when iterated over.
>>
>> That's not referential transparency to me, it feels more like a bug in
>> the code generator.
>>
>> I want to ban this because apparently nobody besides Nick knows about
>> this behavior (I certainly didn't, and from the above it seems you don't
>> either).
>>
>
> This whole thread started as a proposal to fix this bug and to make the
> two forms equivalent, so I don't know what you are talking about.
>

I see. I misread your equivalence example as "this how it works" -- you
meant it as "this is how I propose we fix it".

The fix is not unreasonable but I still would like to retreat to the
territory where `yield [from]` in comprehensions (and generator
expressions) is deemed invalid.


> Also as there appeared arguments of authority (thanks Antoine) its time to
> stop this discussion for me.
>

I'd be happy to stop with the conclusion that we're going to rip out some
confusing syntax rather than trying to generate code for it -- IMO we've
proved to ourselves that this stuff is too complicated to be useful.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 25 November 2017 at 16:57, Guido van Rossum  wrote:

> On Sat, Nov 25, 2017 at 6:55 AM, Ivan Levkivskyi 
> wrote:
>
>> On 25 November 2017 at 04:30, Guido van Rossum  wrote:
>>
>>> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum 
>>> wrote:
>>>
 The more I hear about this topic, the more I think that `await`,
 `yield` and `yield from` should all be banned from occurring in all
 comprehensions and generator expressions. That's not much different from
 disallowing `return` or `break`.

>>>
>>> From the responses it seems that I tried to simplify things too far.
>>> Let's say that `await` in comprehensions is fine, as long as that
>>> comprehension is contained in an `async def`. While we *could* save `yield
>>> [from]` in comprehensions, I still see it as mostly a source of confusion,
>>> and the fact that the presence of `yield [from]` *implicitly* makes the
>>> surrounding `def` a generator makes things worse. It just requires too many
>>> mental contortions to figure out what it does.
>>>
>>
>> [...]
>> If the first example will be allowed, then one will be surprised why it
>> can't be rewritten as
>>
>> def pack_two():
>> return [(yield) for _ in range(2)]
>>
>
> And yet Nick's example shows that that is not equivalent!
>
> [...]
>
> In this example each thing that looks syntactically like a list
> comprehension becomes actually a generator expression at at runtime! And so
> does your example, so instead of a list of two items, it returns a
> generator that will produce two values when iterated over.
>
> That's not referential transparency to me, it feels more like a bug in the
> code generator.
>
> I want to ban this because apparently nobody besides Nick knows about this
> behavior (I certainly didn't, and from the above it seems you don't either).
>

This whole thread started as a proposal to fix this bug and to make the two
forms equivalent, so I don't know what you are talking about.

Also as there appeared arguments of authority (thanks Antoine) its time to
stop this discussion for me.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Sat, Nov 25, 2017 at 6:55 AM, Ivan Levkivskyi 
wrote:

> On 25 November 2017 at 04:30, Guido van Rossum  wrote:
>
>> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum 
>> wrote:
>>
>>> The more I hear about this topic, the more I think that `await`, `yield`
>>> and `yield from` should all be banned from occurring in all comprehensions
>>> and generator expressions. That's not much different from disallowing
>>> `return` or `break`.
>>>
>>
>> From the responses it seems that I tried to simplify things too far.
>> Let's say that `await` in comprehensions is fine, as long as that
>> comprehension is contained in an `async def`. While we *could* save `yield
>> [from]` in comprehensions, I still see it as mostly a source of confusion,
>> and the fact that the presence of `yield [from]` *implicitly* makes the
>> surrounding `def` a generator makes things worse. It just requires too many
>> mental contortions to figure out what it does.
>>
>
> There were some arguments that `await` is like a function call, while
> `yield` is like `return`.
> TBH, I don't really like these arguments since to me they are to vague.
> Continuing this logic one can say that
> `return` is just a fancy function call (calling continuation with the
> result). To me there is one clear distinction:
> `return` and `break` are statements, while `yield`, `yield from`, and
> `await` are expressions.
>

Indeed. However, `yield from` as an expression is mostly a deprecated way
to write `await`, and `yield` as an expression is mostly an alternative way
of writing coroutines (it's the only way that exists in Python 2). Another
big difference is that the use of `yield [from]` affects the surrounding
function, making it a generator.

Continuing the topic of the ban, what exactly should be banned? For example
> will this still be valid?
>
> def pack_two():
> return [(yield), (yield)]  # Just a list display
>

It's not a comprehension so it's still valid.


> I don't see how this is controversial. It is clear that `pack_two` is a
> generator.
> If this is going to be prohibited, then one may be surprised by lack of
> referential transparency, since this will be valid:
>
> def pack_two():
> first = (yield)
> second = (yield)
> return [first, second]
>
> If the first example will be allowed, then one will be surprised why it
> can't be rewritten as
>
> def pack_two():
> return [(yield) for _ in range(2)]
>

And yet Nick's example shows that that is not equivalent!

def example():
comp1 = yield from [(yield x) for x in ('1st', '2nd')]
comp2 = yield from [(yield x) for x in ('3rd', '4th')]
return comp1, comp2

In this example each thing that looks syntactically like a list
comprehension becomes actually a generator expression at at runtime! And so
does your example, so instead of a list of two items, it returns a
generator that will produce two values when iterated over.

That's not referential transparency to me, it feels more like a bug in the
code generator.

I want to ban this because apparently nobody besides Nick knows about this
behavior (I certainly didn't, and from the above it seems you don't either).


> I have found several other examples where it is not clear whether they
> should be prohibited with `yield` or not.
>

Such as?


> I still propose to rule out all of the above from generator expressions,
>> because those can escape from the surrounding scope.
>>
>
> Here I agree. Also note that the above problem does not apply to generator
> expressions since (x, x) and (x for _ in range(2)) are
> two very different expressions.
>

PS. A more radical proposal (not for 3.7) would be to deprecate yield as an
expression. It once was only a statement, but PEP 342 introduced yield as
an expression in order to do coroutines. We now have `async def` and
`await` as a superior coroutine mechanism. But we must continue to support
yield expressions because there is a lot of Python 2/3 compatible code that
depends on it. (E.g. Tornado.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

At this point, the fact that several Python core developers fail to
understand the pieces of code presented as examples should be a hint
that the syntax here is far from desirable...

Regards

Antoine.


On Sat, 25 Nov 2017 15:47:14 +
Paul Moore  wrote:
> On 25 November 2017 at 14:55, Ivan Levkivskyi  wrote:
> > Continuing the topic of the ban, what exactly should be banned? For example
> > will this still be valid?
> >
> > def pack_two():
> > return [(yield), (yield)]  # Just a list display
> >
> > I don't see how this is controversial. It is clear that `pack_two` is a
> > generator.  
> 
> It's not clear to me...
> 
> Seriously, I wouldn't know what this would do. Presumably it needs
> someone calling send() to generate results (because that's now yield
> expressions work) but beyond that, I don't really understand what it
> does. Maybe an example that wasn't artificial would be more obvious,
> I'm not sure.
> 
> > If this is going to be prohibited, then one may be surprised by lack of
> > referential transparency, since this will be valid:
> >
> > def pack_two():
> > first = (yield)
> > second = (yield)
> > return [first, second]  
> 
> The fact that you can't inline first and second doesn't bother me. I
> can't fully explain why, but it doesn't. (Technically, I *can* explain
> why, but you'd disagree with my explanation :-))
> 
> > If the first example will be allowed, then one will be surprised why it
> > can't be rewritten as
> >
> > def pack_two():
> > return [(yield) for _ in range(2)]
> >
> > I have found several other examples where it is not clear whether they
> > should be prohibited with `yield` or not.  
> 
> So far, none of your examples have demonstrated anything that Guido's
> suggestion to ban yield would make confusing *to me*.
> 
> Maybe this demonstrates nothing more than how inconsistent and shallow
> my understanding of yield expressions is. That's fine, I can live with
> that. I can't give you any assurances that my level of understanding
> is common among non-experts, but I will say that in my view, Guido's
> proposal feels sensible and intuitive. And after all, if the use of
> yield expressions becomes significantly more common, and the general
> level of familiarity with the concept increases, it's easy enough to
> relax the restriction later, in line with the average user's level of
> comfort.
> 
> Paul



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 25 November 2017 at 14:55, Ivan Levkivskyi  wrote:
> Continuing the topic of the ban, what exactly should be banned? For example
> will this still be valid?
>
> def pack_two():
> return [(yield), (yield)]  # Just a list display
>
> I don't see how this is controversial. It is clear that `pack_two` is a
> generator.

It's not clear to me...

Seriously, I wouldn't know what this would do. Presumably it needs
someone calling send() to generate results (because that's now yield
expressions work) but beyond that, I don't really understand what it
does. Maybe an example that wasn't artificial would be more obvious,
I'm not sure.

> If this is going to be prohibited, then one may be surprised by lack of
> referential transparency, since this will be valid:
>
> def pack_two():
> first = (yield)
> second = (yield)
> return [first, second]

The fact that you can't inline first and second doesn't bother me. I
can't fully explain why, but it doesn't. (Technically, I *can* explain
why, but you'd disagree with my explanation :-))

> If the first example will be allowed, then one will be surprised why it
> can't be rewritten as
>
> def pack_two():
> return [(yield) for _ in range(2)]
>
> I have found several other examples where it is not clear whether they
> should be prohibited with `yield` or not.

So far, none of your examples have demonstrated anything that Guido's
suggestion to ban yield would make confusing *to me*.

Maybe this demonstrates nothing more than how inconsistent and shallow
my understanding of yield expressions is. That's fine, I can live with
that. I can't give you any assurances that my level of understanding
is common among non-experts, but I will say that in my view, Guido's
proposal feels sensible and intuitive. And after all, if the use of
yield expressions becomes significantly more common, and the general
level of familiarity with the concept increases, it's easy enough to
relax the restriction later, in line with the average user's level of
comfort.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 25 November 2017 at 04:30, Guido van Rossum  wrote:

> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum 
> wrote:
>
>> The more I hear about this topic, the more I think that `await`, `yield`
>> and `yield from` should all be banned from occurring in all comprehensions
>> and generator expressions. That's not much different from disallowing
>> `return` or `break`.
>>
>
> From the responses it seems that I tried to simplify things too far. Let's
> say that `await` in comprehensions is fine, as long as that comprehension
> is contained in an `async def`. While we *could* save `yield [from]` in
> comprehensions, I still see it as mostly a source of confusion, and the
> fact that the presence of `yield [from]` *implicitly* makes the surrounding
> `def` a generator makes things worse. It just requires too many mental
> contortions to figure out what it does.
>

There were some arguments that `await` is like a function call, while
`yield` is like `return`.
TBH, I don't really like these arguments since to me they are to vague.
Continuing this logic one can say that
`return` is just a fancy function call (calling continuation with the
result). To me there is one clear distinction:
`return` and `break` are statements, while `yield`, `yield from`, and
`await` are expressions.

Continuing the topic of the ban, what exactly should be banned? For example
will this still be valid?

def pack_two():
return [(yield), (yield)]  # Just a list display

I don't see how this is controversial. It is clear that `pack_two` is a
generator.
If this is going to be prohibited, then one may be surprised by lack of
referential transparency, since this will be valid:

def pack_two():
first = (yield)
second = (yield)
return [first, second]

If the first example will be allowed, then one will be surprised why it
can't be rewritten as

def pack_two():
return [(yield) for _ in range(2)]

I have found several other examples where it is not clear whether they
should be prohibited with `yield` or not.

I still propose to rule out all of the above from generator expressions,
> because those can escape from the surrounding scope.
>

Here I agree. Also note that the above problem does not apply to generator
expressions since (x, x) and (x for _ in range(2)) are
two very different expressions.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

24.11.17 00:20, Greg Ewing пише:

Serhiy Storchaka wrote:

Ivan explained that this function should be rough equivalent to

   def f():
   t = [(yield i) for i in range(3)]
   return (x for x in t)


This seems useless to me. It turns a lazy iterator
into an eager one, which is a gross violation of the
author's intent in using a generator expression.


This is a *rough* equivalent. There are differences in details.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

24.11.17 02:50, Nick Coghlan пише:

If we went down that path, then a list comprehension like the following:

     results = [(yield future) for future in list_of_futures]

might be compiled as being equivalent to:

     def __listcomp_generator(iterable):
     result = []
     for future in iterable:
     result.append((yield future))
     return result

     results = yield from _listcomp_generator(list_of_futures)

The only difference between the current comprehension code and this idea 
is "an explicit yield expression in a comprehension implies the use of 
'yield from' when calling the nested function".


Oh, nice! This is a much simpler solution! And it solves all related 
problems. I like it.


This has an overhead in comparison with inlining the code, but the 
latter can be considered just as an optimization. We can apply it when 
prove the need in optimizing this construction. For now it is enough if 
it just works.


The fact that two independent mental models lead to the same result is 
an argument for their correctness.


I'm not so sure about "yield" in generators, this will need further 
thoughts and experiments. "yield" can be used not only in the item 
expression, but in conditions and inner iterables.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Sat, 25 Nov 2017 17:10:12 +1000
Nick Coghlan  wrote:
> On 25 November 2017 at 16:18, Nathaniel Smith  wrote:
> > On Fri, Nov 24, 2017 at 9:39 PM, Nick Coghlan  wrote:  
> >> On 25 November 2017 at 15:27, Nathaniel Smith  wrote:  
> >>> On Fri, Nov 24, 2017 at 9:04 PM, Nick Coghlan  wrote: 
> >>>  
>  def example():
>  comp1 = yield from [(yield x) for x in ('1st', '2nd')]
>  comp2 = yield from [(yield x) for x in ('3rd', '4th')]
>  return comp1, comp2  
> >>>
> >>> Isn't this a really confusing way of writing
> >>>
> >>> def example():
> >>> return [(yield '1st'), (yield '2nd')], [(yield '3rd'), (yield '4th')] 
> >>>  
> >>
> >> A real use case  
> >
> > Do you have a real use case? This seems incredibly niche...  
> 
> That's not how backwards compatibility works - we were suggesting
> getting rid of this syntax, because there was no current way to make
> it do anything sensible.
> 
> It turns out there is a way to make it behave reasonably - you just
> need to stick "yield from" in front of it, and it goes back to being
> equivalent to the corresponding for loop (the same as the synchronous
> version).

I don't know if it behaves reasonably, but it's entirely unreadable to
me.  I think there's a case for discouraging unreadable constructs.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Sat, 25 Nov 2017 03:03:02 +
MRAB  wrote:

> On 2017-11-25 02:21, Chris Jerdonek wrote:
> > On Fri, Nov 24, 2017 at 5:06 PM, Nathaniel Smith  wrote:  
> >> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum  
> >> wrote:  
> >>> The more I hear about this topic, the more I think that `await`, `yield` 
> >>> and
> >>> `yield from` should all be banned from occurring in all comprehensions and
> >>> generator expressions. That's not much different from disallowing `return`
> >>> or `break`.  
> >>
> >> I would say that banning `yield` and `yield from` is like banning
> >> `return` and `break`, but banning `await` is like banning function
> >> calls.  
> > 
> > I agree. I was going to make the point earlier in the thread that
> > using "await" can mostly just be thought of as a delayed function
> > call, but it didn't seem at risk of getting banned until Guido's
> > comment so I didn't say anything (and there were too many comments
> > anyways).
> > 
> > I think it's in a different category for that reason. It's much easier
> > to reason about than, say, "yield" and "yield from".
> >   
> +1
> 
> Seeing "await" there didn't/doesn't confuse me; seeing "yield" or "yield 
> from" does.

+1 as well.

Regards

Antoine.





___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 25 November 2017 at 16:18, Nathaniel Smith  wrote:
> On Fri, Nov 24, 2017 at 9:39 PM, Nick Coghlan  wrote:
>> On 25 November 2017 at 15:27, Nathaniel Smith  wrote:
>>> On Fri, Nov 24, 2017 at 9:04 PM, Nick Coghlan  wrote:
 def example():
 comp1 = yield from [(yield x) for x in ('1st', '2nd')]
 comp2 = yield from [(yield x) for x in ('3rd', '4th')]
 return comp1, comp2
>>>
>>> Isn't this a really confusing way of writing
>>>
>>> def example():
>>> return [(yield '1st'), (yield '2nd')], [(yield '3rd'), (yield '4th')]
>>
>> A real use case
>
> Do you have a real use case? This seems incredibly niche...

That's not how backwards compatibility works - we were suggesting
getting rid of this syntax, because there was no current way to make
it do anything sensible.

It turns out there is a way to make it behave reasonably - you just
need to stick "yield from" in front of it, and it goes back to being
equivalent to the corresponding for loop (the same as the synchronous
version).

>> wouldn't be iterating over hardcoded tuples in the
>> comprehensions, it would be something more like:
>>
>> def example(iterable1, iterable2):
>> comp1 = yield from [(yield x) for x in iterable1]
>> comp2 = yield from [(yield x) for x in iterable2]
>> return comp1, comp2
>
> I submit that this would still be easier to understand if written out like:
>
> def map_iterable_to_yield_values(iterable):
> "Yield the values in iterable, then return a list of the values sent 
> back."
> result = []
> for obj in iterable:
> result.append(yield obj)
> return result
>
> def example(iterable1, iterable2):
> values1 = yield from map_iterable_to_yield_values(iterable1)
> values2 = yield from map_iterable_to_yield_values(iterable2)
> return values1, values2

The same can be said for comprehensions in general. Composing them
with coroutines certainly doesn't make either easier to understand,
but I don't think replacing the comprehension with its full imperative
form is particularly helpful in aiding that understanding.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nathaniel Smith]

On Fri, Nov 24, 2017 at 9:39 PM, Nick Coghlan  wrote:
> On 25 November 2017 at 15:27, Nathaniel Smith  wrote:
>> On Fri, Nov 24, 2017 at 9:04 PM, Nick Coghlan  wrote:
>>> def example():
>>> comp1 = yield from [(yield x) for x in ('1st', '2nd')]
>>> comp2 = yield from [(yield x) for x in ('3rd', '4th')]
>>> return comp1, comp2
>>
>> Isn't this a really confusing way of writing
>>
>> def example():
>> return [(yield '1st'), (yield '2nd')], [(yield '3rd'), (yield '4th')]
>
> A real use case

Do you have a real use case? This seems incredibly niche...

> wouldn't be iterating over hardcoded tuples in the
> comprehensions, it would be something more like:
>
> def example(iterable1, iterable2):
> comp1 = yield from [(yield x) for x in iterable1]
> comp2 = yield from [(yield x) for x in iterable2]
> return comp1, comp2

I submit that this would still be easier to understand if written out like:

def map_iterable_to_yield_values(iterable):
"Yield the values in iterable, then return a list of the values sent back."
result = []
for obj in iterable:
result.append(yield obj)
return result

def example(iterable1, iterable2):
values1 = yield from map_iterable_to_yield_values(iterable1)
values2 = yield from map_iterable_to_yield_values(iterable2)
return values1, values2

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 25 November 2017 at 15:27, Nathaniel Smith  wrote:
> On Fri, Nov 24, 2017 at 9:04 PM, Nick Coghlan  wrote:
>> def example():
>> comp1 = yield from [(yield x) for x in ('1st', '2nd')]
>> comp2 = yield from [(yield x) for x in ('3rd', '4th')]
>> return comp1, comp2
>
> Isn't this a really confusing way of writing
>
> def example():
> return [(yield '1st'), (yield '2nd')], [(yield '3rd'), (yield '4th')]

A real use case wouldn't be iterating over hardcoded tuples in the
comprehensions, it would be something more like:

def example(iterable1, iterable2):
comp1 = yield from [(yield x) for x in iterable1]
comp2 = yield from [(yield x) for x in iterable2]
return comp1, comp2

Defining an interesting for loop isn't the point of the example though
- it's just to show that if you're inside a generator, you can already
make a subgenerator comprehension do something sensible by sticking
"yield from" in front of it (and have actually been able to do so
since 3.3, when "yield from" was first introduced).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nathaniel Smith]

On Fri, Nov 24, 2017 at 9:04 PM, Nick Coghlan  wrote:
> def example():
> comp1 = yield from [(yield x) for x in ('1st', '2nd')]
> comp2 = yield from [(yield x) for x in ('3rd', '4th')]
> return comp1, comp2

Isn't this a really confusing way of writing

def example():
return [(yield '1st'), (yield '2nd')], [(yield '3rd'), (yield '4th')]

?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 25 November 2017 at 13:30, Guido van Rossum  wrote:
>
> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum  wrote:
>>
>> The more I hear about this topic, the more I think that `await`, `yield` and 
>> `yield from` should all be banned from occurring in all comprehensions and 
>> generator expressions. That's not much different from disallowing `return` 
>> or `break`.
>
>
> From the responses it seems that I tried to simplify things too far. Let's 
> say that `await` in comprehensions is fine, as long as that comprehension is 
> contained in an `async def`. While we *could* save `yield [from]` in 
> comprehensions, I still see it as mostly a source of confusion, and the fact 
> that the presence of `yield [from]` *implicitly* makes the surrounding `def` 
> a generator makes things worse. It just requires too many mental contortions 
> to figure out what it does.

While I'm OK with ruling the interaction of all of these
subexpressions with generator expression too confusingly weird to be
supportable, in https://bugs.python.org/issue10544?#msg306940 I came
up with an example for the existing comprehension behaviour that I'm
hesitant to break without a clear replacement: code that wraps the
*current* comprehension behaviour in an explicit "yield from"
expression.

That is, code like:

def example():
comp1 = yield from [(yield x) for x in ('1st', '2nd')]
comp2 = yield from [(yield x) for x in ('3rd', '4th')]
return comp1, comp2

Creates a generator that returns a 2-tuple containing two 2-item
lists, but the exact contents of those lists depend on the values you
send into the generator while it is running.

The basic principle behind the construct is that a comprehension
containing a yield expression essentially defines an inline
subgenerator, so you then have to yield from that subgenerator in
order to produce the desired list.

If the implicit "yield from" idea seems too magical, then the other
direction we could go is to make the immediate "yield from" mandatory,
such that leaving it out produces a SyntaxWarning in 3.7, and then a
SyntaxError in 3.8+. (Something like "'yield from' clause missing
before subgenerator comprehension")

While we don't generally like bolting two pieces of syntax together
like that, I think it's defensible in this case based on the fact that
allowing "[expr for var in iterable]" to ever return anything other
than a list (and similarly for the other comprehension types) is
genuinely confusing.

The nicest aspect of the "an explicit and immediate 'yield from' is
required when using subgenerator comprehensions" approach is that the
above syntax already works today, and has actually worked since 3.3 -
we'd just never put the pieces together properly to establish this as
a potential pattern for comprehension use in coroutines.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum  wrote:

> The more I hear about this topic, the more I think that `await`, `yield`
> and `yield from` should all be banned from occurring in all comprehensions
> and generator expressions. That's not much different from disallowing
> `return` or `break`.
>

>From the responses it seems that I tried to simplify things too far. Let's
say that `await` in comprehensions is fine, as long as that comprehension
is contained in an `async def`. While we *could* save `yield [from]` in
comprehensions, I still see it as mostly a source of confusion, and the
fact that the presence of `yield [from]` *implicitly* makes the surrounding
`def` a generator makes things worse. It just requires too many mental
contortions to figure out what it does.

I still propose to rule out all of the above from generator expressions,
because those can escape from the surrounding scope.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[MRAB]

On 2017-11-25 02:21, Chris Jerdonek wrote:

On Fri, Nov 24, 2017 at 5:06 PM, Nathaniel Smith  wrote:

On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum  wrote:

The more I hear about this topic, the more I think that `await`, `yield` and
`yield from` should all be banned from occurring in all comprehensions and
generator expressions. That's not much different from disallowing `return`
or `break`.


I would say that banning `yield` and `yield from` is like banning
`return` and `break`, but banning `await` is like banning function
calls.


I agree. I was going to make the point earlier in the thread that
using "await" can mostly just be thought of as a delayed function
call, but it didn't seem at risk of getting banned until Guido's
comment so I didn't say anything (and there were too many comments
anyways).

I think it's in a different category for that reason. It's much easier
to reason about than, say, "yield" and "yield from".


+1

Seeing "await" there didn't/doesn't confuse me; seeing "yield" or "yield 
from" does.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 25 November 2017 at 11:04, Ivan Levkivskyi  wrote:

> On 25 November 2017 at 01:22, Guido van Rossum  wrote:
>
>> The more I hear about this topic, the more I think that `await`, `yield`
>> and `yield from` should all be banned from occurring in all comprehensions
>> and generator expressions. That's not much different from disallowing
>> `return` or `break`.
>>
>>
> IIUC this would essentially mean rejecting PEP 530.
> What do you think about banning `await`, `yield` and `yield from` only
> from generator expressions?
> Comprehensions can be fixed (just make them equivalent to for-loops
> without leaks as discussed).
>

I'm leaning towards this as well - the immediate evaluation on
comprehensions means that injecting an automatic "yield from" when will do
the right thing due to this component of PEP 380:
https://www.python.org/dev/peps/pep-0380/#the-refactoring-principle

Restating how folks intuitively think Py3 comprehensions containing "yield"
and "await" should work (''__var" indicates hidden variables only
accessible to the interpreter):

result = [(yield x) for x in iterable]

   # Readers expect this to work much like
__listcomp1 = []
for __x1 in iterable:
__listcomp1.append(yield __x1)
result = __listcomp1

PEP 380's refactoring principle then means the above is required to be
equivalent to:

def __listcomp1(__outermost_iterable):
__result = []
for  x in __outermost_iterable:
__result.append(yield x)
return __result

result = yield from __listcomp1(iterable)

Which means that only missing piece in the current implementation is
inserting the implicit "yield from" into the function containing the
comprehension. No need for new lexical scoping conventions, so need for
syntactical special cases, just teaching this part of the compiler to
delegate to a subgenerator correctly. The compiler should just need a small
local adjustment to check whether the implicit nested scope is a generator
scope or not (and any Python source compiler, including CPython's, already
needs to track that information in order to ensure it emits a generator
function instead of a regular synchronous function).

>From a language evolution perspective, such a change is relatively
straightforward to explain on the basis of "Comprehensions became
implicitly nested scopes in 3.0, but we didn't implement PEP 380's 'yield
from' construct until Python 3.3, and it then took us until 3.7 to realise
we could combine them to make yield expressions inside comprehensions
behave more intuitively".

For "await", I'm pretty sure the current semantics are actually OK, but I
confess I haven't been in a position to have to explain them to anyone
either.

I honestly don't see a good way to salvage yield or yield from inside
generator expressions though. If we dropped the implicit "yield" when an
explicit one is present, that would break the symmetry between "[(yield x)
for x in iterable]" and "list((yield x) for x in iterable)", while
inserting an implicit "yield from" (as I'm now suggesting we do for
comprehensions) wouldn't work at all (since a genexp returns a generator
*fuction*, not a generator-iterator object). I'm also pretty sure it will
be difficult for the compiler to tell the difference between explicit and
implicit yield expressions in the subfunction (as there's nothing else
anywhere in the language that needs to make that distinction).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Chris Jerdonek]

On Fri, Nov 24, 2017 at 5:06 PM, Nathaniel Smith  wrote:
> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum  wrote:
>> The more I hear about this topic, the more I think that `await`, `yield` and
>> `yield from` should all be banned from occurring in all comprehensions and
>> generator expressions. That's not much different from disallowing `return`
>> or `break`.
>
> I would say that banning `yield` and `yield from` is like banning
> `return` and `break`, but banning `await` is like banning function
> calls.

I agree. I was going to make the point earlier in the thread that
using "await" can mostly just be thought of as a delayed function
call, but it didn't seem at risk of getting banned until Guido's
comment so I didn't say anything (and there were too many comments
anyways).

I think it's in a different category for that reason. It's much easier
to reason about than, say, "yield" and "yield from".

--Chris
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ethan Furman]

On 11/24/2017 04:22 PM, Guido van Rossum wrote:


The more I hear about this topic, the more I think that `await`, `yield` and 
`yield from` should all be banned from
occurring in all comprehensions and generator expressions. That's not much 
different from disallowing `return` or `break`.


For me, the deciding factor would be the the affect upon:

1) the containing function (if any); and
2) the result of the genexp/comprehension

I think of generator expressions / comprehensions as self-contained units of code that will have no (side-)effects upon 
the containing function and/or surrounding code (and a containing function is not necessary), and that each will return 
the type expressed by the syntax.


In other words:

[ l for l in ...]  -> list
{s  for s in ...]  -> set
{k:v for k, v in ...]  -> dict
(g for g in ...]  -> genexp

Since 'yield' and 'yield from' invalidate those, I'm in favor of declaring them 
syntax errors.

If 'await' does not invalidate the above constraints then I'm fine with 
allowing them.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Yury Selivanov]

On Fri, Nov 24, 2017 at 7:22 PM, Guido van Rossum  wrote:
> The more I hear about this topic, the more I think that `await`, `yield` and
> `yield from` should all be banned from occurring in all comprehensions and
> generator expressions. That's not much different from disallowing `return`
> or `break`.

IMO disallowing using await in comprehensions would be a huge mistake.
I've personally seen a lot of code that uses the syntax.

Moreover, I haven't seen any complaints about await expressions in
comprehensions in this thread or *anywhere else*.  In this thread I
only see complaints about 'yield' expression, and that's easy to
understand -- yield in comprehensions is just unusable.  'await' is
fundamentally different, because it's essentially a calling
convention.

While "provisional" status for a PEP ultimately gives us power to
remove every trace of it in the next release, I think that it
decisions like that have to be based on some commonly reported
problems and complaints.  My 2c.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nathaniel Smith]

On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum  wrote:
> The more I hear about this topic, the more I think that `await`, `yield` and
> `yield from` should all be banned from occurring in all comprehensions and
> generator expressions. That's not much different from disallowing `return`
> or `break`.

I would say that banning `yield` and `yield from` is like banning
`return` and `break`, but banning `await` is like banning function
calls. There's no reason for most users to even know that `await` is
related to generators, so a rule disallowing it inside comprehensions
is just confusing. AFAICT 99% of the confusion around async/await is
because people think of them as being related to generators, when from
the user point of view it's not true at all and `await` is just a
funny function-call syntax.

Also, at the language level, there's a key difference between these
cases. A comprehension has implicit `yield`s in it, and then mixing in
explicit `yield`s as well obviously leads to confusion. But when you
use an `await` in a comprehension, that turns it into an async
generator expression (thanks to PEP 530), and in an async generator,
`yield` and `await` use two separate, unrelated channels. So there's
no confusion or problem with having `await` inside a comprehension.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 25 November 2017 at 01:22, Guido van Rossum  wrote:

> The more I hear about this topic, the more I think that `await`, `yield`
> and `yield from` should all be banned from occurring in all comprehensions
> and generator expressions. That's not much different from disallowing
> `return` or `break`.
>
>
IIUC this would essentially mean rejecting PEP 530.
What do you think about banning `await`, `yield` and `yield from` only from
generator expressions?
Comprehensions can be fixed (just make them equivalent to for-loops without
leaks as discussed).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

The more I hear about this topic, the more I think that `await`, `yield`
and `yield from` should all be banned from occurring in all comprehensions
and generator expressions. That's not much different from disallowing
`return` or `break`.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

OK, so my 24 hours are over :-)


On 24 November 2017 at 01:50, Nick Coghlan  wrote:

> On 23 November 2017 at 23:04, Ivan Levkivskyi 
> wrote:
>
>> I don't see why this particular case qualifies for such a radical measure
>> as an exception to syntactic rules,
>> instead of just fixing it (sorry Nick :-)
>>
>
> I've posted in more detail about this to the issue tracker, but the
> argument here is: because making it behave differently from the way it does
> now while still hiding the loop iteration variable potentially requires
> even more radical revisions to the lexical scoping rules :)
>
If somebody can come up with a clever trick to allow yield inside a
> comprehension to jump levels in a relatively intuitive way, that would
> actually be genuinely cool, but the lexical scoping rules mean it's
> trickier than it sounds.
>

"potentially" is the key word here. The plan is to avoid "more radical
revisions".


> Now that I frame the question that way, though, I'm also remembering that
> we didn't have "yield from" yet when I wrote the current comprehension
> implementation, and given that, it may be as simple as having an explicit
> yield expression in a comprehension imply delegation to a subgenerator.
>

My understanding is that this is exactly how async comprehensions are
currently implemented and why they work as one would naively expect, i.e.
`await` is "bound" to the surrounding async def, not to the implicit scope
async def. So that an async comprehension is just equivalent to a for-loop.
However, although "implicit yield from" solution is simpler than the one
proposed by Serhiy, I still more like the latter one. The former has its
strange cases, for example I mentioned before in this thread:

>>> async def f():
... for i in range(3):
... yield i
...
>>> async def g():
... return [(yield i) async for i in f()]
...
>>> g().send(None)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in g
TypeError: object async_generator can't be used in 'await' expression

My understanding is that the strange error is exactly because of the
implicit `yield from`. With Serhiy's approach this would work.
Another minor bonus of Serhiy's idea is a performance gain: we will not
push an execution frame.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Hrvoje Niksic]

Guido van Rossum writes:

And my mind boggles when considering a generator expression
containing yield that is returned from a function. I tried this
and cannot say I expected the outcome:

    def f():
        return ((yield i) for i in range(3))
    print(list(f()))

In both Python 2 and Python 3 this prints

    [0, None, 1, None, 2, None]

Even if there's a totally logical explanation for that, I still
don't like it, and I think yield in a comprehension should be
banned. From this it follows that we should also simply ban
yield  from comprehensions.



Serhiy Storchaka writes:

This behavior doesn't look correct to me and Ivan.
The behavior is surprising, but it seems quite consistent with how 
generator expressions are defined in the language. A generator 
expression is defined by the language reference as "compact generator 
notation in parentheses", which yields (sic!) a "new generator object".


I take that to mean that a generator expression is equivalent to 
defining and calling a generator function. f() can be transformed to:


def f():
def _gen():
for i in range(3):
ret = yield i
yield ret
return _gen()

The transformed version shows that there are *two* yields per iteration 
(one explicitly written and one inserted by the transformation), which 
is the reason why 6 values are produced. The None values come from list 
constructor calling __next__() on the generator, which (as per 
documentation) sends None into the generator. This None value is yielded 
after the "i" is yielded, which is why Nones follow the numbers.


Hrvoje
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 23 November 2017 at 23:04, Ivan Levkivskyi  wrote:

> I don't see why this particular case qualifies for such a radical measure
> as an exception to syntactic rules,
> instead of just fixing it (sorry Nick :-)
>

I've posted in more detail about this to the issue tracker, but the
argument here is: because making it behave differently from the way it does
now while still hiding the loop iteration variable potentially requires
even more radical revisions to the lexical scoping rules :)

If somebody can come up with a clever trick to allow yield inside a
comprehension to jump levels in a relatively intuitive way, that would
actually be genuinely cool, but the lexical scoping rules mean it's
trickier than it sounds.

Now that I frame the question that way, though, I'm also remembering that
we didn't have "yield from" yet when I wrote the current comprehension
implementation, and given that, it may be as simple as having an explicit
yield expression in a comprehension imply delegation to a subgenerator.

If we went down that path, then a list comprehension like the following:

results = [(yield future) for future in list_of_futures]

might be compiled as being equivalent to:

def __listcomp_generator(iterable):
result = []
for future in iterable:
result.append((yield future))
return result

results = yield from _listcomp_generator(list_of_futures)

The only difference between the current comprehension code and this idea is
"an explicit yield expression in a comprehension implies the use of 'yield
from' when calling the nested function".

For generator expressions, the adjustment would need to be slightly
different: for those, we'd either need to prohibit yield expressions, or
else say that if there's an explicit yield expression present anywhere,
then we drop the otherwise implied yield expression.

If we went down the latter path, then:

gen = ((yield future) for future in list_of_futures)

and:

gen = (future for future in list_of_futures)

would be two different ways of writing the same thing.

The pay-off for allowing it would be that you could write things like:

gen = (f(yield future) for future in list_of_futures)

as a shorthand equivalent to:

def gen(list_of_futures=list_of_futures):
for future in list_of_futures:
f(yield future)

(Right now, you instead get "yield f(yield future)" as the innermost
statement, which probably isn't what you wanted)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Guido van Rossum wrote:
the extra scope is now part of the language definition. 
It can't be removed as a "bug fix".


Does anyone actually rely on the scope-ness of comprehensions
in any way other than the fact that it prevents local variable
leakage?

If not, probably nobody would notice if it were changed back.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Guido van Rossum wrote:


The debugger does stop at each iteration. It does see a local named ".0" 
I suppose there currently is no way for the debugger to map the variable 
names to what they are named in the source, right?


If the hidden local were named "a.0" where "a" is the original
name, mapping it back would be easy. It would also be easily
understood even if it weren't mapped back.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Serhiy Storchaka wrote:
Ivan explained that 
this function should be rough equivalent to


   def f():
   t = [(yield i) for i in range(3)]
   return (x for x in t)


This seems useless to me. It turns a lazy iterator
into an eager one, which is a gross violation of the
author's intent in using a generator expression.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Paul Moore wrote:

has anyone confirmed
why a function scope was considered necessary at the time of the
original implementation, but it's apparently not now?


At the time I got the impression that nobody wanted to
spend the time necessary to design and implement a subscope
mechanism. What's changed is that we now have someone
offering to do that.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Thu, Nov 23, 2017 at 9:06 AM, Serhiy Storchaka 
wrote:

> 23.11.17 18:08, Guido van Rossum пише:
>
>> This thread is still going over the speed limit. Don't commit anything
>> without my explicit approval.
>>
>
> I'm not going to write a single line of code while the decision about this
> issue is not made. This is not easy issue.
>

OK, I think Ivan at some point said he was waiting how your fix would look.


> A problem with dropping the "function-ness" of the comprehension by
>> renaming the variables (as Ivan/Serhiy's plan seems to be?) would be what
>> does it look like in the debugger -- can I still step through the loop and
>> print the values of expressions? Or do I have to know the "renamed" names?
>>
>
> Does the debugger supports stepping inside a comprehension? Isn't the
> whole comprehension a single logical line? It this is supported the the
> debugger should see a local variable with a strange name ".0". This is a
> leaked implementation detail. It wouldn't be surprised if details of other
> implementation will be leaked too in the debugger. The debugger can
> unmangle the inner variable names and hide local variables in the outer
> scope.
>

The debugger does stop at each iteration. It does see a local named ".0"
(which of course cannot be printed directly, only using vars()). I suppose
there currently is no way for the debugger to map the variable names to
what they are named in the source, right? (If you think this is a useful
thing to add, please open a separate issue.)


> And my mind boggles when considering a generator expression containing
>> yield that is returned from a function. I tried this and cannot say I
>> expected the outcome:
>>
>>def f():
>>return ((yield i) for i in range(3))
>>print(list(f()))
>>
>> In both Python 2 and Python 3 this prints
>>
>>[0, None, 1, None, 2, None]
>>
>> Even if there's a totally logical explanation for that, I still don't
>> like it, and I think yield in a comprehension should be banned. From this
>> it follows that we should also simply ban yield  from comprehensions.
>>
>
> This behavior doesn't look correct to me and Ivan. Ivan explained that
> this function should be rough equivalent to
>
>def f():
>t = [(yield i) for i in range(3)]
>return (x for x in t)
>
> which should be equivalent to
>
>def f():
>t = []
>for i in range(3):
>   t.append((yield i))
>return (x for x in t)
>
> and list(f()) should be [1, 2, 3].
>
> But while I know how to fix yield in comprehensions (just inline the code
> with some additions), I have no ideas how to fix yield in generators. Me
> and Ivan agreed that SyntaxError in this case is better than the current
> behavior. Maybe we will find how to implement the expected behavior.
>

I doubt that anyone who actually wrote that had a valid expectation for it
-- most likely they were coding by random modification or it was the result
of a lack of understanding of generator expressions in general. I think the
syntax error is the best option here.

I also think we should all stop posting for 24 hours as tempers have gotten
heated.

Happy Thanksgiving!

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

23.11.17 18:08, Guido van Rossum пише:
This thread is still going over the speed limit. Don't commit anything 
without my explicit approval.


I'm not going to write a single line of code while the decision about 
this issue is not made. This is not easy issue.


A problem with dropping the "function-ness" of the comprehension by 
renaming the variables (as Ivan/Serhiy's plan seems to be?) would be 
what does it look like in the debugger -- can I still step through the 
loop and print the values of expressions? Or do I have to know the 
"renamed" names?


Does the debugger supports stepping inside a comprehension? Isn't the 
whole comprehension a single logical line? It this is supported the the 
debugger should see a local variable with a strange name ".0". This is a 
leaked implementation detail. It wouldn't be surprised if details of 
other implementation will be leaked too in the debugger. The debugger 
can unmangle the inner variable names and hide local variables in the 
outer scope.


And my mind boggles when considering a generator expression containing 
yield that is returned from a function. I tried this and cannot say I 
expected the outcome:


   def f():
       return ((yield i) for i in range(3))
   print(list(f()))

In both Python 2 and Python 3 this prints

   [0, None, 1, None, 2, None]

Even if there's a totally logical explanation for that, I still don't 
like it, and I think yield in a comprehension should be banned. From 
this it follows that we should also simply ban yield  from comprehensions.


This behavior doesn't look correct to me and Ivan. Ivan explained that 
this function should be rough equivalent to


   def f():
   t = [(yield i) for i in range(3)]
   return (x for x in t)

which should be equivalent to

   def f():
   t = []
   for i in range(3):
  t.append((yield i))
   return (x for x in t)

and list(f()) should be [1, 2, 3].

But while I know how to fix yield in comprehensions (just inline the 
code with some additions), I have no ideas how to fix yield in 
generators. Me and Ivan agreed that SyntaxError in this case is better 
than the current behavior. Maybe we will find how to implement the 
expected behavior.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Brett Cannon]

I've now ended up in Guido's boat of needing a summary since I think this
thread has grown to cover whether yield should be allowed in
comprehensions, something about await in comprehensions, and now about
leaking the loop variable (or some implementation detail). IOW there seems
to be 3 separate discussions going on in a single thread.

Any chance we can get some clear threads going for each independent topic
to make it easier to discuss? I do have an opinion on all three topics (if
I understand the topics accurately ), but I don't want to contribute to
the confusion of the discussion by mentioning them here.

On Thu, Nov 23, 2017, 06:50 Ivan Levkivskyi,  wrote:

> On 23 November 2017 at 15:30, Paul Moore  wrote:
>
>> On 23 November 2017 at 14:24, Ivan Levkivskyi 
>> wrote:
>> >> My main concern is that comprehension is not equivalent to a for loop
>> >> for a specific reason - the scope issue. Has anyone looked back at the
>> >> original discussions to confirm *why* a function was used?
>> >>
>> >> My recollection:
>> >>
>> >> >>> i = 1
>> >> >>> a = [i for i in (1,2,3)]
>> >> >>> print(i)
>> >> 1
>> >>
>> >> Serihy's approach (and your described expansion) would have print(i)
>> >> return NameError.
>> >
>> >
>> > Absolutely no, it will still print 1. The internal implementation will
>> use
>> > unique ids internally (see https://bugs.python.org/issue10544 for
>> details).
>> >
>>
>> Ok, cool. My main point still applies though - has anyone confirmed
>> why a function scope was considered necessary at the time of the
>> original implementation, but it's apparently not now? I'm pretty sure
>> it was a deliberate choice, not an accident.
>
>
> From what Nick explained on b.p.o. I understand that this is closer to the
> "accident" definition.
> Also the original issue https://bugs.python.org/issue1660500 doesn't have
> any discussion of the implementation _strategy_.
> So I tried to dig the mailing list, in the latest Guido's message I have
> found
> https://mail.python.org/pipermail/python-3000/2006-December/005218.html
> he still likes the idea of unique hidden ids (like Serhiy proposes now)
> and no function scopes. After that there is Nick's message
> https://mail.python.org/pipermail/python-3000/2006-December/005229.html
> where he says that he still likes pseudo-scopes more.
> Then I lost the track of discussion.
>
> It may well be Nick's intentional decision (and it has its merits) but I
> am not sure it was a conscious consensus.
> Nick could probably add more. Also I propose to wait and see when Serhiy
> will show us his complete implementation.
>
> --
> Ivan
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

This thread is still going over the speed limit. Don't commit anything
without my explicit approval.

I know one thing for sure. The choice to make all comprehensions functions
was quite intentional (even though alternatives were also discussed) and
the extra scope is now part of the language definition. It can't be removed
as a "bug fix". One thing that can be done without a PEP is making yield
inside a comprehension a syntax error, since I don't think *that* was
considered when the function scopes were introduced.

A problem with dropping the "function-ness" of the comprehension by
renaming the variables (as Ivan/Serhiy's plan seems to be?) would be what
does it look like in the debugger -- can I still step through the loop and
print the values of expressions? Or do I have to know the "renamed" names?

And my mind boggles when considering a generator expression containing
yield that is returned from a function. I tried this and cannot say I
expected the outcome:

  def f():
  return ((yield i) for i in range(3))
  print(list(f()))

In both Python 2 and Python 3 this prints

  [0, None, 1, None, 2, None]

Even if there's a totally logical explanation for that, I still don't like
it, and I think yield in a comprehension should be banned. From this it
follows that we should also simply ban yield  from comprehensions.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ethan Furman]

On 11/23/2017 04:01 AM, Ivan Levkivskyi wrote:


Lets just forget about two SO questions and dozens people who up-voted it.


Questions/answers are routinely up-voted because they are well-written and/or informative, not just because somebody had 
a need for it or a use for the answer.


The SO question linked in the OP wasn't even about using the yield/yield-from in real code, just about timings and 
differences in byte code and whether or not they "worked" (no real-world use-case).


Hardly a clarion call for a fix.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 15:37, Ethan Furman  wrote:
> On 11/22/2017 11:51 PM, Sven R. Kunze wrote:
>
>> A "yield" within a comprehension is like a "return" in a comprehension. It
>> makes no sense at all.
>> Also a "yield" and a "return with value" is also rarely seen.
>>
>> Comprehensions build new objects, they are not for control flow, IMO.
>
>
> +1

That isn't a principle I've seen described explicitly before, but I
agree it makes sense to me. So +1 here as well.

And yes, I know this might seem to contradict my position that
comprehensions translate to loops, but remember that the translation
is a mental model, not an exact definition (to my way of thinking).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ethan Furman]

On 11/22/2017 11:51 PM, Sven R. Kunze wrote:


A "yield" within a comprehension is like a "return" in a comprehension. It 
makes no sense at all.
Also a "yield" and a "return with value" is also rarely seen.

Comprehensions build new objects, they are not for control flow, IMO.


+1

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 15:30, Paul Moore  wrote:

> On 23 November 2017 at 14:24, Ivan Levkivskyi 
> wrote:
> >> My main concern is that comprehension is not equivalent to a for loop
> >> for a specific reason - the scope issue. Has anyone looked back at the
> >> original discussions to confirm *why* a function was used?
> >>
> >> My recollection:
> >>
> >> >>> i = 1
> >> >>> a = [i for i in (1,2,3)]
> >> >>> print(i)
> >> 1
> >>
> >> Serihy's approach (and your described expansion) would have print(i)
> >> return NameError.
> >
> >
> > Absolutely no, it will still print 1. The internal implementation will
> use
> > unique ids internally (see https://bugs.python.org/issue10544 for
> details).
> >
>
> Ok, cool. My main point still applies though - has anyone confirmed
> why a function scope was considered necessary at the time of the
> original implementation, but it's apparently not now? I'm pretty sure
> it was a deliberate choice, not an accident.


>From what Nick explained on b.p.o. I understand that this is closer to the
"accident" definition.
Also the original issue https://bugs.python.org/issue1660500 doesn't have
any discussion of the implementation _strategy_.
So I tried to dig the mailing list, in the latest Guido's message I have
found
https://mail.python.org/pipermail/python-3000/2006-December/005218.html
he still likes the idea of unique hidden ids (like Serhiy proposes now) and
no function scopes. After that there is Nick's message
https://mail.python.org/pipermail/python-3000/2006-December/005229.html
where he says that he still likes pseudo-scopes more.
Then I lost the track of discussion.

It may well be Nick's intentional decision (and it has its merits) but I am
not sure it was a conscious consensus.
Nick could probably add more. Also I propose to wait and see when Serhiy
will show us his complete implementation.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

23.11.17 16:30, Paul Moore пише:

Ok, cool. My main point still applies though - has anyone confirmed
why a function scope was considered necessary at the time of the
original implementation, but it's apparently not now? I'm pretty sure
it was a deliberate choice, not an accident.


The implementation with an intermediate one-time function is just 
simpler. The one of purposes of Python 3 was simplifying the 
implementation, even at the cost of some performance penalty. I'm pretty 
sure the corner case with "yield" was just missed.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 14:24, Ivan Levkivskyi  wrote:
>> My main concern is that comprehension is not equivalent to a for loop
>> for a specific reason - the scope issue. Has anyone looked back at the
>> original discussions to confirm *why* a function was used?
>>
>> My recollection:
>>
>> >>> i = 1
>> >>> a = [i for i in (1,2,3)]
>> >>> print(i)
>> 1
>>
>> Serihy's approach (and your described expansion) would have print(i)
>> return NameError.
>
>
> Absolutely no, it will still print 1. The internal implementation will use
> unique ids internally (see https://bugs.python.org/issue10544 for details).
>

Ok, cool. My main point still applies though - has anyone confirmed
why a function scope was considered necessary at the time of the
original implementation, but it's apparently not now? I'm pretty sure
it was a deliberate choice, not an accident.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Chris Angelico]

On Fri, Nov 24, 2017 at 1:21 AM, Paul Moore  wrote:
> On 23 November 2017 at 13:04, Ivan Levkivskyi  wrote:
>> Let us forget for a moment about other problems and focus on this one: list
>> comprehension is currently not equivalent to a for-loop.
>> There are two options:
>> - Fix this, i.e. make comprehension equivalent to a for-loop even in edge
>> cases (Serhiy seems ready to do this)
>> - Prohibit all cases when they are not equivalent
>>
>> I still prefer option one. But I see your point, option two is also an
>> acceptable fix.
>> Note that there were not so many situations when some code became
>> SyntaxError later.
>> I don't see why this particular case qualifies for such a radical measure as
>> an exception to syntactic rules,
>> instead of just fixing it (sorry Nick :-)
>
> My main concern is that comprehension is not equivalent to a for loop
> for a specific reason - the scope issue. Has anyone looked back at the
> original discussions to confirm *why* a function was used?
>
> My recollection:
>
 i = 1
 a = [i for i in (1,2,3)]
 print(i)
> 1
>
> Serihy's approach (and your described expansion) would have print(i)
> return NameError.
>
> So - do we actually have a proposal to avoid the implied function that
> *doesn't* break this example? I'm pretty sure this was a real-life
> issue at the time we switched to the current implementation.

A while back I had a POC patch that made "with EXPR as NAME:" create a
new subscope with NAME in it, such that the variable actually
disappeared at the end of the 'with' block. Should I try to track that
down and adapt the technique to comprehensions? The subscope shadows
names exactly the way a nested function does, but it's all within the
same function.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 15:21, Paul Moore  wrote:

> On 23 November 2017 at 13:04, Ivan Levkivskyi 
> wrote:
> > Let us forget for a moment about other problems and focus on this one:
> list
> > comprehension is currently not equivalent to a for-loop.
> > There are two options:
> > - Fix this, i.e. make comprehension equivalent to a for-loop even in edge
> > cases (Serhiy seems ready to do this)
> > - Prohibit all cases when they are not equivalent
> >
> > I still prefer option one. But I see your point, option two is also an
> > acceptable fix.
> > Note that there were not so many situations when some code became
> > SyntaxError later.
> > I don't see why this particular case qualifies for such a radical
> measure as
> > an exception to syntactic rules,
> > instead of just fixing it (sorry Nick :-)
>
> My main concern is that comprehension is not equivalent to a for loop
> for a specific reason - the scope issue. Has anyone looked back at the
> original discussions to confirm *why* a function was used?
>
> My recollection:
>
> >>> i = 1
> >>> a = [i for i in (1,2,3)]
> >>> print(i)
> 1
>
> Serihy's approach (and your described expansion) would have print(i)
> return NameError.
>

Absolutely no, it will still print 1. The internal implementation will use
unique ids internally (see https://bugs.python.org/issue10544 for details).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 13:04, Ivan Levkivskyi  wrote:
> Let us forget for a moment about other problems and focus on this one: list
> comprehension is currently not equivalent to a for-loop.
> There are two options:
> - Fix this, i.e. make comprehension equivalent to a for-loop even in edge
> cases (Serhiy seems ready to do this)
> - Prohibit all cases when they are not equivalent
>
> I still prefer option one. But I see your point, option two is also an
> acceptable fix.
> Note that there were not so many situations when some code became
> SyntaxError later.
> I don't see why this particular case qualifies for such a radical measure as
> an exception to syntactic rules,
> instead of just fixing it (sorry Nick :-)

My main concern is that comprehension is not equivalent to a for loop
for a specific reason - the scope issue. Has anyone looked back at the
original discussions to confirm *why* a function was used?

My recollection:

>>> i = 1
>>> a = [i for i in (1,2,3)]
>>> print(i)
1

Serihy's approach (and your described expansion) would have print(i)
return NameError.

So - do we actually have a proposal to avoid the implied function that
*doesn't* break this example? I'm pretty sure this was a real-life
issue at the time we switched to the current implementation.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Thu, 23 Nov 2017 14:54:27 +0200
Serhiy Storchaka  wrote:

> 23.11.17 14:30, Antoine Pitrou пише:
> > On Thu, 23 Nov 2017 14:17:32 +0200
> > Serhiy Storchaka  wrote:  
> >>
> >> I used the "yield" statement, but I never used the "yield" expressions.
> >> And I can't found examples. Could you please present a real-world use
> >> case for the "yield" (not "yield from") expression?  
> > 
> > Of course I can.  "yield" expressions are important for writing
> > Python 2-compatible asynchronous code while avoiding callback hell:
> > 
> > See e.g. http://www.tornadoweb.org/en/stable/gen.html
> > or https://jdb.github.io/concurrent/smartpython.html
> > 
> > There are tons of real-world code written using this scheme (as opposed
> > to almost no real-world code, even Python 2-only, using "yield" in
> > comprehensions or generation expressions).  
> 
> Thank you. The tornado examples contain the following equivalence code 
> for `results = yield multi(list_of_futures)`:
> 
>  results = []
>  for future in list_of_futures:
>  results.append(yield future)
> 
> Couldn't this by written as `results = [(yield future) for future in 
> list_of_futures]`?

See my answer to Ivan above.  The code isn't actually equivalent :-)
But, yes, this construct *could* be useful if you wanted to schedule
futures serially (as opposed to in parallel).

However, since it doesn't work on Python 3.x, and the main reason to
use "yield" coroutines (even with Tornado) instead of "async/await" is
for compatibility, solving the "yield in a comprehension" problem in
3.7 wouldn't make things any better IMO.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 13:45, Paul Moore  wrote:

> On 23 November 2017 at 12:28, Ivan Levkivskyi 
> wrote:
> > On 23 November 2017 at 13:11, Paul Moore  wrote:
> >>
> >> On 23 November 2017 at 12:01, Ivan Levkivskyi 
> >> wrote:
> >>
> >> > "I don't use it, therefore it is not needed"  is a great argument,
> >> > thanks.
> >> > Lets just forget about two SO questions and dozens people who up-voted
> >> > it.
> >> > Do you use async comprehensions? If not, then we don't need them
> either.
> >>
> >> For those of us trying to keep up with the discussion who don't have
> >> time to chase the various references, and in the interest of keeping
> >> the discussion in one place, can you summarise the real-world use
> >> cases from the SO questions here? (I assume they are real world cases,
> >> and not just theoretical questions)
> [...]
> >
> > My understanding is that none of the case is _pressing_, since they all
> > start with a for-loop, but
> > following this logic comprehensions themselves are not needed.
> Nevertheless
> > people use them because they like it.
> > The problem in all four cases is that they got hard to debug problem,
> since
> > calling `f()` returns a generator,
> > just not the one they would expect.
>
> OK, thanks. I can see why someone would want to do this. However, it
> seems to me that the problem (a hard to debug error) could be solved
> by disallowing yield in comprehensions and generator expressions
> (giving an *easy* to debug error). I don't think the above is a
> compelling argument that we have to support the one-line form. If
> there was a non-trivial body of actual user code that uses the loop
> form, which would be substantially improved by being able to use
> comprehensions, that would be different. To put it another way, the
> example you gave is still artificial. The second link is a real use
> case, but as you say seems to be more a question about "why did this
> not work as I expected" which could be solved with a SyntaxError
> saying "yield expression not allowed in comprehensions".
>

The level of "artificialness" is quite subjective, this is rather matter of
taste (see the tornado example).
Let us forget for a moment about other problems and focus on this one: list
comprehension is currently not equivalent to a for-loop.
There are two options:
- Fix this, i.e. make comprehension equivalent to a for-loop even in edge
cases (Serhiy seems ready to do this)
- Prohibit all cases when they are not equivalent

I still prefer option one. But I see your point, option two is also an
acceptable fix.
Note that there were not so many situations when some code became
SyntaxError later.
I don't see why this particular case qualifies for such a radical measure
as an exception to syntactic rules,
instead of just fixing it (sorry Nick :-)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

23.11.17 14:30, Antoine Pitrou пише:

On Thu, 23 Nov 2017 14:17:32 +0200
Serhiy Storchaka  wrote:


I used the "yield" statement, but I never used the "yield" expressions.
And I can't found examples. Could you please present a real-world use
case for the "yield" (not "yield from") expression?


Of course I can.  "yield" expressions are important for writing
Python 2-compatible asynchronous code while avoiding callback hell:

See e.g. http://www.tornadoweb.org/en/stable/gen.html
or https://jdb.github.io/concurrent/smartpython.html

There are tons of real-world code written using this scheme (as opposed
to almost no real-world code, even Python 2-only, using "yield" in
comprehensions or generation expressions).


Thank you. The tornado examples contain the following equivalence code 
for `results = yield multi(list_of_futures)`:


results = []
for future in list_of_futures:
results.append(yield future)

Couldn't this by written as `results = [(yield future) for future in 
list_of_futures]`?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 12:42, Ivan Levkivskyi  wrote:
>> See e.g. http://www.tornadoweb.org/en/stable/gen.html
>>
>
> Great, so I open this page and see this code:
>
> results = []
> for future in list_of_futures:
> results.append(yield future)
>
> Interesting, why don't they use a comprehension for this and instead need to
> invent a whole `tornado.gen.multi` function?

Because yield expressions in comprehensions are difficult to
understand, and the loop form is easy to understand? :-) (Certainly I
didn't find the explanation in that page confusing, I don't know if
I'd have found a comprehension form confusing, but I suspect I might
have...)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

Le 23/11/2017 à 13:42, Ivan Levkivskyi a écrit :
> 
> Great, so I open this page and see this code:
> 
> results = []
> for future in list_of_futures:
>     results.append(yield future)
> 
> Interesting, why don't they use a comprehension for this and instead
> need to invent a whole `tornado.gen.multi` function?

1) because it schedules the yielded coroutines in parallel (the "for"
loop isn't strictly equivalent, as AFAIU it would schedule the
coroutines serially)

2) because it accepts an optional argument to quiet some exceptions

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 12:28, Ivan Levkivskyi  wrote:
> On 23 November 2017 at 13:11, Paul Moore  wrote:
>>
>> On 23 November 2017 at 12:01, Ivan Levkivskyi 
>> wrote:
>>
>> > "I don't use it, therefore it is not needed"  is a great argument,
>> > thanks.
>> > Lets just forget about two SO questions and dozens people who up-voted
>> > it.
>> > Do you use async comprehensions? If not, then we don't need them either.
>>
>> For those of us trying to keep up with the discussion who don't have
>> time to chase the various references, and in the interest of keeping
>> the discussion in one place, can you summarise the real-world use
>> cases from the SO questions here? (I assume they are real world cases,
>> and not just theoretical questions)
>
>
> OK, here are the links:
>
> https://stackoverflow.com/questions/45190729/differences-between-generator-comprehension-expressions
> https://stackoverflow.com/questions/29334054/why-am-i-getting-different-results-when-using-a-list-comprehension-with-coroutin
> https://bugs.python.org/issue10544
> https://bugs.python.org/issue3267
>
> In all four cases pattern is the same, people were trying to refactor
> something like this:
>
> def f():
> res = []
> for x in y:
> r = yield x
> res.append(r)
> return res
>
> into something like this:
>
> def f():
> return [(yield x) for x in y]
>
> My understanding is that none of the case is _pressing_, since they all
> start with a for-loop, but
> following this logic comprehensions themselves are not needed. Nevertheless
> people use them because they like it.
> The problem in all four cases is that they got hard to debug problem, since
> calling `f()` returns a generator,
> just not the one they would expect.

OK, thanks. I can see why someone would want to do this. However, it
seems to me that the problem (a hard to debug error) could be solved
by disallowing yield in comprehensions and generator expressions
(giving an *easy* to debug error). I don't think the above is a
compelling argument that we have to support the one-line form. If
there was a non-trivial body of actual user code that uses the loop
form, which would be substantially improved by being able to use
comprehensions, that would be different. To put it another way, the
example you gave is still artificial. The second link is a real use
case, but as you say seems to be more a question about "why did this
not work as I expected" which could be solved with a SyntaxError
saying "yield expression not allowed in comprehensions".

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 13:30, Antoine Pitrou  wrote:

> On Thu, 23 Nov 2017 14:17:32 +0200
> Serhiy Storchaka  wrote:
> >
> > I used the "yield" statement, but I never used the "yield" expressions.
> > And I can't found examples. Could you please present a real-world use
> > case for the "yield" (not "yield from") expression?
>
> Of course I can.  "yield" expressions are important for writing
> Python 2-compatible asynchronous code while avoiding callback hell:
>
> See e.g. http://www.tornadoweb.org/en/stable/gen.html
>
> 


Great, so I open this page and see this code:

results = []
for future in list_of_futures:
results.append(yield future)

Interesting, why don't they use a comprehension for this and instead need
to invent a whole `tornado.gen.multi` function?

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 12:01, Ivan Levkivskyi  wrote:

> "I don't use it, therefore it is not needed"  is a great argument, thanks.
> Lets just forget about two SO questions and dozens people who up-voted it.
> Do you use async comprehensions? If not, then we don't need them either.

For those of us trying to keep up with the discussion who don't have
time to chase the various references, and in the interest of keeping
the discussion in one place, can you summarise the real-world use
cases from the SO questions here? (I assume they are real world cases,
and not just theoretical questions)

Thanks,
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Thu, 23 Nov 2017 14:17:32 +0200
Serhiy Storchaka  wrote:
> 
> I used the "yield" statement, but I never used the "yield" expressions. 
> And I can't found examples. Could you please present a real-world use 
> case for the "yield" (not "yield from") expression?

Of course I can.  "yield" expressions are important for writing
Python 2-compatible asynchronous code while avoiding callback hell:

See e.g. http://www.tornadoweb.org/en/stable/gen.html
or https://jdb.github.io/concurrent/smartpython.html

There are tons of real-world code written using this scheme (as opposed
to almost no real-world code, even Python 2-only, using "yield" in
comprehensions or generation expressions).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 13:11, Paul Moore  wrote:

> On 23 November 2017 at 12:01, Ivan Levkivskyi 
> wrote:
>
> > "I don't use it, therefore it is not needed"  is a great argument,
> thanks.
> > Lets just forget about two SO questions and dozens people who up-voted
> it.
> > Do you use async comprehensions? If not, then we don't need them either.
>
> For those of us trying to keep up with the discussion who don't have
> time to chase the various references, and in the interest of keeping
> the discussion in one place, can you summarise the real-world use
> cases from the SO questions here? (I assume they are real world cases,
> and not just theoretical questions)
>

OK, here are the links:

https://stackoverflow.com/questions/45190729/differences-between-generator-comprehension-expressions
https://stackoverflow.com/questions/29334054/why-am-i-getting-different-results-when-using-a-list-comprehension-with-coroutin
https://bugs.python.org/issue10544
https://bugs.python.org/issue3267

In all four cases pattern is the same, people were trying to refactor
something like this:

def f():
res = []
for x in y:
r = yield x
res.append(r)
return res

into something like this:

def f():
return [(yield x) for x in y]

My understanding is that none of the case is _pressing_, since they all
start with a for-loop, but
following this logic comprehensions themselves are not needed. Nevertheless
people use them because they like it.
The problem in all four cases is that they got hard to debug problem, since
calling `f()` returns a generator,
just not the one they would expect.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

23.11.17 13:49, Antoine Pitrou пише:

I'm still in favour of deprecating and then disallowing.


We could disallow it without deprecation. The current behavior 
definitely is wrong, nobody should depend on it. It should be either 
fixed or disallowed.



Nobody seems
to have presented a real-world use case that is made significantly
easier by trying to "fix" the current behaviour (as opposed to
spelling the loop explicitly).  I do asynchronous programming using
"yield" every day in may job (because of compatibility requirements
with Python 2) and I've never had once the need to write a "yield"
inside a comprehension or generator expression.


I used the "yield" statement, but I never used the "yield" expressions. 
And I can't found examples. Could you please present a real-world use 
case for the "yield" (not "yield from") expression?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

Le 23/11/2017 à 13:01, Ivan Levkivskyi a écrit :
> 
> "I don't use it, therefore it is not needed"  is a great argument, thanks.

This is just a data point.  Some people seem to think that the construct
is useful for asynchronous programming.  In my experience it isn't.
YMMV, etc.

> Lets just forget about two SO questions and dozens people who up-voted it.

Just because someone asks a question doesn't mean they have a pressing
use case.  People are curious and will try constructs just for the sake
it.  Today I looked up the Wikipedia page for Gregorian chant.  Did I
have a use case for it?  No, I was just curious.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 12:49, Antoine Pitrou  wrote:

> On Thu, 23 Nov 2017 12:39:46 +0100
> Ivan Levkivskyi  wrote:
> >
> > Also I think it makes sense to keep discussion in one place, i.e. either
> > here xor at https://bugs.python.org/issue10544
>
> The bug tracker can be used for implementation discussions, but general
> language design decisions (such as whether to allow or not a certain
> construct) should take place on python-dev.
>
> I'm still in favour of deprecating and then disallowing.  Nobody seems
> to have presented a real-world use case that is made significantly
> easier by trying to "fix" the current behaviour (as opposed to
> spelling the loop explicitly).  I do asynchronous programming using
> "yield" every day in may job (because of compatibility requirements
> with Python 2) and I've never had once the need to write a "yield"
> inside a comprehension or generator expression.
>
>
"I don't use it, therefore it is not needed"  is a great argument, thanks.
Lets just forget about two SO questions and dozens people who up-voted it.
Do you use async comprehensions? If not, then we don't need them either.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Thu, 23 Nov 2017 09:50:27 +
Paul Moore  wrote:
> On 23 November 2017 at 09:14, Steve Holden  wrote:
> > I would urge developers, in their improvements to the language to support
> > asynchronous programming, to bear in mind that this is (currently) a
> > minority use case. Why the rush to set complex semantics in stone?  
> 
> +1
> 
> Also, given that languages like C# have similar async/await
> functionality, I'd be interested to know how they address questions
> like this. If they have a parallel, we should probably follow it. If
> they don't that would be further indication that no-one has much
> experience of the "best answers" yet, and caution is indicated.

This discussion isn't about async/wait or asynchronous programming.

It's about "yield" (which used to be the standard for asynchronous
programming before async/await, but isn't anymore).  The fact that
"await" is now the standard still weakens the case for "yield" inside
comprehensions and generator expressions.

As someone who does asynchronous programming daily using "yield"
(because of compatibility requirements with Python 2), I don't think
I've even tried to use "yield" in a comprehension or generator
expression.  The use case doesn't seem to exist.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Thu, 23 Nov 2017 12:39:46 +0100
Ivan Levkivskyi  wrote:
> 
> Also I think it makes sense to keep discussion in one place, i.e. either
> here xor at https://bugs.python.org/issue10544

The bug tracker can be used for implementation discussions, but general
language design decisions (such as whether to allow or not a certain
construct) should take place on python-dev.

I'm still in favour of deprecating and then disallowing.  Nobody seems
to have presented a real-world use case that is made significantly
easier by trying to "fix" the current behaviour (as opposed to
spelling the loop explicitly).  I do asynchronous programming using
"yield" every day in may job (because of compatibility requirements
with Python 2) and I've never had once the need to write a "yield"
inside a comprehension or generator expression.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 10:50, Paul Moore  wrote:

> On 23 November 2017 at 09:14, Steve Holden  wrote:
> > I would urge developers, in their improvements to the language to support
> > asynchronous programming, to bear in mind that this is (currently) a
> > minority use case. Why the rush to set complex semantics in stone?
>
> +1
>
> Also, given that languages like C# have similar async/await
> functionality, I'd be interested to know how they address questions
> like this. If they have a parallel, we should probably follow it. If
> they don't that would be further indication that no-one has much
> experience of the "best answers" yet, and caution is indicated.
>
>
Keeping this open for indefinite time is also not a good option.
Note that the issue about `yield` in comprehensions
https://bugs.python.org/issue10544 is 7 (seven) years old.
I don't say that we should fix it _right now_, but I think it makes sense
to spend some time and finally resolve it.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 12:38, Ivan Levkivskyi  wrote:

> On 23 November 2017 at 11:55, Nick Coghlan  wrote:
>
>> On 23 November 2017 at 18:11, Greg Ewing 
>> wrote:
>>
>>> Ivan Levkivskyi wrote:
>>>
 "People sometimes want to refactor for-loops containing `yield` into a
 comprehension but that doesn't work (particularly because of the hidden
 function scope) - lets make it a SyntaxError"

>>>
>>> Personally I'd be fine with removing the implicit function
>>> scope from comprehensions and allowing yield in them, since
>>> the semantics of that are clear.
>>>
>>
>> People keep saying this, but seriously, those semantics aren't clear at
>> all once you actually start trying to implement it.
>>
>>
> If Serhiy will implement his idea (emitting for-loop bytecode inside a
> try-finally), then I see no problems accepting it as a fix.
>

Also I think it makes sense to keep discussion in one place, i.e. either
here xor at https://bugs.python.org/issue10544

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 11:55, Nick Coghlan  wrote:

> On 23 November 2017 at 18:11, Greg Ewing 
> wrote:
>
>> Ivan Levkivskyi wrote:
>>
>>> "People sometimes want to refactor for-loops containing `yield` into a
>>> comprehension but that doesn't work (particularly because of the hidden
>>> function scope) - lets make it a SyntaxError"
>>>
>>
>> Personally I'd be fine with removing the implicit function
>> scope from comprehensions and allowing yield in them, since
>> the semantics of that are clear.
>>
>
> People keep saying this, but seriously, those semantics aren't clear at
> all once you actually start trying to implement it.
>
>
If Serhiy will implement his idea (emitting for-loop bytecode inside a
try-finally), then I see no problems accepting it as a fix.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 23 November 2017 at 18:11, Greg Ewing 
wrote:

> Ivan Levkivskyi wrote:
>
>> "People sometimes want to refactor for-loops containing `yield` into a
>> comprehension but that doesn't work (particularly because of the hidden
>> function scope) - lets make it a SyntaxError"
>>
>
> Personally I'd be fine with removing the implicit function
> scope from comprehensions and allowing yield in them, since
> the semantics of that are clear.
>

People keep saying this, but seriously, those semantics aren't clear at all
once you actually start trying to implement it.

Yes, they're obvious in simple cases, but it isn't the simple cases that
are the problem.

Instead, things start getting hard once you're dealing with:

- unpacking to multiple variable names
- nested loops in the comprehension
- lexical closures inside the comprehension (e.g. lambda expressions,
comprehensions inside comprehensions)

Hence the approach we ended up going with for
https://bugs.python.org/issue1660500, which was to use a real function
scope that already handled all of those potential problems in a well
defined way.

Technically we *could* define new answers to all of those situations, but
then we're stuck explaining to everyone what those new behaviours actually
are, and I think that will actually be harder than the status quo, where we
only have to explain why these implicit scopes act much the same way that
"lambda: await expr" and "lambda: yield expr" do.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 09:14, Steve Holden  wrote:
> I would urge developers, in their improvements to the language to support
> asynchronous programming, to bear in mind that this is (currently) a
> minority use case. Why the rush to set complex semantics in stone?

+1

Also, given that languages like C# have similar async/await
functionality, I'd be interested to know how they address questions
like this. If they have a parallel, we should probably follow it. If
they don't that would be further indication that no-one has much
experience of the "best answers" yet, and caution is indicated.

BTW, I'm assuming that the end goal is for async to be a natural and
fully-integrated part of the language, no more "minority use" than
generators, or context managers. Assuming that's the case, I think
that keeping a very careful eye on how intuitive async feels to
non-specialists is crucial (so thanks to Ivan and Yury for taking the
time to respond to this discussion).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Steve Holden]

On Wed, Nov 22, 2017 at 8:48 PM, Sven R. Kunze  wrote:

> Isn't yield like a return?
>
​Enough like it to make a good case, I'd say.​


> A return in a list/dict/set comprehension makes no sense to me.
>
​Nor me, nor the vast majority of instance. But nowadays yield is more of a
synchronisation point. If something is valid syntax we should presumably
like to have defined semantics.

> So, +1 on SyntaxError from me too.
>
​I'd tend to agree. This would give more time to discuss the intended
semantics: giving it meaning later might be a more cautious approach that
would allow decisions to be made in the light of further experience.

I would urge developers, in their ​improvements to the language to support
asynchronous programming, to bear in mind that this is (currently) a
minority use case. Why the rush to set complex semantics in stone?

>
regards
 Steve​


> Cheers.
>
> On 22.11.2017 21:29, David Mertz wrote:
>
> Inasmuch as I get to opine, I'm +1 on SyntaxError. There is no behavior
> for that spelling that I would find intuitive or easy to explain to
> students. And as far as I can tell, the ONLY time anything has ever been
> spelled that way is in comments saying "look at this weird edge case
> behavior in Python."
>
> On Nov 22, 2017 10:57 AM, "Jelle Zijlstra" 
> wrote:
>
>
>
> 2017-11-22 9:58 GMT-08:00 Guido van Rossum :
>
>> Wow, 44 messages in 4 hours. That must be some kind of record.
>>
>> If/when there's an action item, can someone summarize for me?
>>
>> The main disagreement seems to be about what this code should do:
>
> g = [(yield i) for i in range(3)]
>
> Currently, this makes `g` into a generator, not a list. Everybody seems to
> agree this is nonintuitive and should be changed.
>
> One proposal is to make it so `g` gets assigned a list, and the `yield`
> happens in the enclosing scope (so the enclosing function would have to be
> a generator). This was the way things worked in Python 2, I believe.
>
> Another proposal is to make this code a syntax error, because it's
> confusing either way. (For what it's worth, that would be my preference.)
>
> There is related discussion about the semantics of list comprehensions
> versus calling list() on a generator expression, and of async semantics,
> but I don't think there's any clear point of action there.
>
>
>> --
>> --Guido van Rossum (python.org/~guido )
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jelle.zij
>> lstra%40gmail.com
>>
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mertz%
> 40gnosis.cx
>
>
>
>
> ___
> Python-Dev mailing 
> listPython-Dev@python.orghttps://mail.python.org/mailman/listinfo/python-dev
>
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> steve%40holdenweb.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 09:17, Greg Ewing 
wrote:

> Ivan Levkivskyi wrote:
>
>> "People sometimes want to refactor for-loops containing `yield` into a
>> comprehension
>>
>
> By the way, do we have any real-life examples of people wanting to
> do this? It might help us decide what the semantics should be.
>
>
Yes, there are two SO questions in two first posts here, also there are
some b.p.o. issues. It looks like in all case people expect:

def f():
return [(yield i) for i in range(3)]

to be roughly equivalent to

def f():
res = []
for i in range(3):
r = yield i
res.append(r)
return res

See Serhiy's original post for more detailed proposed semantic equivalence.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 09:11, Greg Ewing 
wrote:

> Ivan Levkivskyi wrote:
>
>> "People sometimes want to refactor for-loops containing `yield` into a
>> comprehension but that doesn't work (particularly because of the hidden
>> function scope) - lets make it a SyntaxError"
>>
>
> Personally I'd be fine with removing the implicit function
> scope from comprehensions and allowing yield in them, since
> the semantics of that are clear.
>
> But I don't see a way to do anything equivalent with
> generator expressions. Since the current effect of
> yield in a generator expression is pretty useless,
> it seems best just to disallow it.
>
> That means a list comprehension won't be equivalent
> to list(generator_expression) in all cases, but I
> don't think there's any great need for it to be.
>

I am also fine with this. Generator expressions are indeed less clear.
Also different people have different mental model about them (especially re
implicit scope and equivalence to comprehensions).

On the contrary, vast majority agrees that comprehensions are just
for-loops without leaking variables.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 09:15, Greg Ewing 
wrote:

> Ivan Levkivskyi wrote:
>
>> On 23 November 2017 at 05:44, Greg Ewing > > wrote:
>>
>>def g():
>>   return ((yield i) for i in range(10))
>>
>>
>> I think this code should be just equivalent to this code
>>
>> def g():
>> temp = [(yield i) for i in range(10)]
>> return (v for v in temp)
>>
>
> But then you get a non-lazy iterable, which defeats the
> purpose of using a generator expression -- you might as
> well have used a comprehension to begin with.
>
>
This could be just a semantic equivalence (mental model), not how it should
be internally implemented.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Ivan Levkivskyi wrote:
"People sometimes want to refactor for-loops containing `yield` into a 
comprehension


By the way, do we have any real-life examples of people wanting to
do this? It might help us decide what the semantics should be.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Ivan Levkivskyi wrote:
On 23 November 2017 at 05:44, Greg Ewing > wrote:


   def g():
  return ((yield i) for i in range(10))


I think this code should be just equivalent to this code

def g():
temp = [(yield i) for i in range(10)]
return (v for v in temp)


But then you get a non-lazy iterable, which defeats the
purpose of using a generator expression -- you might as
well have used a comprehension to begin with.

--
Greg

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Ivan Levkivskyi wrote:
"People sometimes want to refactor for-loops containing `yield` into a 
comprehension but that doesn't work (particularly because of the hidden 
function scope) - lets make it a SyntaxError"


Personally I'd be fine with removing the implicit function
scope from comprehensions and allowing yield in them, since
the semantics of that are clear.

But I don't see a way to do anything equivalent with
generator expressions. Since the current effect of
yield in a generator expression is pretty useless,
it seems best just to disallow it.

That means a list comprehension won't be equivalent
to list(generator_expression) in all cases, but I
don't think there's any great need for it to be.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Sven R. Kunze]

On 23.11.2017 08:38, Ivan Levkivskyi wrote:

I think this code should be just equivalent to this code

    def g():
    temp = [(yield i) for i in range(10)]
    return (v for v in temp)

Semantics of the comprehension should be clear here (just an 
equivalent for-loop without name leaking)


Excuse me if I disagree here. If I were to understand this in real-world 
code, I cannot imagine what will happen here.


A "yield" within a comprehension is like a "return" in a comprehension. 
It makes no sense at all.

Also a "yield" and a "return with value" is also rarely seen.

Comprehensions build new objects, they are not for control flow, IMO.

Cheers,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 05:44, Greg Ewing 
wrote:

> Ivan Levkivskyi wrote:
>
>> The key idea is that neither comprehensions nor generator expressions
>> should create a function scope surrounding the `expr`
>>
>
> I don't see how you can avoid an implicit function scope in
> the case of a generator expression, though. And I can't see
> how to make yield in a generator expression do anything
> sensible.
>
> Consider this:
>
>def g():
>   return ((yield i) for i in range(10))
>
> Presumably the yield should turn g into a generator, but...
> then what? My brain is hurting trying to figure out what
> it should do.
>
>
I think this code should be just equivalent to this code

def g():
temp = [(yield i) for i in range(10)]
return (v for v in temp)

Semantics of the comprehension should be clear here (just an equivalent
for-loop without name leaking)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 01:00, Yury Selivanov 
wrote:

> On Wed, Nov 22, 2017 at 6:46 PM, Ivan Levkivskyi 
> wrote:
> [..]
> > Just found another example of intuitive behaviour:
> >
>  async def f():
> > ... for i in range(3):
> > ... yield i
> > ...
>  async def g():
> > ... return [(yield i) async for i in f()]
> > ...
>  g().send(None)
> > Traceback (most recent call last):
> >   File "", line 1, in 
> >   File "", line 2, in g
> > TypeError: object async_generator can't be used in 'await' expression
> >
> > of course it is obvious for anyone who writes async code, but anyway an
> > interesting example.
>
> I wouldn't say that it's obvious to anyone...
>
> I think this thread has started to discuss the use of 'yield'
> expression in comprehensions, and the outcome of the discussion is
> that everyone thinks that we should deprecate that syntax in 3.7,
> remove in 3.8.  Let's start with that? :)


I am not sure everyone agrees with this. My main obstacle is following,
consider motivation for the `await` part of PEP 530
which is in my understanding is roughly like this:

"People sometimes want to refactor for-loops containing `await` into a
comprehension but that doesn't work (particularly because of the hidden
function scope) - lets fix this"

I don't see how this compare to:

"People sometimes want to refactor for-loops containing `yield` into a
comprehension but that doesn't work (particularly because of the hidden
function scope) - lets make it a SyntaxError"

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Tres Seaver]

On 11/22/2017 07:00 PM, Yury Selivanov wrote:

> I wouldn't say that it's obvious to anyone...
> 
> I think this thread has started to discuss the use of 'yield'
> expression in comprehensions, and the outcome of the discussion is
> that everyone thinks that we should deprecate that syntax in 3.7,
> remove in 3.8.  Let's start with that? :)

You guys are no fun:  such a change would remove at least one evil
"bwah-ha-ha" cackle from David Beazley's next PyCon talk. :)


Tres.
-- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Stephen J. Turnbull]

Greg Ewing writes:

 > Consider this:
 > 
 > def g():
 >return ((yield i) for i in range(10))
 > 
 > Presumably the yield should turn g into a generator, but...
 > then what? My brain is hurting trying to figure out what
 > it should do.

I don't understand why you presume that.  The generator expression
doesn't do that anywhere else.  My model is that implicitly the
generator expression is creating a function that becomes a generator
factory, which is implicitly called to return the iterable generator
object, which contains the yield.  Because the call takes place
implicitly = at compile time, all the containing function "sees" is an
iterable (which happens to be a generator object).  "Look Ma, no
yields left!"  And then g returns the generator object.

What am I missing?

In other words, g above is equivalent to

def g():
def _g():
for i in range(10):
# the outer yield is the usual implicit yield from the
# expansion of the generator expression, and the inner
# yield is explicit in your code.
yield (yield i)
return _g()

(modulo some issues of leaking identifiers).  I have not figured out
why either your g or my g does what it does, but they do the same
thing.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 23 November 2017 at 15:33, Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> However, my model of comprehensions is exactly a for loop that appends
> to an empty list repeatedly, but doesn't leak iteration variables.


Not since Python 3.0. Instead, they create a nested function, the same way
generator expressions do (which is why the name resolution semantics are
now identical between the two cases).

The differences in structure between the four cases (genexp, list/set/dict
comprehensions) then relate mainly to what the innermost loop does:

result.append(expr) # list comp
result.add(expr) # set comp
result[k] = v # dict comp
yield expr # genexp

Thus, when the expression itself is a yield expression, you get:

result.append(yield expr) # list comp
result.add(yield expr) # set comp
result[k] = (yield v) # dict comp
yield (yield expr) # genexp

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Stephen J. Turnbull]

Paul Moore writes:

 > 1. List comprehensions expand into nested for/if statements in the
 > "obvious" way - with an empty list created to start and append used to
 > add items to it.
 >1a. Variables introduced in the comprehension don't "leak" (see below).
 > 2. Generator expressions expand into generators with the same "nested
 > loop" behaviour, and a yield of the generated value.
 > 3. List comprehensions are the same as list(the equivalent generator
 > expression).

I'm a little late to this discussion, but I don't see how 3 can be
true if 1 and 2 are.

Because I believe my model of generator expressions is pretty simple,
I'll present it here:

- the generator expression in 2 and 3 implicitly creates a generator
  function containing the usual loop
  - the point being that it will "capture" all of the yields (implicit
*and* explicit) in the generator expression
- then implicitly invokes it, 
- passing the (iterable) generator object returned to the containing
  expression.

"Look Ma, no yields (or generator functions) left!"

However, my model of comprehensions is exactly a for loop that appends
to an empty list repeatedly, but doesn't leak iteration variables.  So
a yield in a comprehension turns the function *containing* the
comprehension (which may or may not exist) into a generator function.
In other words, I don't agree with Ethan that a yield inside a list
comprehension should not affect the generator-ness of the containing
function.  What would this mean under that condition:

[f(x) for x in (yield iterable)]

then?  IOW,

x = ((yield i) for i in iterable)

IS valid syntax at top level, while

x = [(yield i) for i in iterable]

IS NOT valid syntax at top level given those semantics.  The latter
"works" in Python 3.6;

>>> for i in [(yield i) for i in (1, 2, 3)]:
...  i
... 
1
2
3

though I think it should be a syntax error, and "bare" yield does not
"work":

>>> i = 1
>>> yield i
  File "", line 1
SyntaxError: 'yield' outside function
>>> (yield i)
  File "", line 1
SyntaxError: 'yield' outside function

FWIW, that's the way I'd want it, and the way I've always understood
comprehensions and generator expressions.  I think this is consistent
with Yuri's, Serhiy's, and Ivan's claims, but I admit I'm not sure.
Of course the compiler need not create a generator function and invoke
it, but the resulting bytecode should be the same as if it did.

This means the semantics of [FOR-EXPRESSION] and [(FOR-EXPRESSION)]
should differ in the same way.

Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Chris Angelico]

On Thu, Nov 23, 2017 at 3:36 PM, Greg Ewing  wrote:
> Paul Moore wrote:
>>
>> 3. List comprehensions are the same as list(the equivalent generator
>> expression).
>
>
> I don't think that's ever been quite true -- there have
> always been odd cases such as what happens if you
> raise StopIteration in list(generator_expression).

You mean if the genexp leaks one? That's basically an error either way
- the genexp will raise RuntimeError, but it's still an exception.

>>> from __future__ import generator_stop
>>> def boom(): raise StopIteration
...
>>> [x if x < 3 else boom() for x in range(5)]
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1, in 
  File "", line 1, in boom
StopIteration
>>> list(x if x < 3 else boom() for x in range(5))
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1, in boom
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "", line 1, in 
RuntimeError: generator raised StopIteration
>>>

So that's _one_ difference removed (mostly).

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 23 November 2017 at 14:36, Greg Ewing 
wrote:

> Paul Moore wrote:
>
>> 3. List comprehensions are the same as list(the equivalent generator
>> expression).
>>
>
> I don't think that's ever been quite true -- there have
> always been odd cases such as what happens if you
> raise StopIteration in list(generator_expression).
>
> To my mind, these equivalences have never been intended
> as exact descriptions of the semantics, but just a way
> of quickly getting across the general idea. Further
> words are needed to pin down all the fine details.
>

Getting the name resolution to be identical was definitely one of my goals
when working on the Python 3 comprehension scoping changes.

The fact that implicit scopes and yield expressions interact strangely was
just a pre-existing oddity from when PEP 342 was first implemented (and one
we were able to avoid for async/await by retaining the same "await is only
permitted in async comprehensions" constraint that exists for explicit
scopes).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Ivan Levkivskyi wrote:
The key idea is that neither comprehensions nor generator expressions 
should create a function scope surrounding the `expr`


I don't see how you can avoid an implicit function scope in
the case of a generator expression, though. And I can't see
how to make yield in a generator expression do anything
sensible.

Consider this:

   def g():
  return ((yield i) for i in range(10))

Presumably the yield should turn g into a generator, but...
then what? My brain is hurting trying to figure out what
it should do.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Paul Moore wrote:

3. List comprehensions are the same as list(the equivalent generator
expression).


I don't think that's ever been quite true -- there have
always been odd cases such as what happens if you
raise StopIteration in list(generator_expression).

To my mind, these equivalences have never been intended
as exact descriptions of the semantics, but just a way
of quickly getting across the general idea. Further
words are needed to pin down all the fine details.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com