[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 8c579b1cc86053473eb052b76327279476740c9b by Serhiy Storchaka in 
branch 'master':
bpo-32856: Optimize the assignment idiom in comprehensions. (GH-16814)
https://github.com/python/cpython/commit/8c579b1cc86053473eb052b76327279476740c9b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I want to merge PR 16814 if there are no objections.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Brett Cannon]

Change by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

> However, that's still going to be clearer to most readers than writing

It is subjective. To me, j+1/j looks clearer than (j:=i*i)+1/j. In addition, 
the for-as-assignment idiom is more powerful in context of comprehensions, it 
allows to set an initial value. In any case I want to have a choice.

> OOC, rather than optimizing a fairly ugly use case, might another approach be 
> to make walrus less leaky?

I think this ship is sailed. The semantic of the walrus operator is complex 
enough to make it even more complex by adding more special cases. Also, while 
the function-wide optimization of variables is possible, it much more complex 
problem than the proposed simple optimization.

> You should probably rerun your benchmarks though

$ ./python -m timeit -s 'a = list(range(1000))' -- '[y for x in a for y in [x]]'
Unpatched:  5000 loops, best of 5: 66.8 usec per loop
Patched:   1 loops, best of 5: 21.5 usec per loop

$ ./python -m timeit -s 'a = list(range(1000))' -- '[x for x in a]'
2 loops, best of 5: 17.8 usec per loop

Issue32925 reduce the difference, but it is still large (~12).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Josh Rosenberg]

Josh Rosenberg  added the comment:

OOC, rather than optimizing a fairly ugly use case, might another approach be 
to make walrus less leaky? Even if observable leakage is considered desirable, 
it strikes me that use cases for walrus in genexprs and comprehensions likely 
break up into:

1. 90%: Cases where variable is never used outside genexpr/comprehension 
(because functional programming constructs shouldn't have side-effects, gosh 
darn it!)
2. 5%: Cases where variable is used outside genexpr/comprehension and expects 
leakage
3. 5%: Cases where variable is used outside genexpr/comprehension, but never in 
a way that actually relies on the value set in the genexpr/comprehension (same 
name chosen by happenstance)

If the walrus behavior in genexpr/comprehensions were tweaked to say that it 
only leaks if:

1. It's running at global scope (unavoidable, since there's no way to tell if 
it's an intended part of the module's interface)

or

2. A global or nonlocal statement within the function made it clear the name 
was considered stateful (again, like running at global scope, there is no way 
to know for sure if the name will be used somewhere else)

or

3. At some point in the function, outside the genexpr/comprehension, the value 
of the walrus-assigned name was read.

Case #3 could be even more narrow if the Python AST optimizer was fancier, 
potentially something like "if the value was read *after* the 
genexpr/comprehension, but *before* any following *unconditional* writes to the 
same name" (so [leaked := x for x in it] wouldn't bother to leak "leaked" if 
the next line was "leaked = 1" even if "leaked" were read three lines later, or 
the only reads from leaked occurred before the genexpr/comprehension), but I 
don't think the optimizer is up to that; following simple rules similar to 
those the compiler already follows to identify local names should cover 90% of 
cases anyway.

Aside from the dict returned by locals, and the possibility of earlier 
finalizer invocation (which you couldn't rely on outside CPython anyway), 
there's not much difference in behavior between a leaking and non-leaking 
walrus when the value is never referred to again, and it seems like the 90% 
case for cases where unwanted leakage occurs would be covered by this. Sure, if 
my WAG on use case percentages is correct, 5% of use cases would continue to 
leak even though they didn't benefit from it, but it seems like optimizing the 
90% case would do a lot more good than optimizing what's already a 
micro-optimization that 99% of Python programmers would never use (and 
shouldn't really be encouraged, since it would rely on CPython implementation 
details, and produce uglier code).

I was also inspired by this to look at replacing BUILD_LIST with BUILD_TUPLE 
when followed by GET_ITER (so "[y for x in it for y in [derived(x)]]" would at 
least get the performance benefit of looping over a one-element tuple rather 
than a one-element list), thinking it might reduce the overhead of [y for x in 
a for y in [x]] in your unpatched benchmark by making it equivalent to [y for x 
in a for y in (x,)] while reading more prettily, but it turns out you beat me 
to it with issue32925, so good show there! :-)

You should probably rerun your benchmarks though; with issue32925 committed (a 
month after you posted the benchmarks here), the performance discrepancy should 
be somewhat less (estimate based on local benchmarking says maybe 20% faster 
with BUILD_LIST being optimized to BUILD_TUPLE). Still much faster with the 
proposed optimization than without, but I suspect even optimized, few folks 
will think to write their comprehensions to take advantage of it, which is why 
I was suggesting tweaks to the more obvious walrus operator.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Yury Selivanov]

Change by Yury Selivanov :


--
nosy:  -yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Nick Coghlan]

Nick Coghlan  added the comment:

The benefit offered by the parent local scoping was that it made assignment 
expressions usable as a straightforward way to implement comprehension-based 
accumulators where you actually do want access to the final value after the 
comprehension completes (for example, pulling the example or counter-example 
out of a call to any() or all()).

The downside is that you need an explicit "del j" after the comprehension to 
ensure prompt cleanup in those cases where you *don't* need the temporary 
variable after the comprehension has finished running:

>>> [(j:=i*i)+1/j for i in range(1, 3)]; del j # Clean up temp

However, that's still going to be clearer to most readers than writing:

[j+1/j for i in range(1, 3) for j in [i*i]]

So even with the parent local scoping semantics, PEP 572 is still enough to 
make Yury's comment above still hold (i.e. the use case is too obscure to 
justify the extra code needed to optimise it)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> I just discovered that the assignment operator leaks variables
> out of comprehensions.
> ...
> So it does not supersedes this optimization.

That's a real bummer.  IIRC, it was discussion of this proposal that motivated 
the creation of the walrus operator.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +16365
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/16814

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I just discovered that the assignment operator leaks variables out of 
comprehensions.

>>> [(j:=i*i)+1/j for i in range(1, 3)]
[2.0, 4.25]
>>> j
4
>>> g = ((j:=i*i*i)+1/j for i in range(1, 3))
>>> list(g)
[2.0, 8.125]
>>> j
8

So it does not supersedes this optimization.

--
resolution: rejected -> 
stage: resolved -> 
status: closed -> open
versions: +Python 3.9 -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution: fixed -> rejected

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Closed in favor of PEP 572.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
priority: normal -> low

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Stefan Behnel]

Change by Stefan Behnel :


--
nosy: +scoder

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Yury Selivanov]

Yury Selivanov  added the comment:

I'm still not sure whether we should enable this optimization or not.

I haven't ever seen this pattern used in any Python code I worked with, so I 
suspect it's quite a rare hack.  Giving it a fast-path would give this pattern 
extra visibility and might encourage people to use it.

And the pattern itself... Well, it's quite ugly and barely readable.  IMO it's 
way better to simply rewrite such comprehension to an equivalent 'for' 
statement.

So I guess I'm -0 on this change.  I suggest to ask Guido on the mailing list 
if he agrees that this language patten is worth optimizing/promoting.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Here are some microbenchmarks. But since this code always is a part of complex 
expression it may be not make sense to talk about its pure speed.

$ ./python -m timeit -s 'a = list(range(1000))' -- '[y for x in a for y in [x]]'
Unpatched:  5000 loops, best of 5: 81.4 usec per loop
Patched:   1 loops, best of 5: 19.8 usec per loop

For comparison the variant without temporary variable:

$ ./python -m timeit -s 'a = list(range(1000))' -- '[x for x in a]'
2 loops, best of 5: 15.6 usec per loop

The overhead of using temporary variable is decreased from 66 usec to 4.2 usec 
(by 16 times).

In more realistic examples the subexpression assigned to temporary variable is 
slow. Otherwise it would be not worth to use a temporary variable. Therefore 
the relative speed up of the whole comprehension expression caused by this 
optimization is much smaller.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

It is.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Steven D'Aprano]

Steven D'Aprano  added the comment:

+1

Please also support using a one-element tuple:

`for y in (f(x),)`

--
nosy: +steven.daprano

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +5487
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch, patch
pull_requests: +5486, 5487
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

There were a number of discussions about adding new syntax for temporary 
variables in comprehensions. The last was started yesterday on Python-Ideas 
(https://mail.python.org/pipermail/python-ideas/2018-February/048971.html). The 
problem is that all syntaxes proposed before are ugly. There are common 
solutions of this problem (calculating common subexpression only once): using 
internal comprehension or generator, or refactoring the inner expression as a 
local function where local variables can be used. For example [f(x) + g(f(x)) 
for x in range(10)] can be rewritten as

f_samples = (f(x) for x in range(10))
[y+g(y) for y in f_samples]

or

def func(x):
y = f(x)
return y + g(y)
[func(x) for x in range(10)]

Stephan Houben suggested other idea 
(https://mail.python.org/pipermail/python-ideas/2018-February/048971.html): 
perform an assignment by iterating a one-element list.

[y + g(y) for x in range(10) for y in [f(x)]]

I never seen this idiom before, but seems it is well known for some other 
developers, and it looks less clumsy than other solutions with current syntax. 
Its advantage over hypothetical syntax ideas is that it is an existing syntax. 
Its disadvantage over hypothetical syntax ideas is that iterating a one-element 
list is slightly slower that a simple assignment.

The proposed PR makes `for y in [f(x)]` in comprehensions as fast as just an 
assignment `y = f(x)`. This will make this idiom more preferable for 
performance reasons. Other existing solutions, iterating an inner generator and 
calling a local function in a loop, have an overhead.

--
components: Interpreter Core
messages: 312228
nosy: benjamin.peterson, brett.cannon, ncoghlan, serhiy.storchaka, yselivanov
priority: normal
severity: normal
status: open
title: Optimize the `for y in [x]` idiom in comprehensions
type: performance
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com