Re: [Python-Dev] Generator objects and list comprehensions?
On Jan 25, 2017 8:16 AM, "Joe Jevnik" wrote:
List, set, and dict comprehensions compile like:
# input
result = [expression for lhs in iterator_expression]
# output
def comprehension(iterator):
out = []
for lhs in iterator:
out.append(expression)
return out
result = comprehension(iter(iterator_expression))
When you make `expression` a `yield` the compiler thinks that
`comprehension` is a generator function instead of a normal function.
We can manually translate the following comprehension:
result = [(yield n) for n in (0, 1)]
def comprehension(iterator):
out = []
for n in iterator:
# (yield n) as an expression is the value sent into this generator
out.append((yield n))
return out
result = comprehension(iter((0, 1)))
We can see this in the behavior of `send` on the resulting generator:
In [1]: g = [(yield n) for n in (0, 1)]
In [2]: next(g)
Out[2]: 0
In [3]: g.send('hello')
Out[3]: 1
In [4]: g.send('world')
---
StopIteration Traceback (most recent call last)
in ()
> 1 g.send('world')
StopIteration: ['hello', 'world']
The `return out` gets translated into `raise StopIteration(out)` because
the code is a generator.
The bytecode for this looks like:
In [5]: %%dis
...: [(yield n) for n in (0, 1)]
...:
1 0 LOAD_CONST 0 ( at
0x7f4bae68eed0, file "", line 1>)
3 LOAD_CONST 1 ('')
6 MAKE_FUNCTION0
9 LOAD_CONST 5 ((0, 1))
12 GET_ITER
13 CALL_FUNCTION1 (1 positional, 0 keyword pair)
16 POP_TOP
17 LOAD_CONST 4 (None)
20 RETURN_VALUE
.
---
1 0 BUILD_LIST 0
3 LOAD_FAST0 (.0)
>>6 FOR_ITER13 (to 22)
9 STORE_FAST 1 (n)
12 LOAD_FAST1 (n)
15 YIELD_VALUE
16 LIST_APPEND 2
19 JUMP_ABSOLUTE6
>> 22 RETURN_VALUE
In `` you can see us create the function, call `iter` on
`(0, 1)`, and then call `(iter(0, 1))`. In `` you can
see the loop like we had above, but unlike a normal comprehension we have a
`YIELD_VALUE` (from the `(yield n)`) in the comprehension.
The reason that this is different in Python 2 is that list comprehension,
and only list comprehensions, compile inline. Instead of creating a new
function for the loop, it is done in the scope of the comprehension. For
example:
# input
result = [expression for lhs in iterator_expression]
# output
result = []
for lhs in iterator_expression:
result.append(lhs)
This is why `lhs` bleeds out of the comprehension. In Python 2, adding
making `expression` a `yield` causes the _calling_ function to become a
generator:
def f():
[(yield n) for n in range(3)]
# no return
# py2
>>> type(f())
generator
# py3
>> type(f())
NoneType
In Python 2 the following will even raise a syntax error:
In [5]: def f():
...: return [(yield n) for n in range(3)]
...:
...:
File "", line 2
return [(yield n) for n in range(3)]
SyntaxError: 'return' with argument inside generator
Generator expressions are a little different because the compilation
already includes a `yield`.
# input
(expression for lhs in iterator_expression)
# output
def genexpr(iterator):
for lhs in iterator:
yield expression
You can actually abuse this to write a cute `flatten` function:
`flatten = lambda seq: (None for sub in seq if (yield from sub) and False)`
because it translates to:
def genexpr(seq):
for sub in seq:
# (yield from sub) as an expression returns the last sent in value
if (yield from sub) and False:
# we never enter this branch
yield None
That was a long way to explain what the problem was. I think that that
solution is to stop using `yield` in comprehensions because it is
confusing, or to make `yield` in a comprehension a syntax error.
Another option would be to restore the py2 behavior by inserting an
implicit 'yield from' whenever the embedded expression contains a yield. So
in your example where
result = [(yield n) for n in (0, 1)]
currently gets expanded to
result = comprehension(iter((0, 1)))
it could notice that the expanded function 'comprehension' is a generator,
and emit code like this instead:
result = yield from comprehension(iter((0, 1)))
At least, this would work for list/dict/set comprehensions; not so much for
generator expressions. I assume this is basically how 'await' in
comprehensions works in 3.6.
I'm not sure this is actually a good idea, given the potential for
ambiguity and backwards compatibility considerations -- I'm kind of leaning
towards the deprecate/error option on balance :-).
Re: [Python-Dev] Generator objects and list comprehensions?
On Wed, Jan 25, 2017 at 6:28 AM, Joe Jevnik wrote: > That was a long way to explain what the problem was. I think that that > solution is to stop using `yield` in comprehensions because it is > confusing, or to make `yield` in a comprehension a syntax error. > > Thanks for the very clear explanation. Would an addition to the documentation for 3.6 giving an abbreviated version help while the devs consider whether any changes are appropriate for 3.7? Steve Holden ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Investigating Python memory footprint of one real Web application
On Thu, Jan 26, 2017 at 2:33 AM, Antoine Pitrou wrote: > On Wed, 25 Jan 2017 20:54:02 +0900 > INADA Naoki wrote: >> >> ## Stripped annotation + without pydebug > > Does this mean the other measurements were done with pydebug enabled? > pydebug is not meant to be used on production systems so, without > wanting to disparage the effort this went into these measurements, I'm > afraid that makes them not very useful. > > Regards > > Antoine. Yes. I used sys.getobjects() API which is available only in pydebug mode. Since it adds two words to all objects for doubly linked list, I did sys.getsizeof(o) - 16 when calculating memory used by each type. While it may bit different from --without-pydebug, I believe it's useful enough to estimate how much memory is used by each types. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
On 25.01.2017 07:28, Joe Jevnik wrote: That was a long way to explain what the problem was. I think that that solution is to stop using `yield` in comprehensions because it is confusing, or to make `yield` in a comprehension a syntax error. Same here; mixing comprehensions and yield (from) can't be explained easily. A SyntaxError would be most appropriate. Regards, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Investigating Python memory footprint of one real Web application
On Wed, 25 Jan 2017 20:54:02 +0900 INADA Naoki wrote: > > ## Stripped annotation + without pydebug Does this mean the other measurements were done with pydebug enabled? pydebug is not meant to be used on production systems so, without wanting to disparage the effort this went into these measurements, I'm afraid that makes them not very useful. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Have problem when building python3.5.1 rpm with default SPEC file
Hello Dahui, So I'll wear my downstream maintainer hat and try to elaborate on the situation that you are facing. Currently Red Hat does not ship Python 3 with rhel 7, so it is not supported. However package maintainers have created a SPEC file for python 3 that works for EPEL7 (aka centos7 or rhel7 etc). The name of the package is python34 [1] and you can install it by following the instructions at [0], it is at version 3.4.5 Currently there is no package for Python 3.5 in the EPEL repositories, you could request that at epel-devel mailing list [2], which is for these kind of queries, someone could be able to provide you with more info there. [0] https://admin.fedoraproject.org/pkgdb/package/rpms/python34/ [1] https://fedoraproject.org/wiki/EPEL [2] https://lists.fedoraproject.org/archives/list/epel-de...@lists.fedoraproject.org/ Regards, Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat - Original Message - From: "Dahui Jiang" To: "Charalampos Stratakis" Cc: python-dev@python.org Sent: Sunday, January 22, 2017 2:58:39 AM Subject: RE: [Python-Dev] Have problem when building python3.5.1 rpm with default SPEC file Hi Charalampos: Thank you very much for your help, and I have built python3.5.1 rpm on redhat7 successfully. But I find that there are quite a few content in SPEC file of fedora, and it seems that the file has been developed for a long time for patches and other supplementary, now that the SPEC file of fedora is open source, how about redhat's, how could I get it? Regards Dahui -Original Message- From: Charalampos Stratakis [mailto:cstra...@redhat.com] Sent: Friday, January 20, 2017 2:40 To: Dahui Jiang Cc: python-dev@python.org Subject: Re: [Python-Dev] Have problem when building python3.5.1 rpm with default SPEC file Hello, This is a distro specific issue so this list might not be the best for resolving that, you should contact your distro's package maintainers of python. For Fedora 25 we currently ship Python 3.5.2, which builds fine with this SPEC file [0], so maybe you could give this a try. [0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/python3.spec?h=f25 Regards, Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat - Original Message - From: "Dahui Jiang" To: python-dev@python.org Sent: Thursday, January 19, 2017 12:54:16 PM Subject: [Python-Dev] Have problem when building python3.5.1 rpm with default SPEC file Hi all: I’m try to build a python rpm from source python3.5.1, and I use the spec file in the source tree. But the building is not success as print the following error: *** running build running build_ext error: [Errno 2] No such file or directory: 'Modules/Setup' error: Bad exit status from /var/tmp/rpm-tmp.DDm3jI (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.DDm3jI (%build) Regards Dahui ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/cstratak%40redhat.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
List, set, and dict comprehensions compile like:
# input
result = [expression for lhs in iterator_expression]
# output
def comprehension(iterator):
out = []
for lhs in iterator:
out.append(expression)
return out
result = comprehension(iter(iterator_expression))
When you make `expression` a `yield` the compiler thinks that
`comprehension` is a generator function instead of a normal function.
We can manually translate the following comprehension:
result = [(yield n) for n in (0, 1)]
def comprehension(iterator):
out = []
for n in iterator:
# (yield n) as an expression is the value sent into this generator
out.append((yield n))
return out
result = comprehension(iter((0, 1)))
We can see this in the behavior of `send` on the resulting generator:
In [1]: g = [(yield n) for n in (0, 1)]
In [2]: next(g)
Out[2]: 0
In [3]: g.send('hello')
Out[3]: 1
In [4]: g.send('world')
---
StopIteration Traceback (most recent call last)
in ()
> 1 g.send('world')
StopIteration: ['hello', 'world']
The `return out` gets translated into `raise StopIteration(out)` because
the code is a generator.
The bytecode for this looks like:
In [5]: %%dis
...: [(yield n) for n in (0, 1)]
...:
1 0 LOAD_CONST 0 ( at
0x7f4bae68eed0, file "", line 1>)
3 LOAD_CONST 1 ('')
6 MAKE_FUNCTION0
9 LOAD_CONST 5 ((0, 1))
12 GET_ITER
13 CALL_FUNCTION1 (1 positional, 0 keyword pair)
16 POP_TOP
17 LOAD_CONST 4 (None)
20 RETURN_VALUE
.
---
1 0 BUILD_LIST 0
3 LOAD_FAST0 (.0)
>>6 FOR_ITER13 (to 22)
9 STORE_FAST 1 (n)
12 LOAD_FAST1 (n)
15 YIELD_VALUE
16 LIST_APPEND 2
19 JUMP_ABSOLUTE6
>> 22 RETURN_VALUE
In `` you can see us create the function, call `iter` on
`(0, 1)`, and then call `(iter(0, 1))`. In `` you can
see the loop like we had above, but unlike a normal comprehension we have a
`YIELD_VALUE` (from the `(yield n)`) in the comprehension.
The reason that this is different in Python 2 is that list comprehension,
and only list comprehensions, compile inline. Instead of creating a new
function for the loop, it is done in the scope of the comprehension. For
example:
# input
result = [expression for lhs in iterator_expression]
# output
result = []
for lhs in iterator_expression:
result.append(lhs)
This is why `lhs` bleeds out of the comprehension. In Python 2, adding
making `expression` a `yield` causes the _calling_ function to become a
generator:
def f():
[(yield n) for n in range(3)]
# no return
# py2
>>> type(f())
generator
# py3
>> type(f())
NoneType
In Python 2 the following will even raise a syntax error:
In [5]: def f():
...: return [(yield n) for n in range(3)]
...:
...:
File "", line 2
return [(yield n) for n in range(3)]
SyntaxError: 'return' with argument inside generator
Generator expressions are a little different because the compilation
already includes a `yield`.
# input
(expression for lhs in iterator_expression)
# output
def genexpr(iterator):
for lhs in iterator:
yield expression
You can actually abuse this to write a cute `flatten` function:
`flatten = lambda seq: (None for sub in seq if (yield from sub) and False)`
because it translates to:
def genexpr(seq):
for sub in seq:
# (yield from sub) as an expression returns the last sent in value
if (yield from sub) and False:
# we never enter this branch
yield None
That was a long way to explain what the problem was. I think that that
solution is to stop using `yield` in comprehensions because it is
confusing, or to make `yield` in a comprehension a syntax error.
On Wed, Jan 25, 2017 at 12:38 AM, Craig Rodrigues
wrote:
> Hi,
>
> Glyph pointed this out to me here: http://twistedmatrix.
> com/pipermail/twisted-python/2017-January/031106.html
>
> If I do this on Python 3.6:
>
> >> [(yield 1) for x in range(10)]
> at 0x10cd210f8>
>
> If I understand this: https://docs.python.org/
> 3/reference/expressions.html#list-displays
> then this is a list display and should give a list, not a generator object.
> Is there a bug in Python, or does the documentation need to be updated?
>
> --
> Craig
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> joe%40quantopian.com
>
>
___
Python-D
Re: [Python-Dev] Investigating Python memory footprint of one real Web application
More detailed information:
## With annotations
=== tracemalloc stat ===
traced: (46969277, 46983753)
18,048,888 / 181112
File "", line 488
File "", line 780
File "", line 675
=== size by types ===
dict 9,083,816 (8,870.91KB) / 21846 = 415.811bytes (21.38%)
tuple6,420,960 (6,270.47KB) / 86781 = 73.990bytes (15.11%)
str 6,050,213 (5,908.41KB) / 77527 = 78.040bytes (14.24%)
function 2,772,224 (2,707.25KB) / 20384 = 136.000bytes (6.53%)
code 2,744,888 (2,680.55KB) / 18987 = 144.567bytes (6.46%)
type 2,713,552 (2,649.95KB) / 2769 = 979.975bytes (6.39%)
bytes2,650,838 (2,588.71KB) / 38723 = 68.456bytes (6.24%)
set 2,445,280 (2,387.97KB) / 6969 = 350.880bytes (5.76%)
weakref 1,255,600 (1,226.17KB) / 15695 = 80.000bytes (2.96%)
list 707,336 (690.76KB) / 6628 = 106.719bytes (1.66%)
=== dict stat ===
t, size,total (%) / count
3, 256,1,479,424 (15.68%) / 5779.0
3,1,200,1,330,800 (14.11%) / 1109.0
3,1,310,832,1,310,832 (13.90%) / 1.0
3, 664,1,287,496 (13.65%) / 1939.0
7, 128, 756,352 (8.02%) / 5909.0
3, 384, 707,328 (7.50%) / 1842.0
3,2,296, 642,880 (6.81%) / 280.0
0, 256, 378,112 (4.01%) / 1477.0
7, 168, 251,832 (2.67%) / 1499.0
3,4,720, 221,840 (2.35%) / 47.0
3,9,336, 130,704 (1.39%) / 14.0
7, 88, 105,072 (1.11%) / 1194.0
* t=7 key-sharing dict, t=3 interned string key only, t=1 string key
only, t=0 non string key is used
## Stripped annotations
=== tracemalloc stat ===
traced: (42383739, 42397983)
18,069,806 / 181346
File "", line 488
File "", line 780
File "", line 675
=== size by types ===
dict 7,913,144 (7,727.68KB) / 17598 = 449.662bytes (20.62%)
tuple6,149,120 (6,005.00KB) / 82734 = 74.324bytes (16.02%)
str 6,070,083 (5,927.82KB) / 77741 = 78.081bytes (15.82%)
code 2,744,312 (2,679.99KB) / 18983 = 144.567bytes (7.15%)
type 2,713,552 (2,649.95KB) / 2769 = 979.975bytes (7.07%)
bytes2,650,464 (2,588.34KB) / 38715 = 68.461bytes (6.91%)
function 2,547,280 (2,487.58KB) / 18730 = 136.000bytes (6.64%)
set 1,423,520 (1,390.16KB) / 4627 = 307.655bytes (3.71%)
list 634,472 (619.60KB) / 5454 = 116.331bytes (1.65%)
int608,784 (594.52KB) / 21021 = 28.961bytes (1.59%)
=== dict stat ===
t, size,total (%) / count
3,1,200,1,316,400 (16.06%) / 1097.0
3,1,310,832,1,310,832 (16.00%) / 1.0
3, 664, 942,216 (11.50%) / 1419.0
3, 256, 861,184 (10.51%) / 3364.0
3, 384, 657,024 (8.02%) / 1711.0
3,2,296, 640,584 (7.82%) / 279.0
7, 128, 606,464 (7.40%) / 4738.0
0, 256, 379,904 (4.64%) / 1484.0
7, 168, 251,832 (3.07%) / 1499.0
3,4,720, 221,840 (2.71%) / 47.0
3,9,336, 130,704 (1.59%) / 14.0
7, 88, 105,248 (1.28%) / 1196.0
7, 256, 86,784 (1.06%) / 339.0
## Stripped annotation + without pydebug
=== tracemalloc stat ===
traced: (37371660, 40814265)
9,812,083 / 111082
File "", line 205
File "", line 742
File "", line 782
6,761,207 / 85614
File "", line 488
File "", line 780
File "", line 675
## Ideas about memory optimize
a) Split PyGC_Head from object
Reduces 2words (16byte) from each tuples.
>>> 82734 * 16 / 1024
1292.71875
So estimated -1.2MB
b) concat co_consts, co_names, co_varnames, co_freevars into one
tuple, or embed them into code.
Each tuple has 3 (gc head) + 3 (refcnt, *type, length) = 6 words
overhead. (or 4 words if (a) is applied)
If we can reduce 3 tuples, 18 words = 144byte (or 12 words=96byte) can
be reuduced.
>>> 18983 * 144
2733552
>>> 18983 * 96
1822368
But co_freevars is empty tuple in most cases. So real effect is
smaller than 2.7MB.
If we can embed them into code object, we can estimate -2.7MB.
(There are co_cellvars too. But I don't know about it much, especially
it is GC tracked or not)
c) (interned) string key only dict.
20% of memory is used for dict, and 70% of dict has interned string keys.
Current DictKeyEntry is 3 words: {key, hash, value}.
But if we can assume all key is string, hash can be get from the key.
If we use 2 words entry: {key, value} for such dict, I think dict can
be 25% smaller.
>>> 7913144 * 0.25 / 1024
1931.919921875
So I estimate -1.9MB
If we can implement (a)~(c) I estimate memory usage on Python
(--without-pydebug)
can be reduced from 35.6MB to 30MB, roughly.
But I think -Onoannotation option is top priority. It can reduce 4MB,
even we use
annotations only in our code.
If major libraries s
Re: [Python-Dev] Generator objects and list comprehensions?
On 25 January 2017 at 07:01, Chris Angelico wrote: > >>> [(yield 1) for x in range(10)] > > at 0x10cd210f8> > This is an old bug, see e.g. http://bugs.python.org/issue10544 The ``yield`` inside comprehensions is bound to the auxiliary function. Instead it should be bound to an enclosing function, like it is done for ``await``. The behaviour of ``await`` in comprehensions is intuitive (since it is simply equivalent to a for-loop): >>> async def f(i): ... return i >>> async def g_for(): ... lst = [] ... for i in range(5): ... lst.append(await f(i)) ... print(lst) >>> g_for().send(None) [0, 1, 2, 3, 4] Traceback (most recent call last): File "", line 1, in StopIteration >>> async def g_comp(): ... print([await f(i) for i in range(5)]) >>> g_comp().send(None)# exactly the same as g_for [0, 1, 2, 3, 4] Traceback (most recent call last): File "", line 1, in StopIteration While current behaviour of ``yield`` in comprehensions is confusing: >>> def c_for(): ... lst = [] ... for i in range(5): ... lst.append((yield i)) ... print(lst) >>> c_for().send(None) 0 >>> c_for().send(None) 1 # etc. >>> def c_comp(): ... print([(yield i) for i in range(5)]) >>> c_comp().send(None) # Naively this should be equivalent to the above, but... . at 0x7f1fd1faa630> Traceback (most recent call last): File "", line 1, in AttributeError: 'NoneType' object has no attribute 'send' I promised myself to write a patch, but didn't have time for this yet. I hope I will do this at some point soon. -- Ivan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
