Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[MrJean1]

Similarly, on  macOS 10.12.3 Sierra:

% python3.5
Python 3.5.3 (v3.5.3:1880cb95a742, Jan 16 2017, 08:49:46) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> s = set(range(10))
>>> sys.getsizeof(s)
736
>>> sys.getsizeof(set(s))
736
>>> sys.getsizeof(set(set(s)))
736
>>>

% python3
Python 3.6.0 (v3.6.0:41df79263a11, Dec 22 2016, 17:23:13) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> s = set(range(10))
>>> sys.getsizeof(s)
736
>>> sys.getsizeof(set(s))
1248
>>> sys.getsizeof(set(set(s)))
1248
>>>

% python2
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 12:39:47) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> s = set(range(10))
>>> sys.getsizeof(s)
744
>>> sys.getsizeof(set(s))
744
>>> sys.getsizeof(set(set(s)))
744
>>> 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 03/30/2017 02:19 PM, INADA Naoki wrote:

FYI, this small patch may fix your issue:
https://gist.github.com/methane/8faf12621cdb2166019bbcee65987e99

I can verify that the patch fixes the issue for me.

Do you still need more information about the `transitive_closure` 
function and my usage of sets and frozensets? Or any other way I can be 
helpful in fixing this? (There are a few questions in this thread that I 
haven't answered so far, but as the problem seems to be identified it 
might not be worth spending time on that.)


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 03/29/2017 11:31 PM, Chris Angelico wrote:

On Thu, Mar 30, 2017 at 2:19 PM, Rick Johnson
 wrote:

[...]
Really? How could your clients not notice 60 GB of memory
usage unless they are running some kind of mad-dog insane
top-of-the-line hardware? (Got Benjamins???) Of course, in
the case they are not DARPA scientist supported by a
viturally unlimited supply of tax dollars provided by the
endless toils of American slave-bots, how could they ignore
the thrashing? o_O

Did you read the project's README? This is a dramatic reduction from
the normal memory usage of this kind of job. So, yes, they *are* going
to have top-of-the-line hardware. If you're doing a job that normally
would require 256GB of RAM, and instead of being able to run in 40GB,
it needs 60GB, are you going to notice? You probably have 64GB or
128GB.

ChrisA
Actually, it is not memory, but time improvements. Most people won't run 
models of the size of Spaun, so usually the memory requirements will be 
much lower


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Pavol Lisy]

On 3/30/17, INADA Naoki  wrote:
> Maybe, this commit make this regression.
>
> https://github.com/python/cpython/commit/4897300276d870f99459c82b937f0ac22450f0b6
>
> Old:
> minused = (so->used + other->used)*2  (L619)
>
> New:
> minused = so->used + other->used  (L620)
> minused = (minused > 5) ? minused * 2 : minused * 4;  (L293)
>
> So size of small set is doubled.
>
> $ /usr/bin/python3
> Python 3.5.2+ (default, Sep 22 2016, 12:18:14)
> [GCC 6.2.0 20160927] on linux
> Type "help", "copyright", "credits" or "license" for more information.
 import sys
 s = set(range(10))
 sys.getsizeof(frozenset(s))
> 736

>
> $ python3
> Python 3.6.0 (default, Dec 30 2016, 20:49:54)
> [GCC 6.2.0 20161005] on linux
> Type "help", "copyright", "credits" or "license" for more information.
 import  sys
 s = set(range(10))
 sys.getsizeof(frozenset(s))
> 1248

>
>
>
> On Fri, Mar 31, 2017 at 2:34 AM, INADA Naoki 
> wrote:
>> I reproduced the issue.
>> This is very usual, memory usage issue.  Slashing is just a result of
>> large memory usage.
>>
>> After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and
>> 30GB on Python 3.6.
>> And Python 3.6 starts slashing in 2nd optimization pass.
>>
>> I enabled tracemalloc while 1st pass.  Results is end of this mail.
>> It seems frozenset() cause regression, but I'm not sure yet.
>> I don't know what is contents of frozenset yet.  (I know almost
>> nothing about this application).
>>
>> Jan, do you know about what this is?
>> Could you make script which just runs `transitive_closure(edges)` with
>> edges similar to
>> `log_reduction.py spaun`?
>>
>> I'll dig into it later, maybe next week.
>>
>> ---
>> Python 3.6.1
>> 1191896 memory blocks: 22086104.2 KiB
>>   File
>> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
>> line 85
>> reachables[vertex] = frozenset(reachables[vertex])
>>   File
>> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
>> line 410
>> self.dependents = transitive_closure(self.dg.forward)
>> 602986 memory blocks: 51819.1 KiB
>>   File "", line 14
>>   File
>> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
>> line 634
>> first_view=None, v_offset=0, v_size=0, v_base=None)
>>
>> Python 3.5.3
>> 1166804 memory blocks: 6407.0 KiB
>>   File
>> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
>> line 85
>> reachables[vertex] = frozenset(reachables[vertex])
>>   File
>> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
>> line 410
>> self.dependents = transitive_closure(self.dg.forward)
>> 602989 memory blocks: 51819.3 KiB
>>   File "", line 14
>>   File
>> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
>> line 634
>> first_view=None, v_offset=0, v_size=0, v_base=None)
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Interesting.

1. using set_table_resize with growing factor 2 or 4 doesn't seem to
be optimal for constructing (immutable) frozenset from set.

2. growth factor 2 could be too big too (
https://en.wikipedia.org/wiki/Dynamic_array#Growth_factor
,https://github.com/python/cpython/blob/80ec8364f15857c405ef0ecb1e758c8fc6b332f7/Objects/listobject.c#L58
)

PL.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[MRAB]

On 2017-03-30 19:04, INADA Naoki wrote:

Maybe, this commit make this regression.

https://github.com/python/cpython/commit/4897300276d870f99459c82b937f0ac22450f0b6

Old:
minused = (so->used + other->used)*2  (L619)

New:
minused = so->used + other->used  (L620)
minused = (minused > 5) ? minused * 2 : minused * 4;  (L293)

So size of small set is doubled.

$ /usr/bin/python3
Python 3.5.2+ (default, Sep 22 2016, 12:18:14)
[GCC 6.2.0 20160927] on linux
Type "help", "copyright", "credits" or "license" for more information.

import sys
s = set(range(10))
sys.getsizeof(frozenset(s))

736




$ python3
Python 3.6.0 (default, Dec 30 2016, 20:49:54)
[GCC 6.2.0 20161005] on linux
Type "help", "copyright", "credits" or "license" for more information.

import  sys
s = set(range(10))
sys.getsizeof(frozenset(s))

1248



Copying a small set _might_ double its memory usage.


Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 
bit (AMD64)] on win32

Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> s = set(range(10))
>>> sys.getsizeof(s)
736
>>> sys.getsizeof(set(s))
736
>>>
>>> sys.getsizeof(set(set(s)))
736


Python 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 18:41:36) [MSC v.1900 64 bit 
(AMD64)] on win32

Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> s = set(range(10))
>>> sys.getsizeof(s)
736
>>> sys.getsizeof(set(s))
1248
>>>
>>> sys.getsizeof(set(set(s)))
1248
>>>

--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[INADA Naoki]

FYI, this small patch may fix your issue:
https://gist.github.com/methane/8faf12621cdb2166019bbcee65987e99
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

That's great news. I'm busy with other things right now, but will look 
into your findings in more detail later.



On 03/30/2017 02:09 PM, INADA Naoki wrote:

Filed an issue: https://bugs.python.org/issue29949

Thanks for your report, Jan.

On Fri, Mar 31, 2017 at 3:04 AM, INADA Naoki  wrote:

Maybe, this commit make this regression.

https://github.com/python/cpython/commit/4897300276d870f99459c82b937f0ac22450f0b6

Old:
minused = (so->used + other->used)*2  (L619)

New:
minused = so->used + other->used  (L620)
minused = (minused > 5) ? minused * 2 : minused * 4;  (L293)

So size of small set is doubled.

$ /usr/bin/python3
Python 3.5.2+ (default, Sep 22 2016, 12:18:14)
[GCC 6.2.0 20160927] on linux
Type "help", "copyright", "credits" or "license" for more information.

import sys
s = set(range(10))
sys.getsizeof(frozenset(s))

736
$ python3
Python 3.6.0 (default, Dec 30 2016, 20:49:54)
[GCC 6.2.0 20161005] on linux
Type "help", "copyright", "credits" or "license" for more information.

import  sys
s = set(range(10))
sys.getsizeof(frozenset(s))

1248


On Fri, Mar 31, 2017 at 2:34 AM, INADA Naoki  wrote:

I reproduced the issue.
This is very usual, memory usage issue.  Slashing is just a result of
large memory usage.

After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and
30GB on Python 3.6.
And Python 3.6 starts slashing in 2nd optimization pass.

I enabled tracemalloc while 1st pass.  Results is end of this mail.
It seems frozenset() cause regression, but I'm not sure yet.
I don't know what is contents of frozenset yet.  (I know almost
nothing about this application).

Jan, do you know about what this is?
Could you make script which just runs `transitive_closure(edges)` with
edges similar to
`log_reduction.py spaun`?

I'll dig into it later, maybe next week.

---
Python 3.6.1
1191896 memory blocks: 22086104.2 KiB
   File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 85
 reachables[vertex] = frozenset(reachables[vertex])
   File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 410
 self.dependents = transitive_closure(self.dg.forward)
602986 memory blocks: 51819.1 KiB
   File "", line 14
   File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 634
 first_view=None, v_offset=0, v_size=0, v_base=None)

Python 3.5.3
1166804 memory blocks: 6407.0 KiB
   File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 85
 reachables[vertex] = frozenset(reachables[vertex])
   File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 410
 self.dependents = transitive_closure(self.dg.forward)
602989 memory blocks: 51819.3 KiB
   File "", line 14
   File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 634
 first_view=None, v_offset=0, v_size=0, v_base=None)


--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[INADA Naoki]

Filed an issue: https://bugs.python.org/issue29949

Thanks for your report, Jan.

On Fri, Mar 31, 2017 at 3:04 AM, INADA Naoki  wrote:
> Maybe, this commit make this regression.
>
> https://github.com/python/cpython/commit/4897300276d870f99459c82b937f0ac22450f0b6
>
> Old:
> minused = (so->used + other->used)*2  (L619)
>
> New:
> minused = so->used + other->used  (L620)
> minused = (minused > 5) ? minused * 2 : minused * 4;  (L293)
>
> So size of small set is doubled.
>
> $ /usr/bin/python3
> Python 3.5.2+ (default, Sep 22 2016, 12:18:14)
> [GCC 6.2.0 20160927] on linux
> Type "help", "copyright", "credits" or "license" for more information.
 import sys
 s = set(range(10))
 sys.getsizeof(frozenset(s))
> 736

>
> $ python3
> Python 3.6.0 (default, Dec 30 2016, 20:49:54)
> [GCC 6.2.0 20161005] on linux
> Type "help", "copyright", "credits" or "license" for more information.
 import  sys
 s = set(range(10))
 sys.getsizeof(frozenset(s))
> 1248

>
>
>
> On Fri, Mar 31, 2017 at 2:34 AM, INADA Naoki  wrote:
>> I reproduced the issue.
>> This is very usual, memory usage issue.  Slashing is just a result of
>> large memory usage.
>>
>> After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and
>> 30GB on Python 3.6.
>> And Python 3.6 starts slashing in 2nd optimization pass.
>>
>> I enabled tracemalloc while 1st pass.  Results is end of this mail.
>> It seems frozenset() cause regression, but I'm not sure yet.
>> I don't know what is contents of frozenset yet.  (I know almost
>> nothing about this application).
>>
>> Jan, do you know about what this is?
>> Could you make script which just runs `transitive_closure(edges)` with
>> edges similar to
>> `log_reduction.py spaun`?
>>
>> I'll dig into it later, maybe next week.
>>
>> ---
>> Python 3.6.1
>> 1191896 memory blocks: 22086104.2 KiB
>>   File 
>> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
>> line 85
>> reachables[vertex] = frozenset(reachables[vertex])
>>   File 
>> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
>> line 410
>> self.dependents = transitive_closure(self.dg.forward)
>> 602986 memory blocks: 51819.1 KiB
>>   File "", line 14
>>   File 
>> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
>> line 634
>> first_view=None, v_offset=0, v_size=0, v_base=None)
>>
>> Python 3.5.3
>> 1166804 memory blocks: 6407.0 KiB
>>   File 
>> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
>> line 85
>> reachables[vertex] = frozenset(reachables[vertex])
>>   File 
>> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
>> line 410
>> self.dependents = transitive_closure(self.dg.forward)
>> 602989 memory blocks: 51819.3 KiB
>>   File "", line 14
>>   File 
>> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
>> line 634
>> first_view=None, v_offset=0, v_size=0, v_base=None)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[INADA Naoki]

Maybe, this commit make this regression.

https://github.com/python/cpython/commit/4897300276d870f99459c82b937f0ac22450f0b6

Old:
minused = (so->used + other->used)*2  (L619)

New:
minused = so->used + other->used  (L620)
minused = (minused > 5) ? minused * 2 : minused * 4;  (L293)

So size of small set is doubled.

$ /usr/bin/python3
Python 3.5.2+ (default, Sep 22 2016, 12:18:14)
[GCC 6.2.0 20160927] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> s = set(range(10))
>>> sys.getsizeof(frozenset(s))
736
>>>

$ python3
Python 3.6.0 (default, Dec 30 2016, 20:49:54)
[GCC 6.2.0 20161005] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import  sys
>>> s = set(range(10))
>>> sys.getsizeof(frozenset(s))
1248
>>>



On Fri, Mar 31, 2017 at 2:34 AM, INADA Naoki  wrote:
> I reproduced the issue.
> This is very usual, memory usage issue.  Slashing is just a result of
> large memory usage.
>
> After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and
> 30GB on Python 3.6.
> And Python 3.6 starts slashing in 2nd optimization pass.
>
> I enabled tracemalloc while 1st pass.  Results is end of this mail.
> It seems frozenset() cause regression, but I'm not sure yet.
> I don't know what is contents of frozenset yet.  (I know almost
> nothing about this application).
>
> Jan, do you know about what this is?
> Could you make script which just runs `transitive_closure(edges)` with
> edges similar to
> `log_reduction.py spaun`?
>
> I'll dig into it later, maybe next week.
>
> ---
> Python 3.6.1
> 1191896 memory blocks: 22086104.2 KiB
>   File 
> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
> line 85
> reachables[vertex] = frozenset(reachables[vertex])
>   File 
> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
> line 410
> self.dependents = transitive_closure(self.dg.forward)
> 602986 memory blocks: 51819.1 KiB
>   File "", line 14
>   File 
> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
> line 634
> first_view=None, v_offset=0, v_size=0, v_base=None)
>
> Python 3.5.3
> 1166804 memory blocks: 6407.0 KiB
>   File 
> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
> line 85
> reachables[vertex] = frozenset(reachables[vertex])
>   File 
> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
> line 410
> self.dependents = transitive_closure(self.dg.forward)
> 602989 memory blocks: 51819.3 KiB
>   File "", line 14
>   File 
> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
> line 634
> first_view=None, v_offset=0, v_size=0, v_base=None)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[INADA Naoki]

I reproduced the issue.
This is very usual, memory usage issue.  Slashing is just a result of
large memory usage.

After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and
30GB on Python 3.6.
And Python 3.6 starts slashing in 2nd optimization pass.

I enabled tracemalloc while 1st pass.  Results is end of this mail.
It seems frozenset() cause regression, but I'm not sure yet.
I don't know what is contents of frozenset yet.  (I know almost
nothing about this application).

Jan, do you know about what this is?
Could you make script which just runs `transitive_closure(edges)` with
edges similar to
`log_reduction.py spaun`?

I'll dig into it later, maybe next week.

---
Python 3.6.1
1191896 memory blocks: 22086104.2 KiB
  File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 85
reachables[vertex] = frozenset(reachables[vertex])
  File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 410
self.dependents = transitive_closure(self.dg.forward)
602986 memory blocks: 51819.1 KiB
  File "", line 14
  File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 634
first_view=None, v_offset=0, v_size=0, v_base=None)

Python 3.5.3
1166804 memory blocks: 6407.0 KiB
  File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 85
reachables[vertex] = frozenset(reachables[vertex])
  File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 410
self.dependents = transitive_closure(self.dg.forward)
602989 memory blocks: 51819.3 KiB
  File "", line 14
  File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 634
first_view=None, v_offset=0, v_size=0, v_base=None)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Pavol Lisy]

On 3/29/17, Jan Gosmann  wrote:
> On 28 Mar 2017, at 14:21, INADA Naoki wrote:
>
>> On Wed, Mar 29, 2017 at 12:29 AM, Jan Gosmann 
>> wrote:
>>
>> I suppose smaller and faster benchmark is better to others looking for
>> it.
>> I already stopped the azure instance.
>> [...]
>> There are no maxrss difference in "smaller existing examples"?
>> [...]
>> I want to investigate RAM usage, without any swapping.
>
> Running further trials indicate that the problem actually is related to
> swapping. If I reduce the model size in the benchmark slightly so that
> everything fits into the main memory, the problem disappears. Only when
> the memory usage exceeds the 32GB that I have, Python 3.6 will acquire
> way more memory (from the swap) than Python 3.5.
>
> Jan
> --
> https://mail.python.org/mailman/listinfo/python-list

Could you add table comparing time benchmarks when memory is bigger?
(if your hypothesis is true and memory measurement tools are right
than time difference has to be huge)

Did you compare "pip list" results? There could be more differences in
your environments (not only python version). For example different
numpy versions or some missing packages could change game.

I tried to search "except.*ImportError" in your repository, but I am
not sure that it could change it significantly...

( 
https://github.com/ctn-archive/gosmann-frontiers2017/search?utf8=%E2%9C%93=ImportError=

This one seems suspitious - sparse matrix class could be game changer

from scipy.sparse import bsr_matrix
assert bsr_matrix
except (ValueError, ImportError):
return False

)


This one doesn't seems suspicious to me (but who knows?):

try:
import faulthandler
faulthandler.enable()
except:
pass

PL.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[INADA Naoki]

>
> Running further trials indicate that the problem actually is related to
> swapping. If I reduce the model size in the benchmark slightly so that
> everything fits into the main memory, the problem disappears. Only when the
> memory usage exceeds the 32GB that I have, Python 3.6 will acquire way more
> memory (from the swap) than Python 3.5.
>
> Jan
> --

It's very hard to believe...
I think there are some factor other than swap cause the problem.
Or, can't it reproducible in 64GB RAM machine?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Chris Angelico]

On Thu, Mar 30, 2017 at 2:19 PM, Rick Johnson
 wrote:
> On Wednesday, March 29, 2017 at 8:17:01 PM UTC-5, Jan Gosmann wrote:
>> On 29 Mar 2017, at 20:12, Steve D'Aprano wrote:
>>
>> > If you can demonstrate this effect using simple example
>> > code without the external dependencies (using nothing but
>> > the standard library) and people can replicate it, I think
>> > it should be reported as a bug.
>>
>> I probably won't be able to demonstrate the effect with
>> simple example code because I haven't the slightest idea
>> what is causing it. Also, I'm not sure how much more time I
>> want to invest in this. After all it is a problem that
>> might never get noticed by any users of the software.
>
> Really? How could your clients not notice 60 GB of memory
> usage unless they are running some kind of mad-dog insane
> top-of-the-line hardware? (Got Benjamins???) Of course, in
> the case they are not DARPA scientist supported by a
> viturally unlimited supply of tax dollars provided by the
> endless toils of American slave-bots, how could they ignore
> the thrashing? o_O

Did you read the project's README? This is a dramatic reduction from
the normal memory usage of this kind of job. So, yes, they *are* going
to have top-of-the-line hardware. If you're doing a job that normally
would require 256GB of RAM, and instead of being able to run in 40GB,
it needs 60GB, are you going to notice? You probably have 64GB or
128GB.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Rick Johnson]

On Wednesday, March 29, 2017 at 8:17:01 PM UTC-5, Jan Gosmann wrote:
> On 29 Mar 2017, at 20:12, Steve D'Aprano wrote:
> 
> > If you can demonstrate this effect using simple example
> > code without the external dependencies (using nothing but
> > the standard library) and people can replicate it, I think
> > it should be reported as a bug.
> 
> I probably won't be able to demonstrate the effect with
> simple example code because I haven't the slightest idea
> what is causing it. Also, I'm not sure how much more time I
> want to invest in this. After all it is a problem that
> might never get noticed by any users of the software.

Really? How could your clients not notice 60 GB of memory
usage unless they are running some kind of mad-dog insane
top-of-the-line hardware? (Got Benjamins???) Of course, in
the case they are not DARPA scientist supported by a
viturally unlimited supply of tax dollars provided by the
endless toils of American slave-bots, how could they ignore
the thrashing? o_O
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 29 Mar 2017, at 20:12, Steve D'Aprano wrote:

If you can demonstrate this effect using simple example code without 
the
external dependencies (using nothing but the standard library) and 
people

can replicate it, I think it should be reported as a bug.


I probably won't be able to demonstrate the effect with simple example 
code because I haven't the slightest idea what is causing it. Also, I'm 
not sure how much more time I want to invest in this. After all it is a 
problem that might never get noticed by any users of the software.


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Steve D'Aprano]

On Thu, 30 Mar 2017 07:19 am, Jan Gosmann wrote:

> Running further trials indicate that the problem actually is related to
> swapping. If I reduce the model size in the benchmark slightly so that
> everything fits into the main memory, the problem disappears. Only when
> the memory usage exceeds the 32GB that I have, Python 3.6 will acquire
> way more memory (from the swap) than Python 3.5.

If you can demonstrate this effect using simple example code without the
external dependencies (using nothing but the standard library) and people
can replicate it, I think it should be reported as a bug.

If you are right, it does sound like a performance regression. Maybe there's
a way to fix that.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 28 Mar 2017, at 14:21, INADA Naoki wrote:

On Wed, Mar 29, 2017 at 12:29 AM, Jan Gosmann  
wrote:


I suppose smaller and faster benchmark is better to others looking for 
it.

I already stopped the azure instance.
[...]
There are no maxrss difference in "smaller existing examples"?
[...]
I want to investigate RAM usage, without any swapping.


Running further trials indicate that the problem actually is related to 
swapping. If I reduce the model size in the benchmark slightly so that 
everything fits into the main memory, the problem disappears. Only when 
the memory usage exceeds the 32GB that I have, Python 3.6 will acquire 
way more memory (from the swap) than Python 3.5.


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[INADA Naoki]

On Wed, Mar 29, 2017 at 12:29 AM, Jan Gosmann  wrote:
> On 28 Mar 2017, at 6:11, INADA Naoki wrote:
>
>> I managed to install pyopencl and run the script.  It takes more than
>> 2 hours, and uses only 7GB RAM.
>> Maybe, some faster backend for OpenCL is required?
>>
>> I used Microsoft Azure Compute, Standard_A4m_v2 (4 cores, 32 GB
>> memory) instance.
>
>
> I suppose that the computing power of the Azure instance might not be
> sufficient and it takes much longer to get to the phase where the memory
> requirements increase? Have you access to the output that was produced?
>

I suppose smaller and faster benchmark is better to others looking for it.
I already stopped the azure instance.

> By the way, this has nothing to do with OpenCL. OpenCL isn't used by the
> log_reduction.py script at all. It is listed in the dependencies because
> some other things use it.

Oh, it took my time much to run the benchmark...
All difficult dependencies are required to run the benchmark.

>
>> More easy way to reproduce is needed...
>
>
> Yes, I agree, but it's not super easy (all the smaller existing examples
> don't exhibit the problem so far), but I'll see what I can do.
>

There are no need to make swapping.  Looking difference of normal RAM
usage is enough.
There are no maxrss difference in "smaller existing examples"?

>
> I suppose you are right that from the VMM and RSS numbers one cannot deduce
> fragmentation. But I think RSS in this case might not be meaningful either.
> My understanding from [the Wikipedia description] is that it doesn't account
> for parts of the memory that have been written to the swap. Or in other
> words RSS will never exceed the size of the physical RAM. VSS is also only
> partially useful because it just gives the size of the address space of
> which not all might be used?

I just meant "VSS is not evidence of fragmentation.  Don't suppose
fragmentation cause the problem.".


>
> Anyways, I'm getting a swap usage of about 30GB with Python 3.6 and zsh's
> time reports 2339977 page faults from disk vs. 107 for Python 3.5.
>
> I have some code to measure the unique set size (USS) and will see what
> numbers I get with that.
>
> Jan

I want to investigate RAM usage, without any swapping.


Regards,
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 28 Mar 2017, at 6:11, INADA Naoki wrote:


I managed to install pyopencl and run the script.  It takes more than
2 hours, and uses only 7GB RAM.
Maybe, some faster backend for OpenCL is required?

I used Microsoft Azure Compute, Standard_A4m_v2 (4 cores, 32 GB
memory) instance.


I suppose that the computing power of the Azure instance might not be 
sufficient and it takes much longer to get to the phase where the memory 
requirements increase? Have you access to the output that was produced?


By the way, this has nothing to do with OpenCL. OpenCL isn't used by the 
log_reduction.py script at all. It is listed in the dependencies because 
some other things use it.



More easy way to reproduce is needed...


Yes, I agree, but it's not super easy (all the smaller existing examples 
don't exhibit the problem so far), but I'll see what I can do.



My best idea about what's going on at the moment is that memory
fragmentation is worse in Python 3.6 for some reason. The virtual 
memory
size indicates that a large address space is acquired, but the 
resident
memory size is smaller indicating that not all of that address space 
is
actually used. In fact, the code might be especially bad to 
fragmentation
because it takes a lot of small NumPy arrays and concatenates them 
into
larger arrays. But I'm still surprised that this is only a problem 
with

Python 3.6 (if this hypothesis is correct).

Jan


Generally speaking, VMM vs RSS doesn't mean fragmentation.
If RSS : total allocated memory ratio is bigger than 1.5, it may be
fragmentation.
And large VMM won't cause swap.  Only RSS is meaningful.


I suppose you are right that from the VMM and RSS numbers one cannot 
deduce fragmentation. But I think RSS in this case might not be 
meaningful either. My understanding from [the Wikipedia description] is 
that it doesn't account for parts of the memory that have been written 
to the swap. Or in other words RSS will never exceed the size of the 
physical RAM. VSS is also only partially useful because it just gives 
the size of the address space of which not all might be used?


Anyways, I'm getting a swap usage of about 30GB with Python 3.6 and 
zsh's time reports 2339977 page faults from disk vs. 107 for Python 3.5.


I have some code to measure the unique set size (USS) and will see what 
numbers I get with that.


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 28 Mar 2017, at 3:08, Peter Otten wrote:

> Perhaps numpy's default integer type has changed (assuming you are using
> integer arrays, I did look at, but not into your code)?
>
> You could compare
>
 numpy.array([42]).itemsize
> 8
>
> for the two interpreters.

Both report 8 for integer and float point arrays.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[INADA Naoki]

I can't reproduce it.

I managed to install pyopencl and run the script.  It takes more than
2 hours, and uses only 7GB RAM.
Maybe, some faster backend for OpenCL is required?

I used Microsoft Azure Compute, Standard_A4m_v2 (4 cores, 32 GB
memory) instance.

More easy way to reproduce is needed...

> My best idea about what's going on at the moment is that memory
> fragmentation is worse in Python 3.6 for some reason. The virtual memory
> size indicates that a large address space is acquired, but the resident
> memory size is smaller indicating that not all of that address space is
> actually used. In fact, the code might be especially bad to fragmentation
> because it takes a lot of small NumPy arrays and concatenates them into
> larger arrays. But I'm still surprised that this is only a problem with
> Python 3.6 (if this hypothesis is correct).
>
> Jan

Generally speaking, VMM vs RSS doesn't mean fragmentation.
If RSS : total allocated memory ratio is bigger than 1.5, it may be
fragmentation.
And large VMM won't cause swap.  Only RSS is meaningful.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Peter Otten]

Jan Gosmann wrote:

> On 27 Mar 2017, at 18:30, Peter Otten wrote:
> 
>> Are you perchance comparing 32-bit Python 3.5 with 64-bit Python 3.6?
> 
> I don't think so.
> [sys.maxsize](https://docs.python.org/3/library/platform.html#cross-platform)
> indicates both to be 64-bit.

While my original idea was obviously nonsense given the amount of memory you 
deal with I still cling to the underlying idea.

Perhaps numpy's default integer type has changed (assuming you are using 
integer arrays, I did look at, but not into your code)?

You could compare

>>> numpy.array([42]).itemsize
8

for the two interpreters.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 27 Mar 2017, at 20:12, Chris Angelico wrote:

On Tue, Mar 28, 2017 at 11:00 AM, Chris Angelico  
wrote:

In any case, I've installed nvidia-opencl-dev and it seems to be
happy. How long should I expect this to run for?

Now testing under CPython 3.7.



It ran for about 14 minutes, then memory usage spiked and went into
the page file. Used roughly 8GB of RAM prior to that point, I think?
Unfortunately, I don't think my hard disk is fast enough for this to
run through the page file in any reasonable time.


Yeah, it will be problematic with less than 32GB.

My latest findings: The zsh `time` built-in reports a maximum of only 
30GB used which seems to correspond to the resident memory size, while 
the virtual memory size exceeds 40GB for the process and the total 
memory allocated over all processes exceeds 60GB (with the main 
contribution from the process that supposedly only uses ~40GB).


My best idea about what's going on at the moment is that memory 
fragmentation is worse in Python 3.6 for some reason. The virtual memory 
size indicates that a large address space is acquired, but the resident 
memory size is smaller indicating that not all of that address space is 
actually used. In fact, the code might be especially bad to 
fragmentation because it takes a lot of small NumPy arrays and 
concatenates them into larger arrays. But I'm still surprised that this 
is only a problem with Python 3.6 (if this hypothesis is correct).


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 27 Mar 2017, at 20:00, Chris Angelico wrote:


By the way, since you're aiming this at recent Python versions, you
could skip the 'virtualenv' external dependency and use the built-in
'venv' package:

$ python3 -m venv env


Yeah, I know about venv. The last time I tried it, there was still some 
issue that prevented me from using it and I'm also still targeting 2.7 
too.



I'm a little confused here. When I install PIP requirements, it wants
to set up OpenCL, but your README seems to be contrasting your
implementation with the OpenCL one. Or is that for the sake of
benchmarking, so you can compare one against the other?


Right, the PIP requirements are more than actually required for this 
particular problem. But it is indeed for benchmarking comparing 
different backends for a neural network simulator.


For the log_reduction.py script the nengo==2.3.1 and numpy==1.12.0 are 
probably sufficient.



In any case, I've installed nvidia-opencl-dev and it seems to be
happy. How long should I expect this to run for?


On an Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz with 32GB RAM it 
probably takes about 30 minutes with Python 3.5; longer with 3.6 because 
things get written to the swap partition.


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Chris Angelico]

On Tue, Mar 28, 2017 at 11:00 AM, Chris Angelico  wrote:
> In any case, I've installed nvidia-opencl-dev and it seems to be
> happy. How long should I expect this to run for?
>
> Now testing under CPython 3.7.
>

It ran for about 14 minutes, then memory usage spiked and went into
the page file. Used roughly 8GB of RAM prior to that point, I think?
Unfortunately, I don't think my hard disk is fast enough for this to
run through the page file in any reasonable time.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Chris Angelico]

On Tue, Mar 28, 2017 at 10:01 AM, Jan Gosmann  wrote:
> Yes, it is on GitHub (use the fixes branch):
> https://github.com/ctn-archive/gosmann-frontiers2017/tree/fixes
> Installation instructions are in the readme.
> The command I'm running is python scripts/log_reduction.py spaun

Working on it.

By the way, since you're aiming this at recent Python versions, you
could skip the 'virtualenv' external dependency and use the built-in
'venv' package:

$ python3 -m venv env

I'm a little confused here. When I install PIP requirements, it wants
to set up OpenCL, but your README seems to be contrasting your
implementation with the OpenCL one. Or is that for the sake of
benchmarking, so you can compare one against the other?

In any case, I've installed nvidia-opencl-dev and it seems to be
happy. How long should I expect this to run for?

Now testing under CPython 3.7.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 27 Mar 2017, at 18:42, Chris Angelico wrote:


Are you able to share the program? I could try it on my system and see
if the same thing happens.


Yes, it is on GitHub (use the fixes branch): 
https://github.com/ctn-archive/gosmann-frontiers2017/tree/fixes

Installation instructions are in the readme.
The command I'm running is `python scripts/log_reduction.py spaun`

I was looking at the output of htop to see the memory consumption. The 
`time` built-in of the zsh might report a lower memory usage (I'm 
running it right now to make sure), which would also be surprising. But 
I'm certain that Python 3.6 uses more memory because the run time 
increases due to using the swap partition.


If you run the program, it will take an initial 10–15 minutes of 
processing during which the memory usage should increase to roughly 
10GB. Then the output “[INFO] Optimizing model...” will be printed 
after which the memory usage continues to increase to around 30GB (maybe 
a bit more with Python 3.6). This constitutes a first “optimization 
pass” that will be ended with the message “[INFO] Pass 1 [views]: 
Reduced 996917 to 913666 operators in 62.519006s.” Then in the second 
optimization pass the memory consumption will continue to increase with 
Python 3.6 (up to ~60GB being consumed, where less than 5GB should be 
due to other processes, though the virtual and resident memory reported 
for the Python process are only ~42GB and ~30GB respectively) and in 
later passes it will start decreasing again. With Python 3.5 the memory 
consumption stays about 30GB and decreases in later passes.


Let me know if you have more questions.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Chris Angelico]

On Tue, Mar 28, 2017 at 8:57 AM, Jan Gosmann  wrote:
> I have a program which uses twice as much memory when I run it in Python 3.6
> than when I run it in Python 3.5 (about 60GB instead of 30GB). I tried to
> find the reason for that, but the cumulated size (measured with
> sys.getsizeof) of all objects returned by gc.get_objects accumulates only to
> about 17GB in both cases. The program also uses NumPy and I tried tracking
> allocations with the NumPy allocation tracker in the relevant part of the
> program, but again the number of allocations are almost identical and the
> reported maximum memory usage perfectly agrees (it is about 18GB).
>
> Any ideas where this significant increase in memory consumption could come
> from? Or any ideas how to further debug this?

Are you able to share the program? I could try it on my system and see
if the same thing happens.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 27 Mar 2017, at 18:30, Peter Otten wrote:


Are you perchance comparing 32-bit Python 3.5 with 64-bit Python 3.6?


I don't think so. 
[sys.maxsize](https://docs.python.org/3/library/platform.html#cross-platform) 
indicates both to be 64-bit.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Peter Otten]

Jan Gosmann wrote:

> Hi,
> 
> I have a program which uses twice as much memory when I run it in Python
> 3.6 than when I run it in Python 3.5 (about 60GB instead of 30GB). I
> tried to find the reason for that, but the cumulated size (measured with
> sys.getsizeof) of all objects returned by gc.get_objects accumulates
> only to about 17GB in both cases. The program also uses NumPy and I
> tried tracking allocations with the NumPy allocation tracker in the
> relevant part of the program, but again the number of allocations are
> almost identical and the reported maximum memory usage perfectly agrees
> (it is about 18GB).
> 
> Any ideas where this significant increase in memory consumption could
> come from? Or any ideas how to further debug this?
> 
> Looking at the changelog it seems that the only change in Python 3.6
> affecting memory usage is the new dict implementation which is supposed
> to be more memory efficient. In fact, this is what I find if I run the
> program with a smaller test cases. There the memory consumption is less
> with Python 3.6.

Are you perchance comparing 32-bit Python 3.5 with 64-bit Python 3.6?

-- 
https://mail.python.org/mailman/listinfo/python-list

Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

Hi,

I have a program which uses twice as much memory when I run it in Python 
3.6 than when I run it in Python 3.5 (about 60GB instead of 30GB). I 
tried to find the reason for that, but the cumulated size (measured with 
sys.getsizeof) of all objects returned by gc.get_objects accumulates 
only to about 17GB in both cases. The program also uses NumPy and I 
tried tracking allocations with the NumPy allocation tracker in the 
relevant part of the program, but again the number of allocations are 
almost identical and the reported maximum memory usage perfectly agrees 
(it is about 18GB).


Any ideas where this significant increase in memory consumption could 
come from? Or any ideas how to further debug this?


Looking at the changelog it seems that the only change in Python 3.6 
affecting memory usage is the new dict implementation which is supposed 
to be more memory efficient. In fact, this is what I find if I run the 
program with a smaller test cases. There the memory consumption is less 
with Python 3.6.


Cheers, Jan
--
https://mail.python.org/mailman/listinfo/python-list