Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
Similarly, on macOS 10.12.3 Sierra: % python3.5 Python 3.5.3 (v3.5.3:1880cb95a742, Jan 16 2017, 08:49:46) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> s = set(range(10)) >>> sys.getsizeof(s) 736 >>> sys.getsizeof(set(s)) 736 >>> sys.getsizeof(set(set(s))) 736 >>> % python3 Python 3.6.0 (v3.6.0:41df79263a11, Dec 22 2016, 17:23:13) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> s = set(range(10)) >>> sys.getsizeof(s) 736 >>> sys.getsizeof(set(s)) 1248 >>> sys.getsizeof(set(set(s))) 1248 >>> % python2 Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 12:39:47) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> s = set(range(10)) >>> sys.getsizeof(s) 744 >>> sys.getsizeof(set(s)) 744 >>> sys.getsizeof(set(set(s))) 744 >>> -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 03/30/2017 02:19 PM, INADA Naoki wrote: FYI, this small patch may fix your issue: https://gist.github.com/methane/8faf12621cdb2166019bbcee65987e99 I can verify that the patch fixes the issue for me. Do you still need more information about the `transitive_closure` function and my usage of sets and frozensets? Or any other way I can be helpful in fixing this? (There are a few questions in this thread that I haven't answered so far, but as the problem seems to be identified it might not be worth spending time on that.) Jan -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 03/29/2017 11:31 PM, Chris Angelico wrote: On Thu, Mar 30, 2017 at 2:19 PM, Rick Johnsonwrote: [...] Really? How could your clients not notice 60 GB of memory usage unless they are running some kind of mad-dog insane top-of-the-line hardware? (Got Benjamins???) Of course, in the case they are not DARPA scientist supported by a viturally unlimited supply of tax dollars provided by the endless toils of American slave-bots, how could they ignore the thrashing? o_O Did you read the project's README? This is a dramatic reduction from the normal memory usage of this kind of job. So, yes, they *are* going to have top-of-the-line hardware. If you're doing a job that normally would require 256GB of RAM, and instead of being able to run in 40GB, it needs 60GB, are you going to notice? You probably have 64GB or 128GB. ChrisA Actually, it is not memory, but time improvements. Most people won't run models of the size of Spaun, so usually the memory requirements will be much lower Jan -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 3/30/17, INADA Naokiwrote: > Maybe, this commit make this regression. > > https://github.com/python/cpython/commit/4897300276d870f99459c82b937f0ac22450f0b6 > > Old: > minused = (so->used + other->used)*2 (L619) > > New: > minused = so->used + other->used (L620) > minused = (minused > 5) ? minused * 2 : minused * 4; (L293) > > So size of small set is doubled. > > $ /usr/bin/python3 > Python 3.5.2+ (default, Sep 22 2016, 12:18:14) > [GCC 6.2.0 20160927] on linux > Type "help", "copyright", "credits" or "license" for more information. import sys s = set(range(10)) sys.getsizeof(frozenset(s)) > 736 > > $ python3 > Python 3.6.0 (default, Dec 30 2016, 20:49:54) > [GCC 6.2.0 20161005] on linux > Type "help", "copyright", "credits" or "license" for more information. import sys s = set(range(10)) sys.getsizeof(frozenset(s)) > 1248 > > > > On Fri, Mar 31, 2017 at 2:34 AM, INADA Naoki > wrote: >> I reproduced the issue. >> This is very usual, memory usage issue. Slashing is just a result of >> large memory usage. >> >> After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and >> 30GB on Python 3.6. >> And Python 3.6 starts slashing in 2nd optimization pass. >> >> I enabled tracemalloc while 1st pass. Results is end of this mail. >> It seems frozenset() cause regression, but I'm not sure yet. >> I don't know what is contents of frozenset yet. (I know almost >> nothing about this application). >> >> Jan, do you know about what this is? >> Could you make script which just runs `transitive_closure(edges)` with >> edges similar to >> `log_reduction.py spaun`? >> >> I'll dig into it later, maybe next week. >> >> --- >> Python 3.6.1 >> 1191896 memory blocks: 22086104.2 KiB >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 85 >> reachables[vertex] = frozenset(reachables[vertex]) >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 410 >> self.dependents = transitive_closure(self.dg.forward) >> 602986 memory blocks: 51819.1 KiB >> File "", line 14 >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 634 >> first_view=None, v_offset=0, v_size=0, v_base=None) >> >> Python 3.5.3 >> 1166804 memory blocks: 6407.0 KiB >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 85 >> reachables[vertex] = frozenset(reachables[vertex]) >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 410 >> self.dependents = transitive_closure(self.dg.forward) >> 602989 memory blocks: 51819.3 KiB >> File "", line 14 >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 634 >> first_view=None, v_offset=0, v_size=0, v_base=None) > -- > https://mail.python.org/mailman/listinfo/python-list > Interesting. 1. using set_table_resize with growing factor 2 or 4 doesn't seem to be optimal for constructing (immutable) frozenset from set. 2. growth factor 2 could be too big too ( https://en.wikipedia.org/wiki/Dynamic_array#Growth_factor ,https://github.com/python/cpython/blob/80ec8364f15857c405ef0ecb1e758c8fc6b332f7/Objects/listobject.c#L58 ) PL. -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 2017-03-30 19:04, INADA Naoki wrote: Maybe, this commit make this regression. https://github.com/python/cpython/commit/4897300276d870f99459c82b937f0ac22450f0b6 Old: minused = (so->used + other->used)*2 (L619) New: minused = so->used + other->used (L620) minused = (minused > 5) ? minused * 2 : minused * 4; (L293) So size of small set is doubled. $ /usr/bin/python3 Python 3.5.2+ (default, Sep 22 2016, 12:18:14) [GCC 6.2.0 20160927] on linux Type "help", "copyright", "credits" or "license" for more information. import sys s = set(range(10)) sys.getsizeof(frozenset(s)) 736 $ python3 Python 3.6.0 (default, Dec 30 2016, 20:49:54) [GCC 6.2.0 20161005] on linux Type "help", "copyright", "credits" or "license" for more information. import sys s = set(range(10)) sys.getsizeof(frozenset(s)) 1248 Copying a small set _might_ double its memory usage. Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> s = set(range(10)) >>> sys.getsizeof(s) 736 >>> sys.getsizeof(set(s)) 736 >>> >>> sys.getsizeof(set(set(s))) 736 Python 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 18:41:36) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> s = set(range(10)) >>> sys.getsizeof(s) 736 >>> sys.getsizeof(set(s)) 1248 >>> >>> sys.getsizeof(set(set(s))) 1248 >>> -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
FYI, this small patch may fix your issue: https://gist.github.com/methane/8faf12621cdb2166019bbcee65987e99 -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
That's great news. I'm busy with other things right now, but will look into your findings in more detail later. On 03/30/2017 02:09 PM, INADA Naoki wrote: Filed an issue: https://bugs.python.org/issue29949 Thanks for your report, Jan. On Fri, Mar 31, 2017 at 3:04 AM, INADA Naokiwrote: Maybe, this commit make this regression. https://github.com/python/cpython/commit/4897300276d870f99459c82b937f0ac22450f0b6 Old: minused = (so->used + other->used)*2 (L619) New: minused = so->used + other->used (L620) minused = (minused > 5) ? minused * 2 : minused * 4; (L293) So size of small set is doubled. $ /usr/bin/python3 Python 3.5.2+ (default, Sep 22 2016, 12:18:14) [GCC 6.2.0 20160927] on linux Type "help", "copyright", "credits" or "license" for more information. import sys s = set(range(10)) sys.getsizeof(frozenset(s)) 736 $ python3 Python 3.6.0 (default, Dec 30 2016, 20:49:54) [GCC 6.2.0 20161005] on linux Type "help", "copyright", "credits" or "license" for more information. import sys s = set(range(10)) sys.getsizeof(frozenset(s)) 1248 On Fri, Mar 31, 2017 at 2:34 AM, INADA Naoki wrote: I reproduced the issue. This is very usual, memory usage issue. Slashing is just a result of large memory usage. After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and 30GB on Python 3.6. And Python 3.6 starts slashing in 2nd optimization pass. I enabled tracemalloc while 1st pass. Results is end of this mail. It seems frozenset() cause regression, but I'm not sure yet. I don't know what is contents of frozenset yet. (I know almost nothing about this application). Jan, do you know about what this is? Could you make script which just runs `transitive_closure(edges)` with edges similar to `log_reduction.py spaun`? I'll dig into it later, maybe next week. --- Python 3.6.1 1191896 memory blocks: 22086104.2 KiB File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", line 85 reachables[vertex] = frozenset(reachables[vertex]) File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", line 410 self.dependents = transitive_closure(self.dg.forward) 602986 memory blocks: 51819.1 KiB File "", line 14 File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", line 634 first_view=None, v_offset=0, v_size=0, v_base=None) Python 3.5.3 1166804 memory blocks: 6407.0 KiB File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", line 85 reachables[vertex] = frozenset(reachables[vertex]) File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", line 410 self.dependents = transitive_closure(self.dg.forward) 602989 memory blocks: 51819.3 KiB File "", line 14 File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", line 634 first_view=None, v_offset=0, v_size=0, v_base=None) -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
Filed an issue: https://bugs.python.org/issue29949 Thanks for your report, Jan. On Fri, Mar 31, 2017 at 3:04 AM, INADA Naokiwrote: > Maybe, this commit make this regression. > > https://github.com/python/cpython/commit/4897300276d870f99459c82b937f0ac22450f0b6 > > Old: > minused = (so->used + other->used)*2 (L619) > > New: > minused = so->used + other->used (L620) > minused = (minused > 5) ? minused * 2 : minused * 4; (L293) > > So size of small set is doubled. > > $ /usr/bin/python3 > Python 3.5.2+ (default, Sep 22 2016, 12:18:14) > [GCC 6.2.0 20160927] on linux > Type "help", "copyright", "credits" or "license" for more information. import sys s = set(range(10)) sys.getsizeof(frozenset(s)) > 736 > > $ python3 > Python 3.6.0 (default, Dec 30 2016, 20:49:54) > [GCC 6.2.0 20161005] on linux > Type "help", "copyright", "credits" or "license" for more information. import sys s = set(range(10)) sys.getsizeof(frozenset(s)) > 1248 > > > > On Fri, Mar 31, 2017 at 2:34 AM, INADA Naoki wrote: >> I reproduced the issue. >> This is very usual, memory usage issue. Slashing is just a result of >> large memory usage. >> >> After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and >> 30GB on Python 3.6. >> And Python 3.6 starts slashing in 2nd optimization pass. >> >> I enabled tracemalloc while 1st pass. Results is end of this mail. >> It seems frozenset() cause regression, but I'm not sure yet. >> I don't know what is contents of frozenset yet. (I know almost >> nothing about this application). >> >> Jan, do you know about what this is? >> Could you make script which just runs `transitive_closure(edges)` with >> edges similar to >> `log_reduction.py spaun`? >> >> I'll dig into it later, maybe next week. >> >> --- >> Python 3.6.1 >> 1191896 memory blocks: 22086104.2 KiB >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 85 >> reachables[vertex] = frozenset(reachables[vertex]) >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 410 >> self.dependents = transitive_closure(self.dg.forward) >> 602986 memory blocks: 51819.1 KiB >> File "", line 14 >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 634 >> first_view=None, v_offset=0, v_size=0, v_base=None) >> >> Python 3.5.3 >> 1166804 memory blocks: 6407.0 KiB >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 85 >> reachables[vertex] = frozenset(reachables[vertex]) >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 410 >> self.dependents = transitive_closure(self.dg.forward) >> 602989 memory blocks: 51819.3 KiB >> File "", line 14 >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 634 >> first_view=None, v_offset=0, v_size=0, v_base=None) -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
Maybe, this commit make this regression. https://github.com/python/cpython/commit/4897300276d870f99459c82b937f0ac22450f0b6 Old: minused = (so->used + other->used)*2 (L619) New: minused = so->used + other->used (L620) minused = (minused > 5) ? minused * 2 : minused * 4; (L293) So size of small set is doubled. $ /usr/bin/python3 Python 3.5.2+ (default, Sep 22 2016, 12:18:14) [GCC 6.2.0 20160927] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> s = set(range(10)) >>> sys.getsizeof(frozenset(s)) 736 >>> $ python3 Python 3.6.0 (default, Dec 30 2016, 20:49:54) [GCC 6.2.0 20161005] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> s = set(range(10)) >>> sys.getsizeof(frozenset(s)) 1248 >>> On Fri, Mar 31, 2017 at 2:34 AM, INADA Naokiwrote: > I reproduced the issue. > This is very usual, memory usage issue. Slashing is just a result of > large memory usage. > > After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and > 30GB on Python 3.6. > And Python 3.6 starts slashing in 2nd optimization pass. > > I enabled tracemalloc while 1st pass. Results is end of this mail. > It seems frozenset() cause regression, but I'm not sure yet. > I don't know what is contents of frozenset yet. (I know almost > nothing about this application). > > Jan, do you know about what this is? > Could you make script which just runs `transitive_closure(edges)` with > edges similar to > `log_reduction.py spaun`? > > I'll dig into it later, maybe next week. > > --- > Python 3.6.1 > 1191896 memory blocks: 22086104.2 KiB > File > "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", > line 85 > reachables[vertex] = frozenset(reachables[vertex]) > File > "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", > line 410 > self.dependents = transitive_closure(self.dg.forward) > 602986 memory blocks: 51819.1 KiB > File "", line 14 > File > "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", > line 634 > first_view=None, v_offset=0, v_size=0, v_base=None) > > Python 3.5.3 > 1166804 memory blocks: 6407.0 KiB > File > "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", > line 85 > reachables[vertex] = frozenset(reachables[vertex]) > File > "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", > line 410 > self.dependents = transitive_closure(self.dg.forward) > 602989 memory blocks: 51819.3 KiB > File "", line 14 > File > "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", > line 634 > first_view=None, v_offset=0, v_size=0, v_base=None) -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
I reproduced the issue. This is very usual, memory usage issue. Slashing is just a result of large memory usage. After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and 30GB on Python 3.6. And Python 3.6 starts slashing in 2nd optimization pass. I enabled tracemalloc while 1st pass. Results is end of this mail. It seems frozenset() cause regression, but I'm not sure yet. I don't know what is contents of frozenset yet. (I know almost nothing about this application). Jan, do you know about what this is? Could you make script which just runs `transitive_closure(edges)` with edges similar to `log_reduction.py spaun`? I'll dig into it later, maybe next week. --- Python 3.6.1 1191896 memory blocks: 22086104.2 KiB File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", line 85 reachables[vertex] = frozenset(reachables[vertex]) File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", line 410 self.dependents = transitive_closure(self.dg.forward) 602986 memory blocks: 51819.1 KiB File "", line 14 File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", line 634 first_view=None, v_offset=0, v_size=0, v_base=None) Python 3.5.3 1166804 memory blocks: 6407.0 KiB File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", line 85 reachables[vertex] = frozenset(reachables[vertex]) File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", line 410 self.dependents = transitive_closure(self.dg.forward) 602989 memory blocks: 51819.3 KiB File "", line 14 File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", line 634 first_view=None, v_offset=0, v_size=0, v_base=None) -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 3/29/17, Jan Gosmannwrote: > On 28 Mar 2017, at 14:21, INADA Naoki wrote: > >> On Wed, Mar 29, 2017 at 12:29 AM, Jan Gosmann >> wrote: >> >> I suppose smaller and faster benchmark is better to others looking for >> it. >> I already stopped the azure instance. >> [...] >> There are no maxrss difference in "smaller existing examples"? >> [...] >> I want to investigate RAM usage, without any swapping. > > Running further trials indicate that the problem actually is related to > swapping. If I reduce the model size in the benchmark slightly so that > everything fits into the main memory, the problem disappears. Only when > the memory usage exceeds the 32GB that I have, Python 3.6 will acquire > way more memory (from the swap) than Python 3.5. > > Jan > -- > https://mail.python.org/mailman/listinfo/python-list Could you add table comparing time benchmarks when memory is bigger? (if your hypothesis is true and memory measurement tools are right than time difference has to be huge) Did you compare "pip list" results? There could be more differences in your environments (not only python version). For example different numpy versions or some missing packages could change game. I tried to search "except.*ImportError" in your repository, but I am not sure that it could change it significantly... ( https://github.com/ctn-archive/gosmann-frontiers2017/search?utf8=%E2%9C%93=ImportError= This one seems suspitious - sparse matrix class could be game changer from scipy.sparse import bsr_matrix assert bsr_matrix except (ValueError, ImportError): return False ) This one doesn't seems suspicious to me (but who knows?): try: import faulthandler faulthandler.enable() except: pass PL. -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
> > Running further trials indicate that the problem actually is related to > swapping. If I reduce the model size in the benchmark slightly so that > everything fits into the main memory, the problem disappears. Only when the > memory usage exceeds the 32GB that I have, Python 3.6 will acquire way more > memory (from the swap) than Python 3.5. > > Jan > -- It's very hard to believe... I think there are some factor other than swap cause the problem. Or, can't it reproducible in 64GB RAM machine? -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On Thu, Mar 30, 2017 at 2:19 PM, Rick Johnsonwrote: > On Wednesday, March 29, 2017 at 8:17:01 PM UTC-5, Jan Gosmann wrote: >> On 29 Mar 2017, at 20:12, Steve D'Aprano wrote: >> >> > If you can demonstrate this effect using simple example >> > code without the external dependencies (using nothing but >> > the standard library) and people can replicate it, I think >> > it should be reported as a bug. >> >> I probably won't be able to demonstrate the effect with >> simple example code because I haven't the slightest idea >> what is causing it. Also, I'm not sure how much more time I >> want to invest in this. After all it is a problem that >> might never get noticed by any users of the software. > > Really? How could your clients not notice 60 GB of memory > usage unless they are running some kind of mad-dog insane > top-of-the-line hardware? (Got Benjamins???) Of course, in > the case they are not DARPA scientist supported by a > viturally unlimited supply of tax dollars provided by the > endless toils of American slave-bots, how could they ignore > the thrashing? o_O Did you read the project's README? This is a dramatic reduction from the normal memory usage of this kind of job. So, yes, they *are* going to have top-of-the-line hardware. If you're doing a job that normally would require 256GB of RAM, and instead of being able to run in 40GB, it needs 60GB, are you going to notice? You probably have 64GB or 128GB. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On Wednesday, March 29, 2017 at 8:17:01 PM UTC-5, Jan Gosmann wrote: > On 29 Mar 2017, at 20:12, Steve D'Aprano wrote: > > > If you can demonstrate this effect using simple example > > code without the external dependencies (using nothing but > > the standard library) and people can replicate it, I think > > it should be reported as a bug. > > I probably won't be able to demonstrate the effect with > simple example code because I haven't the slightest idea > what is causing it. Also, I'm not sure how much more time I > want to invest in this. After all it is a problem that > might never get noticed by any users of the software. Really? How could your clients not notice 60 GB of memory usage unless they are running some kind of mad-dog insane top-of-the-line hardware? (Got Benjamins???) Of course, in the case they are not DARPA scientist supported by a viturally unlimited supply of tax dollars provided by the endless toils of American slave-bots, how could they ignore the thrashing? o_O -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 29 Mar 2017, at 20:12, Steve D'Aprano wrote: If you can demonstrate this effect using simple example code without the external dependencies (using nothing but the standard library) and people can replicate it, I think it should be reported as a bug. I probably won't be able to demonstrate the effect with simple example code because I haven't the slightest idea what is causing it. Also, I'm not sure how much more time I want to invest in this. After all it is a problem that might never get noticed by any users of the software. Jan -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On Thu, 30 Mar 2017 07:19 am, Jan Gosmann wrote: > Running further trials indicate that the problem actually is related to > swapping. If I reduce the model size in the benchmark slightly so that > everything fits into the main memory, the problem disappears. Only when > the memory usage exceeds the 32GB that I have, Python 3.6 will acquire > way more memory (from the swap) than Python 3.5. If you can demonstrate this effect using simple example code without the external dependencies (using nothing but the standard library) and people can replicate it, I think it should be reported as a bug. If you are right, it does sound like a performance regression. Maybe there's a way to fix that. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 28 Mar 2017, at 14:21, INADA Naoki wrote: On Wed, Mar 29, 2017 at 12:29 AM, Jan Gosmannwrote: I suppose smaller and faster benchmark is better to others looking for it. I already stopped the azure instance. [...] There are no maxrss difference in "smaller existing examples"? [...] I want to investigate RAM usage, without any swapping. Running further trials indicate that the problem actually is related to swapping. If I reduce the model size in the benchmark slightly so that everything fits into the main memory, the problem disappears. Only when the memory usage exceeds the 32GB that I have, Python 3.6 will acquire way more memory (from the swap) than Python 3.5. Jan -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On Wed, Mar 29, 2017 at 12:29 AM, Jan Gosmannwrote: > On 28 Mar 2017, at 6:11, INADA Naoki wrote: > >> I managed to install pyopencl and run the script. It takes more than >> 2 hours, and uses only 7GB RAM. >> Maybe, some faster backend for OpenCL is required? >> >> I used Microsoft Azure Compute, Standard_A4m_v2 (4 cores, 32 GB >> memory) instance. > > > I suppose that the computing power of the Azure instance might not be > sufficient and it takes much longer to get to the phase where the memory > requirements increase? Have you access to the output that was produced? > I suppose smaller and faster benchmark is better to others looking for it. I already stopped the azure instance. > By the way, this has nothing to do with OpenCL. OpenCL isn't used by the > log_reduction.py script at all. It is listed in the dependencies because > some other things use it. Oh, it took my time much to run the benchmark... All difficult dependencies are required to run the benchmark. > >> More easy way to reproduce is needed... > > > Yes, I agree, but it's not super easy (all the smaller existing examples > don't exhibit the problem so far), but I'll see what I can do. > There are no need to make swapping. Looking difference of normal RAM usage is enough. There are no maxrss difference in "smaller existing examples"? > > I suppose you are right that from the VMM and RSS numbers one cannot deduce > fragmentation. But I think RSS in this case might not be meaningful either. > My understanding from [the Wikipedia description] is that it doesn't account > for parts of the memory that have been written to the swap. Or in other > words RSS will never exceed the size of the physical RAM. VSS is also only > partially useful because it just gives the size of the address space of > which not all might be used? I just meant "VSS is not evidence of fragmentation. Don't suppose fragmentation cause the problem.". > > Anyways, I'm getting a swap usage of about 30GB with Python 3.6 and zsh's > time reports 2339977 page faults from disk vs. 107 for Python 3.5. > > I have some code to measure the unique set size (USS) and will see what > numbers I get with that. > > Jan I want to investigate RAM usage, without any swapping. Regards, -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 28 Mar 2017, at 6:11, INADA Naoki wrote: I managed to install pyopencl and run the script. It takes more than 2 hours, and uses only 7GB RAM. Maybe, some faster backend for OpenCL is required? I used Microsoft Azure Compute, Standard_A4m_v2 (4 cores, 32 GB memory) instance. I suppose that the computing power of the Azure instance might not be sufficient and it takes much longer to get to the phase where the memory requirements increase? Have you access to the output that was produced? By the way, this has nothing to do with OpenCL. OpenCL isn't used by the log_reduction.py script at all. It is listed in the dependencies because some other things use it. More easy way to reproduce is needed... Yes, I agree, but it's not super easy (all the smaller existing examples don't exhibit the problem so far), but I'll see what I can do. My best idea about what's going on at the moment is that memory fragmentation is worse in Python 3.6 for some reason. The virtual memory size indicates that a large address space is acquired, but the resident memory size is smaller indicating that not all of that address space is actually used. In fact, the code might be especially bad to fragmentation because it takes a lot of small NumPy arrays and concatenates them into larger arrays. But I'm still surprised that this is only a problem with Python 3.6 (if this hypothesis is correct). Jan Generally speaking, VMM vs RSS doesn't mean fragmentation. If RSS : total allocated memory ratio is bigger than 1.5, it may be fragmentation. And large VMM won't cause swap. Only RSS is meaningful. I suppose you are right that from the VMM and RSS numbers one cannot deduce fragmentation. But I think RSS in this case might not be meaningful either. My understanding from [the Wikipedia description] is that it doesn't account for parts of the memory that have been written to the swap. Or in other words RSS will never exceed the size of the physical RAM. VSS is also only partially useful because it just gives the size of the address space of which not all might be used? Anyways, I'm getting a swap usage of about 30GB with Python 3.6 and zsh's time reports 2339977 page faults from disk vs. 107 for Python 3.5. I have some code to measure the unique set size (USS) and will see what numbers I get with that. Jan -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 28 Mar 2017, at 3:08, Peter Otten wrote: > Perhaps numpy's default integer type has changed (assuming you are using > integer arrays, I did look at, but not into your code)? > > You could compare > numpy.array([42]).itemsize > 8 > > for the two interpreters. Both report 8 for integer and float point arrays. -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
I can't reproduce it. I managed to install pyopencl and run the script. It takes more than 2 hours, and uses only 7GB RAM. Maybe, some faster backend for OpenCL is required? I used Microsoft Azure Compute, Standard_A4m_v2 (4 cores, 32 GB memory) instance. More easy way to reproduce is needed... > My best idea about what's going on at the moment is that memory > fragmentation is worse in Python 3.6 for some reason. The virtual memory > size indicates that a large address space is acquired, but the resident > memory size is smaller indicating that not all of that address space is > actually used. In fact, the code might be especially bad to fragmentation > because it takes a lot of small NumPy arrays and concatenates them into > larger arrays. But I'm still surprised that this is only a problem with > Python 3.6 (if this hypothesis is correct). > > Jan Generally speaking, VMM vs RSS doesn't mean fragmentation. If RSS : total allocated memory ratio is bigger than 1.5, it may be fragmentation. And large VMM won't cause swap. Only RSS is meaningful. -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
Jan Gosmann wrote: > On 27 Mar 2017, at 18:30, Peter Otten wrote: > >> Are you perchance comparing 32-bit Python 3.5 with 64-bit Python 3.6? > > I don't think so. > [sys.maxsize](https://docs.python.org/3/library/platform.html#cross-platform) > indicates both to be 64-bit. While my original idea was obviously nonsense given the amount of memory you deal with I still cling to the underlying idea. Perhaps numpy's default integer type has changed (assuming you are using integer arrays, I did look at, but not into your code)? You could compare >>> numpy.array([42]).itemsize 8 for the two interpreters. -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 27 Mar 2017, at 20:12, Chris Angelico wrote: On Tue, Mar 28, 2017 at 11:00 AM, Chris Angelicowrote: In any case, I've installed nvidia-opencl-dev and it seems to be happy. How long should I expect this to run for? Now testing under CPython 3.7. It ran for about 14 minutes, then memory usage spiked and went into the page file. Used roughly 8GB of RAM prior to that point, I think? Unfortunately, I don't think my hard disk is fast enough for this to run through the page file in any reasonable time. Yeah, it will be problematic with less than 32GB. My latest findings: The zsh `time` built-in reports a maximum of only 30GB used which seems to correspond to the resident memory size, while the virtual memory size exceeds 40GB for the process and the total memory allocated over all processes exceeds 60GB (with the main contribution from the process that supposedly only uses ~40GB). My best idea about what's going on at the moment is that memory fragmentation is worse in Python 3.6 for some reason. The virtual memory size indicates that a large address space is acquired, but the resident memory size is smaller indicating that not all of that address space is actually used. In fact, the code might be especially bad to fragmentation because it takes a lot of small NumPy arrays and concatenates them into larger arrays. But I'm still surprised that this is only a problem with Python 3.6 (if this hypothesis is correct). Jan -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 27 Mar 2017, at 20:00, Chris Angelico wrote: By the way, since you're aiming this at recent Python versions, you could skip the 'virtualenv' external dependency and use the built-in 'venv' package: $ python3 -m venv env Yeah, I know about venv. The last time I tried it, there was still some issue that prevented me from using it and I'm also still targeting 2.7 too. I'm a little confused here. When I install PIP requirements, it wants to set up OpenCL, but your README seems to be contrasting your implementation with the OpenCL one. Or is that for the sake of benchmarking, so you can compare one against the other? Right, the PIP requirements are more than actually required for this particular problem. But it is indeed for benchmarking comparing different backends for a neural network simulator. For the log_reduction.py script the nengo==2.3.1 and numpy==1.12.0 are probably sufficient. In any case, I've installed nvidia-opencl-dev and it seems to be happy. How long should I expect this to run for? On an Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz with 32GB RAM it probably takes about 30 minutes with Python 3.5; longer with 3.6 because things get written to the swap partition. Jan -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On Tue, Mar 28, 2017 at 11:00 AM, Chris Angelicowrote: > In any case, I've installed nvidia-opencl-dev and it seems to be > happy. How long should I expect this to run for? > > Now testing under CPython 3.7. > It ran for about 14 minutes, then memory usage spiked and went into the page file. Used roughly 8GB of RAM prior to that point, I think? Unfortunately, I don't think my hard disk is fast enough for this to run through the page file in any reasonable time. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On Tue, Mar 28, 2017 at 10:01 AM, Jan Gosmannwrote: > Yes, it is on GitHub (use the fixes branch): > https://github.com/ctn-archive/gosmann-frontiers2017/tree/fixes > Installation instructions are in the readme. > The command I'm running is python scripts/log_reduction.py spaun Working on it. By the way, since you're aiming this at recent Python versions, you could skip the 'virtualenv' external dependency and use the built-in 'venv' package: $ python3 -m venv env I'm a little confused here. When I install PIP requirements, it wants to set up OpenCL, but your README seems to be contrasting your implementation with the OpenCL one. Or is that for the sake of benchmarking, so you can compare one against the other? In any case, I've installed nvidia-opencl-dev and it seems to be happy. How long should I expect this to run for? Now testing under CPython 3.7. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 27 Mar 2017, at 18:42, Chris Angelico wrote: Are you able to share the program? I could try it on my system and see if the same thing happens. Yes, it is on GitHub (use the fixes branch): https://github.com/ctn-archive/gosmann-frontiers2017/tree/fixes Installation instructions are in the readme. The command I'm running is `python scripts/log_reduction.py spaun` I was looking at the output of htop to see the memory consumption. The `time` built-in of the zsh might report a lower memory usage (I'm running it right now to make sure), which would also be surprising. But I'm certain that Python 3.6 uses more memory because the run time increases due to using the swap partition. If you run the program, it will take an initial 10–15 minutes of processing during which the memory usage should increase to roughly 10GB. Then the output “[INFO] Optimizing model...” will be printed after which the memory usage continues to increase to around 30GB (maybe a bit more with Python 3.6). This constitutes a first “optimization pass” that will be ended with the message “[INFO] Pass 1 [views]: Reduced 996917 to 913666 operators in 62.519006s.” Then in the second optimization pass the memory consumption will continue to increase with Python 3.6 (up to ~60GB being consumed, where less than 5GB should be due to other processes, though the virtual and resident memory reported for the Python process are only ~42GB and ~30GB respectively) and in later passes it will start decreasing again. With Python 3.5 the memory consumption stays about 30GB and decreases in later passes. Let me know if you have more questions. -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On Tue, Mar 28, 2017 at 8:57 AM, Jan Gosmannwrote: > I have a program which uses twice as much memory when I run it in Python 3.6 > than when I run it in Python 3.5 (about 60GB instead of 30GB). I tried to > find the reason for that, but the cumulated size (measured with > sys.getsizeof) of all objects returned by gc.get_objects accumulates only to > about 17GB in both cases. The program also uses NumPy and I tried tracking > allocations with the NumPy allocation tracker in the relevant part of the > program, but again the number of allocations are almost identical and the > reported maximum memory usage perfectly agrees (it is about 18GB). > > Any ideas where this significant increase in memory consumption could come > from? Or any ideas how to further debug this? Are you able to share the program? I could try it on my system and see if the same thing happens. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 27 Mar 2017, at 18:30, Peter Otten wrote: Are you perchance comparing 32-bit Python 3.5 with 64-bit Python 3.6? I don't think so. [sys.maxsize](https://docs.python.org/3/library/platform.html#cross-platform) indicates both to be 64-bit. -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
Jan Gosmann wrote: > Hi, > > I have a program which uses twice as much memory when I run it in Python > 3.6 than when I run it in Python 3.5 (about 60GB instead of 30GB). I > tried to find the reason for that, but the cumulated size (measured with > sys.getsizeof) of all objects returned by gc.get_objects accumulates > only to about 17GB in both cases. The program also uses NumPy and I > tried tracking allocations with the NumPy allocation tracker in the > relevant part of the program, but again the number of allocations are > almost identical and the reported maximum memory usage perfectly agrees > (it is about 18GB). > > Any ideas where this significant increase in memory consumption could > come from? Or any ideas how to further debug this? > > Looking at the changelog it seems that the only change in Python 3.6 > affecting memory usage is the new dict implementation which is supposed > to be more memory efficient. In fact, this is what I find if I run the > program with a smaller test cases. There the memory consumption is less > with Python 3.6. Are you perchance comparing 32-bit Python 3.5 with 64-bit Python 3.6? -- https://mail.python.org/mailman/listinfo/python-list
Program uses twice as much memory in Python 3.6 than in Python 3.5
Hi, I have a program which uses twice as much memory when I run it in Python 3.6 than when I run it in Python 3.5 (about 60GB instead of 30GB). I tried to find the reason for that, but the cumulated size (measured with sys.getsizeof) of all objects returned by gc.get_objects accumulates only to about 17GB in both cases. The program also uses NumPy and I tried tracking allocations with the NumPy allocation tracker in the relevant part of the program, but again the number of allocations are almost identical and the reported maximum memory usage perfectly agrees (it is about 18GB). Any ideas where this significant increase in memory consumption could come from? Or any ideas how to further debug this? Looking at the changelog it seems that the only change in Python 3.6 affecting memory usage is the new dict implementation which is supposed to be more memory efficient. In fact, this is what I find if I run the program with a smaller test cases. There the memory consumption is less with Python 3.6. Cheers, Jan -- https://mail.python.org/mailman/listinfo/python-list