Re: [Python-Dev] Reduce memory footprint of Python

2013-10-09 Thread Barry Warsaw
On Oct 06, 2013, at 11:41 AM, Benjamin Peterson wrote: >What is the usecase for minimizing the memory usage of Python in the >middle of a program? There are environments where Python is being actively discouraged in part due to its memory footprint. For example, we can't use Python on Ubuntu Tou

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-09 Thread Antoine Pitrou
Le Wed, 09 Oct 2013 11:01:01 +1300, Greg Ewing a écrit : > R. David Murray wrote: > > I can give you one data point: a mobile platform that (currently) > > uses Python3, and does not use linecache because of how much memory > > it consumes. > > Wouldn't a sensible approach be to discard the linec

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-09 Thread Antoine Pitrou
Le Tue, 8 Oct 2013 15:43:40 -0400, Benjamin Peterson a écrit : > 2013/10/8 R. David Murray : > > In this context, if we'd been *really* smart-lazy in CPython > > development, we'd have kept the memory and startup-time and...well, > > we probably do pretty well on CPU actually...smaller, so that w

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-08 Thread Greg Ewing
R. David Murray wrote: I can give you one data point: a mobile platform that (currently) uses Python3, and does not use linecache because of how much memory it consumes. Wouldn't a sensible approach be to discard the linecache when you've finished generating a traceback? You're not likely to be

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-08 Thread Benjamin Peterson
2013/10/8 R. David Murray : > In this context, if we'd been *really* smart-lazy in CPython development, > we'd have kept the memory and startup-time and...well, we probably do > pretty well on CPU actually...smaller, so that when smartphones came > along Python would have been the first high level

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-08 Thread R. David Murray
On Tue, 08 Oct 2013 14:43:03 -0400, Benjamin Peterson wrote: > 2013/10/8 R. David Murray : > > PS: I have always thought it sad that the ready availability of memory, > > CPU speed, and disk space tends to result in lazy programs. I understand > > there is an effort/value tradeoff, and I make th

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-08 Thread Benjamin Peterson
2013/10/8 R. David Murray : > PS: I have always thought it sad that the ready availability of memory, > CPU speed, and disk space tends to result in lazy programs. I understand > there is an effort/value tradeoff, and I make those tradeoffs myself > all the time...but it still makes me sad. Then,

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-08 Thread R. David Murray
On Sun, 06 Oct 2013 22:27:52 +0200, mar...@v.loewis.de wrote: > > Quoting Benjamin Peterson : > > >> If you know that your application uses a lot of memory, it is > >> interesting to sometimes (when the application is idle) try to release > >> some bytes to not use all the system memory. On embed

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-08 Thread Antoine Pitrou
For the record, if you want to reduce memory footprint of Python on a x86-64 bit, you can use the "x32" ABI for which Debian/Ubuntu provide some support. (install the required base packages and use CFLAGS=-mx32 LDFLAGS=-mx32) Regards Antoine. Le Sun, 6 Oct 2013 17:32:37 +0200, Victor Stinner

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Nick Coghlan
On 7 Oct 2013 07:28, "Victor Stinner" wrote: > > 2013/10/6 : > > Quoting Antoine Pitrou : > > > >> The linecache module is used implicitly by the traceback and warnings > >> module, so perhaps quite a bit more than one would imagine :-) > >> > >> I think limiting the linecache cache size would be

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Olemis Lang
On 10/6/13, Benjamin Peterson wrote: > 2013/10/6 Victor Stinner : >> Hi, >> :) [...] >> >> unittest doesn't look to release memory (the TestCase class) after the >> execution of a test. > > Is it important to optimize unittests for memory usage? > AFAICT , test results will stored the outcomes

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Victor Stinner
2013/10/6 : > Quoting Antoine Pitrou : > >> The linecache module is used implicitly by the traceback and warnings >> module, so perhaps quite a bit more than one would imagine :-) >> >> I think limiting the linecache cache size would be good enough. > > So what specific limit would you suggest? B

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Antoine Pitrou
On Sun, 06 Oct 2013 22:47:27 +0200 mar...@v.loewis.de wrote: > > Quoting Antoine Pitrou : > > > The linecache module is used implicitly by the traceback and warnings > > module, so perhaps quite a bit more than one would imagine :-) > > > > I think limiting the linecache cache size would be good

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread martin
Quoting Antoine Pitrou : The linecache module is used implicitly by the traceback and warnings module, so perhaps quite a bit more than one would imagine :-) I think limiting the linecache cache size would be good enough. So what specific limit would you suggest? Regards, Martin _

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread martin
Quoting Victor Stinner : Slowly, I'm trying to see if it would be possible to reduce the memory footprint of Python using the tracemalloc module. [...] Should I open a separated issue for each idea to track them in the bug tracker, or a global issue? There is a third alternative which I wou

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Antoine Pitrou
On Sun, 06 Oct 2013 22:27:52 +0200 mar...@v.loewis.de wrote: > > Quoting Benjamin Peterson : > > >> If you know that your application uses a lot of memory, it is > >> interesting to sometimes (when the application is idle) try to release > >> some bytes to not use all the system memory. On embedd

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread martin
Quoting Benjamin Peterson : If you know that your application uses a lot of memory, it is interesting to sometimes (when the application is idle) try to release some bytes to not use all the system memory. On embedded devices, memory is expensive and very limited. Each byte is important :-) H

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Victor Stinner
Benjamin wrote: > Is it important to optimize unittests for memory usage? 2013/10/6 Georg Brandl : > That does not seem very important, except if people execute test_import > on every interpreter startup :) Oh, I just realized that I forgot to explain why I'm starting with unit tests. I ran the P

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Stefan Behnel
Benjamin Peterson, 06.10.2013 17:41: > 2013/10/6 Victor Stinner: >> unittest doesn't look to release memory (the TestCase class) after the >> execution of a test. > > Is it important to optimize unittests for memory usage? Maybe "optimise" isn't the best word, but test suites can sometimes be hug

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Benjamin Peterson
2013/10/6 Victor Stinner : > 2013/10/6 Benjamin Peterson : >> 2013/10/6 Victor Stinner : >>> Hi, >>> >>> Slowly, I'm trying to see if it would be possible to reduce the memory >>> footprint of Python using the tracemalloc module. >>> >>> First, I noticed that linecache can allocate more than 2 MB.

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Victor Stinner
2013/10/6 Benjamin Peterson : > 2013/10/6 Victor Stinner : >> Hi, >> >> Slowly, I'm trying to see if it would be possible to reduce the memory >> footprint of Python using the tracemalloc module. >> >> First, I noticed that linecache can allocate more than 2 MB. What do >> you think of adding a reg

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Georg Brandl
Am 06.10.2013 17:32, schrieb Victor Stinner: > Hi, > > Slowly, I'm trying to see if it would be possible to reduce the memory > footprint of Python using the tracemalloc module. > > First, I noticed that linecache can allocate more than 2 MB. What do > you think of adding a registry of "clear cac

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Antoine Pitrou
Hi, On Sun, 6 Oct 2013 17:32:37 +0200 Victor Stinner wrote: > > Slowly, I'm trying to see if it would be possible to reduce the memory > footprint of Python using the tracemalloc module. > > First, I noticed that linecache can allocate more than 2 MB. What do > you think of adding a registry o

Re: [Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Benjamin Peterson
2013/10/6 Victor Stinner : > Hi, > > Slowly, I'm trying to see if it would be possible to reduce the memory > footprint of Python using the tracemalloc module. > > First, I noticed that linecache can allocate more than 2 MB. What do > you think of adding a registry of "clear cache" functions? For >

[Python-Dev] Reduce memory footprint of Python

2013-10-06 Thread Victor Stinner
Hi, Slowly, I'm trying to see if it would be possible to reduce the memory footprint of Python using the tracemalloc module. First, I noticed that linecache can allocate more than 2 MB. What do you think of adding a registry of "clear cache" functions? For exemple, re.purge() and linecache.clearc