subject:"Re\: \[matplotlib\-devel\] Memory leaks"

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Michael Droettboom


Eric Firing wrote:
Attached are runs with gtk, wx, qtagg, and tkagg.  Quite a variety of 
results: tkagg is best, with only slow memory growth and a constant 
number of python objects; qtagg grows by 2.2k per loop, with no 
increase in python object count; wx (which is built on gtk) consumes 
3.5k per loop, with an increasing object count; gtk consumes 1.8k per 
loop with an increasing object count.


All runs are on stock ubuntu feisty python 2.5.
Thanks for these results.  Unfortunately, I'm seeing different results 
here.  [dagnabbit!]  None of them have an increasing object count for 
me, which leads me to suspect there's some version difference between 
your environment and mine that isn't being accounted for.


Gtk[Agg|Cairo] -- 1.3k per loop.
Wx[Agg] -- 0.010k per loop
QtAgg -- 2.3k per loop (which is in the same ballpark as your result)
Qt4Agg -- 1.4k per loop (which seems to be in the same ballpark as 
Darren Dale's result)

TkAgg -- 0.29k per loop

I don't know if the size of memory per loop is directly comparable 
between your environment and mine, but certainly the shape of the curve, 
and whether the number of Python objects is growing is very relevant.


I made some more commits to SVN on 07/03/07 necessary for recent 
versions of gtk+ and qt.  Did you (by any chance) not get those 
patches?  It would also be interesting to know which versions of the 
toolkits you have, as they are probably different from mine.  Is it safe 
to assume that they are all the stock Ubuntu feisty packages?  In any 
case, I have updated memleak_gui.py to display the relevant toolkit 
versions.  I've also attached a script to display the toolkit versions.  
Its output on my machine is:


# pygtk version: (2, 10, 4), gtk version: (2, 10, 9)
# PyQt4 version: 4.2, Qt version 40300
# pyqt version: 3.17.2, qt version: 30303
# wxPython version: 2.8.4.0
# Tkinter version: $Revision: 50704 $, Tk version: 8.4, Tcl version: 8.4

Cheers,
Mike


toolkit_versions.py
Description: application/python
# columns are: iteration, OS memory (k), number of python objects
#
   0 963056145
  10 966556145
  20 966456145
  30 968056145
  40 969256145
  50 970656145
  60 972156145
  70 973456145
  80 974756145
  90 976156145
 100 977456145
 110 978856145
 120 980256145
 130 981556145
 140 982956145
 150 984256145
 160 985556145
 170 987056145
 180 988356145
 190 989656145
 200 991156145
 210 992356145
 220 993756145
 230 995156145
 240 996456145
 250 997856145
 260 999256145
 2701000656145
 2801001956145
 2901003256145
 3001004756145
 3101005956145
 3201007356145
 3301008656145
 3401010156145
 3501011356145
 3601012756145
 3701014056145
 3801015556145
 3901016756145
 4001018156145
 4101019456145
 4201020956145
 4301022356145
 4401023656145
 4501024956145
 4601026256145
 4701027656145
 4801028956145
 4901030356145
 5001031756145
 5101033156145
 5201034356145
 5301035856145
 5401037156145
 5501038556145
 5601039956145
 5701041156145
 5801042556145
 5901043856145
 6001045256145
 6101046556145
 6201047956145
 6301049256145
 6401050656145
 6501051956145
 6601053356145
 6701054656145
 6801056056145
 6901057456145
 7001058756145
 7101060256145
 7201061556145
 7301062856145
 7401064156145
 7501065556145
 7601066856145
 7701068256145
 7801069656145
 7901071056145
 8001072256145
 8101073856145
 8201075156145
 8301076556145
 8401077856145
 8501079356145
 8601080556145
 8701081956145
 8801083356145
 8901084656145
 9001086056145
 9101087356145
 9201088856145
 9301090056145
 9401091456145
 9501092756145
 9601094156145
 9701095556145
 9801096856145
 9901098156145
10001099556145
10101100956145
10201102356145
10301103656145
10401104956145
10501106356145
10601107756145
10701109056145
10801110556145
1090756145
11001113156145
11101114456145
11201115856145
11301117156145
11401118556145
11501119856145
11601121256145
11701122556145
11801123956145
11901125256145
12001126656145
12101127956145
12201129456145
12301130756145
12401132056145
12501133456145
126011347

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Michael Droettboom

Yes -- the global wxapp variable was removed (a very good thing).  I 
just committed a patch to fix this crash (r3460)

Cheers,
Mike

Christopher Barker wrote:
 Eric Firing wrote:
   
 I just updated from svn and tried to rerun the wx test, but ran into an 
 error:

 [EMAIL PROTECTED]:~/programs/py/mpl/tests$ python 
  wxapp.Yield()
 NameError: global name 'wxapp' is not defined
 

 I think I just saw a note that Ken had committed a patch that a user had 
 provided that kept the wx back-end from re-starting an event loop if 
 there was one already running -- maybe that has something to do with 
 this bug?

 -Chris




   


-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Christopher Barker

Eric Firing wrote:
 I just updated from svn and tried to rerun the wx test, but ran into an 
 error:
 
 [EMAIL PROTECTED]:~/programs/py/mpl/tests$ python 
  wxapp.Yield()
 NameError: global name 'wxapp' is not defined

I think I just saw a note that Ken had committed a patch that a user had 
provided that kept the wx back-end from re-starting an event loop if 
there was one already running -- maybe that has something to do with 
this bug?

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

[EMAIL PROTECTED]

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Eric Firing

Mike,

New exception:

[EMAIL PROTECTED]:~/programs/py/mpl/tests$ python 
../matplotlib_units/unit/memleak_gui.py -dwx -s1000 -e2000  
~/temp/memleak_wx_0705.asc
Traceback (most recent call last):
   File ../matplotlib_units/unit/memleak_gui.py, line 58, in module
 pylab.close(fig)
   File /usr/local/lib/python2.5/site-packages/matplotlib/pylab.py, 
line 742, in close
 _pylab_helpers.Gcf.destroy(manager.num)
   File 
/usr/local/lib/python2.5/site-packages/matplotlib/_pylab_helpers.py, 
line 28, in destroy
 figManager.destroy()
   File 
/usr/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.py, 
line 1405, in destroy
 self.frame.Destroy()
   File 
/usr/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.py, 
line 1364, in Destroy
 wxapp.Yield()
AttributeError: 'listiterator' object has no attribute 'Yield'


Eric

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Michael Droettboom

Interesting.  I don't get that, but I do get some random segfaults (I 
got lucky the first time I tested).

I'm awfully surprised that wx.GetApp() would return an iterator, as you 
are getting, so maybe it's corruption of some sort?

Reverting to revision 3441 on backend_wx.py does resolve this issue for 
me, so it is related to removing the wxapp global variable.  While I 
like the idea of removing global variables, that was problematic, since 
when the wxapp variable is dereferenced, the whole wx.App is destructed, 
(hence, I believe the segfaults).  Since I didn't want to just put the 
wxapp global variable back in, I assigned it to the figure that creates 
it, therefore stick around as long as the figure does.  (Is that the 
correct thing for its lifetime?)  Anyway, it seems to fix memleak_gui.py 
for me.  Ken and Tim will probably want to check that I didn't cause 
more mainloops to start than necessary in the process.

Also, I'm a little puzzled by this code in show() in backend_wx.py:

wxapp = wx.GetApp()
if wxapp is not None:
# wxPython 2.4 has no wx.App.IsMainLoopRunning() method
imlr = getattr(wxapp, 'IsMainLoopRunning', lambda: False)
if imlr():
wxapp.MainLoop()

If I'm reading this correctly, shouldn't it be if not imlr()?  If it 
is correct, maybe it needs a comment as to why mainloops should be 
started if a mainloop is already running.

Cheers,
Mike

Eric Firing wrote:
 Mike,

 New exception:

 [EMAIL PROTECTED]:~/programs/py/mpl/tests$ python 
 ../matplotlib_units/unit/memleak_gui.py -dwx -s1000 -e2000  
 ~/temp/memleak_wx_0705.asc
 Traceback (most recent call last):
   File ../matplotlib_units/unit/memleak_gui.py, line 58, in module
 pylab.close(fig)
   File /usr/local/lib/python2.5/site-packages/matplotlib/pylab.py, 
 line 742, in close
 _pylab_helpers.Gcf.destroy(manager.num)
   File 
 /usr/local/lib/python2.5/site-packages/matplotlib/_pylab_helpers.py, 
 line 28, in destroy
 figManager.destroy()
   File 
 /usr/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.py, 
 line 1405, in destroy
 self.frame.Destroy()
   File 
 /usr/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.py, 
 line 1364, in Destroy
 wxapp.Yield()
 AttributeError: 'listiterator' object has no attribute 'Yield'


 Eric



-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Eric Firing


Michael Droettboom wrote:
Interesting.  I don't get that, but I do get some random segfaults (I 
got lucky the first time I tested).


I'm awfully surprised that wx.GetApp() would return an iterator, as you 
are getting, so maybe it's corruption of some sort?


Reverting to revision 3441 on backend_wx.py does resolve this issue for 
me, so it is related to removing the wxapp global variable.  While I 


[...]

Works for me now, and the result is attached.  Object count is still 
climbing.


Eric
# columns are: iteration, OS memory (k), number of python objects
#
   01884975791
  101884975831
  201884975871
  301884975911
  401893075951
  501893075991
  601903876031
  701903876071
  801903876111
  901903876151
 1001912476191
 1101912476231
 1201923576271
 1301923576311
 1401931676351
 1501931676391
 1601941776431
 1701941776471
 1801941776511
 1901951376551
 2001951376591
 2101951376631
 2201961276671
 2301961276711
 2401961276751
 2501971076791
 2601971076831
 2701971076871
 2801980076911
 2901980076951
 3001980076991
 3101989377031
 3201989377071
 3301989377111
 3401997877151
 3501997877191
 3602008877231
 3702008877271
 3802008877311
 3902019277351
 4002019277391
 4102019277431
 4202019277471
 4302027477511
 4402027477551
 4502037477591
 4602037477631
 4702037477671
 4802048477711
 4902048477751
 5002048477791
 5102058877831
 5202058877871
 5302058877911
 5402058877951
 5502068377991
 5602068378031
 5702068378071
 5802076378111
 5902079678151
 6002086178191
 6102096178231
 6202096178271
 6302096178311
 6402106078351
 6502106078391
 6602106078431
 6702115678471
 6802115678511
 6902115678551
 7002125478591
 7102125478631
 7202125478671
 7302135378711
 7402135378751
 7502135378791
 7602144278831
 7702144278871
 7802144278911
 7902152878951
 8002152878991
 8102163879031
 8202163879071
 8302163879111
 8402163879151
 8502173379191
 8602173379231
 8702173379271
 8802182079311
 8902182079351
 9002193179391
 9102193179431
 9202193179471
 9302193179511
 9402201579551
 9502201579591
 9602212679631
 9702212679671
 9802212679711
 9902212679751
10002220779791
10102220779831
10202229879871
10302229879911
10402233579951
10502240179991
10602250380031
10702250380071
10802250380111
10902260180151
11002260180191
11102260180231
11202269980271
11302269980311
11402269980351
11502279880391
11602279880431
11702279880471
11802289780511
11902289780551
12002289780591
12102299580631
12202299580671
12302299580711
12402308680751
12502308680791
12602308680831
12702318080871
12802318080911
12902328380951
13002328380991
13102328381031
13202328381071
1330233668
13402336681151
13502347581191
13602347581231
13702347581271
13802347581311
13902355281351
14002355281391
14102364381431
14202364381471
14302359581511
14402370081551
14502380881591
14602380881631
14702380881671
14802378781711
14902385281751
15002395181791
15102404881831
15202404881871
15302404881911
15402414681951
15502414681991
15602414682031
15702424782071
15802424782111
15902424782151
16002434482191
16102434482231
16202434482271
16302444282311
16402444282351
16502444282391
16602454282431
16702454282471
16802454282511
16902463982551
17002463982591
17102463982631
17202473882671
17302473882711
17402473882751
17502483782791
17602483782831
17702483782871
17802493582911
17902493582951
18002493582991
18102502583031
18202502583071
18302502583111
18402511883151
18502511883191
18602510483231
18702517083271
18802528183311
18902528183351
19002528183391
19102538583431
1920

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Michael Droettboom


Interesting...

When you get a chance, would you mind running the attached script?  This 
is how I was finding object leaks before.  It takes a single commandline 
argument that is the number of iterations.  Can you send me the outputs 
from 1 and 2 iterations?  That way we should be able to see what type of 
object is being leaked, which is a good first step.


If that doesn't make it immediately obvious, I'll try this on my Ubuntu 
box at home and see if I can reproduce what you're seeing.


Cheers,
Mike

Eric Firing wrote:

Michael Droettboom wrote:
Interesting.  I don't get that, but I do get some random segfaults (I 
got lucky the first time I tested).


I'm awfully surprised that wx.GetApp() would return an iterator, as 
you are getting, so maybe it's corruption of some sort?


Reverting to revision 3441 on backend_wx.py does resolve this issue 
for me, so it is related to removing the wxapp global variable.  While I 


[...]

Works for me now, and the result is attached.  Object count is still 
climbing.


Eric


# columns are: iteration, OS memory (k), number of python objects
#
   01884975791
  101884975831
  201884975871
  301884975911
  401893075951
  501893075991
  601903876031
  701903876071
  801903876111
  901903876151
 1001912476191
 1101912476231
 1201923576271
 1301923576311
 1401931676351
 1501931676391
 1601941776431
 1701941776471
 1801941776511
 1901951376551
 2001951376591
 2101951376631
 2201961276671
 2301961276711
 2401961276751
 2501971076791
 2601971076831
 2701971076871
 2801980076911
 2901980076951
 3001980076991
 3101989377031
 3201989377071
 3301989377111
 3401997877151
 3501997877191
 3602008877231
 3702008877271
 3802008877311
 3902019277351
 4002019277391
 4102019277431
 4202019277471
 4302027477511
 4402027477551
 4502037477591
 4602037477631
 4702037477671
 4802048477711
 4902048477751
 5002048477791
 5102058877831
 5202058877871
 5302058877911
 5402058877951
 5502068377991
 5602068378031
 5702068378071
 5802076378111
 5902079678151
 6002086178191
 6102096178231
 6202096178271
 6302096178311
 6402106078351
 6502106078391
 6602106078431
 6702115678471
 6802115678511
 6902115678551
 7002125478591
 7102125478631
 7202125478671
 7302135378711
 7402135378751
 7502135378791
 7602144278831
 7702144278871
 7802144278911
 7902152878951
 8002152878991
 8102163879031
 8202163879071
 8302163879111
 8402163879151
 8502173379191
 8602173379231
 8702173379271
 8802182079311
 8902182079351
 9002193179391
 9102193179431
 9202193179471
 9302193179511
 9402201579551
 9502201579591
 9602212679631
 9702212679671
 9802212679711
 9902212679751
10002220779791
10102220779831
10202229879871
10302229879911
10402233579951
10502240179991
10602250380031
10702250380071
10802250380111
10902260180151
11002260180191
11102260180231
11202269980271
11302269980311
11402269980351
11502279880391
11602279880431
11702279880471
11802289780511
11902289780551
12002289780591
12102299580631
12202299580671
12302299580711
12402308680751
12502308680791
12602308680831
12702318080871
12802318080911
12902328380951
13002328380991
13102328381031
13202328381071
1330233668
13402336681151
13502347581191
13602347581231
13702347581271
13802347581311
13902355281351
14002355281391
14102364381431
14202364381471
14302359581511
14402370081551
14502380881591
14602380881631
14702380881671
14802378781711
14902385281751
15002395181791
15102404881831
15202404881871
15302404881911
15402414681951
15502414681991
15602414682031
15702424782071
15802424782111
15902424782151
16002434482191
16102434482231
16202434482271
16302444282311
16402444282351
16502444282391

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Eric Firing

Michael Droettboom wrote:
 Interesting...
 
 When you get a chance, would you mind running the attached script?  This 
 is how I was finding object leaks before.  It takes a single commandline 
 argument that is the number of iterations.  Can you send me the outputs 
 from 1 and 2 iterations?  That way we should be able to see what type of 
 object is being leaked, which is a good first step.

[EMAIL PROTECTED]:~/programs/py/mpl/tests$ python memleak_gui_wx.py 1
75891 76010
*** class 'wx._core.PySimpleApp'
*** class 'wx._core._wxPyDeadObject'

uncollectable list: []

[EMAIL PROTECTED]:~/programs/py/mpl/tests$ python memleak_gui_wx.py 2
GnomePrintCupsPlugin-Message: The ppd file for the CUPS printer Dell424 
could not be loaded.
GnomePrintCupsPlugin-Message: The ppd file for the CUPS printer pslj4m 
could not be loaded.
75891 76014
*** class 'wx._core.PySimpleApp'
*** class 'wx._core._wxPyDeadObject'

uncollectable list: []


Eric

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Eric Firing

Michael Droettboom wrote:
 Interesting...
 
 When you get a chance, would you mind running the attached script?  This 
 is how I was finding object leaks before.  It takes a single commandline 
 argument that is the number of iterations.  Can you send me the outputs 
 from 1 and 2 iterations?  That way we should be able to see what type of 
 object is being leaked, which is a good first step.

And here is the result of the script modified for gtk:

[EMAIL PROTECTED]:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 1
55352 55417
*** type 'gtk.gdk.Colormap'
*** class 'matplotlib.backends.backend_gtk.FileChooserDialog'
*** class 'gtk._gtk.WidgetFlags'
*** class 'gtk._gtk.WidgetFlags'
*** type 'gtk.gdk.Window'
*** class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'
*** class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'
*** type 'gtk.gdk.Pixmap'
*** type 'gtk.Tooltips'
*** type 'gtk.Label'
*** type 'gtk.Window'
*** type 'gtk.VBox'

uncollectable list: []

[EMAIL PROTECTED]:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 2
55352 55421
*** type 'gtk.gdk.Colormap'
*** class 'matplotlib.backends.backend_gtk.FileChooserDialog'
*** class 'gtk._gtk.WidgetFlags'
*** class 'gtk._gtk.WidgetFlags'
*** type 'gtk.gdk.Window'
*** class 'matplotlib.backends.backend_gtk.FileChooserDialog'
*** class 'gtk._gtk.WidgetFlags'
*** class 'gtk._gtk.WidgetFlags'
*** type 'gtk.gdk.Window'
*** class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'
*** class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'
*** type 'gtk.gdk.Pixmap'
*** type 'gtk.Tooltips'
*** type 'gtk.Label'
*** type 'gtk.Window'
*** type 'gtk.VBox'

uncollectable list: []

Eric

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Ken McIvor

On Jul 5, 2007, at 2:13 PM, Michael Droettboom wrote:

 Interesting.  I don't get that, but I do get some random segfaults (I
 got lucky the first time I tested).

It looks like wxPython doesn't retain a reference to the wxApp PyObj  
for you:

[EMAIL PROTECTED]:~/Projects/matplotlib-svn$ pythonw
Python 2.4.4 (#1, Oct 18 2006, 10:34:39)
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type help, copyright, credits or license for more information.
 import wx
 app = wx.PySimpleApp()
 del app
 wx.GetApp()
Segmentation fault

 I'm awfully surprised that wx.GetApp() would return an iterator, as  
 you
 are getting, so maybe it's corruption of some sort?

My guess is that Eric got lucky and ob_type was pointing to the  
listiterator's C type instance.

 Since I didn't want to just put the wxapp global variable back in,  
 I assigned it to the figure that creates it, therefore stick around  
 as long as the figure does.  (Is that the correct thing for its  
 lifetime?)

I don't think this will work if you create two figures, destroy the  
first one, and then create another figure.  Once created, the wxApp  
needs to exist for the life of the python process.  I'll go ahead an  
put the global variable back in.

 Also, I'm a little puzzled by this code in show() in backend_wx.py:

 wxapp = wx.GetApp()
 if wxapp is not None:
 # wxPython 2.4 has no wx.App.IsMainLoopRunning() method
 imlr = getattr(wxapp, 'IsMainLoopRunning', lambda: False)
 if imlr():
 wxapp.MainLoop()

 If I'm reading this correctly, shouldn't it be if not imlr()?

Yes, it should be.  I'll try to code with my eyes open from now on.  :-/

Ken

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Michael Droettboom

Yep.  Nothing obvious.  I'll have to have a look on Ubuntu and see if 
that makes a difference.

Cheers,
Mike

Eric Firing wrote:
 Michael Droettboom wrote:
 Interesting...

 When you get a chance, would you mind running the attached script?  
 This is how I was finding object leaks before.  It takes a single 
 commandline argument that is the number of iterations.  Can you send 
 me the outputs from 1 and 2 iterations?  That way we should be able 
 to see what type of object is being leaked, which is a good first step.

 [EMAIL PROTECTED]:~/programs/py/mpl/tests$ python memleak_gui_wx.py 1
 75891 76010
 *** class 'wx._core.PySimpleApp'
 *** class 'wx._core._wxPyDeadObject'

 uncollectable list: []

 [EMAIL PROTECTED]:~/programs/py/mpl/tests$ python memleak_gui_wx.py 2
 GnomePrintCupsPlugin-Message: The ppd file for the CUPS printer 
 Dell424 could not be loaded.
 GnomePrintCupsPlugin-Message: The ppd file for the CUPS printer pslj4m 
 could not be loaded.
 75891 76014
 *** class 'wx._core.PySimpleApp'
 *** class 'wx._core._wxPyDeadObject'

 uncollectable list: []


 Eric



-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Michael Droettboom

That is at least something to go by.  ;)

Thanks,
Mike

Eric Firing wrote:
 Michael Droettboom wrote:
 Interesting...

 When you get a chance, would you mind running the attached script?  
 This is how I was finding object leaks before.  It takes a single 
 commandline argument that is the number of iterations.  Can you send 
 me the outputs from 1 and 2 iterations?  That way we should be able 
 to see what type of object is being leaked, which is a good first step.

 And here is the result of the script modified for gtk:

 [EMAIL PROTECTED]:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 1
 55352 55417
 *** type 'gtk.gdk.Colormap'
 *** class 'matplotlib.backends.backend_gtk.FileChooserDialog'
 *** class 'gtk._gtk.WidgetFlags'
 *** class 'gtk._gtk.WidgetFlags'
 *** type 'gtk.gdk.Window'
 *** class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'
 *** class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'
 *** type 'gtk.gdk.Pixmap'
 *** type 'gtk.Tooltips'
 *** type 'gtk.Label'
 *** type 'gtk.Window'
 *** type 'gtk.VBox'

 uncollectable list: []

 [EMAIL PROTECTED]:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 2
 55352 55421
 *** type 'gtk.gdk.Colormap'
 *** class 'matplotlib.backends.backend_gtk.FileChooserDialog'
 *** class 'gtk._gtk.WidgetFlags'
 *** class 'gtk._gtk.WidgetFlags'
 *** type 'gtk.gdk.Window'
 *** class 'matplotlib.backends.backend_gtk.FileChooserDialog'
 *** class 'gtk._gtk.WidgetFlags'
 *** class 'gtk._gtk.WidgetFlags'
 *** type 'gtk.gdk.Window'
 *** class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'
 *** class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'
 *** type 'gtk.gdk.Pixmap'
 *** type 'gtk.Tooltips'
 *** type 'gtk.Label'
 *** type 'gtk.Window'
 *** type 'gtk.VBox'

 uncollectable list: []

 Eric



-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Eric Firing

Ken McIvor wrote:
 On Jul 5, 2007, at 2:13 PM, Michael Droettboom wrote:

 Interesting.  I don't get that, but I do get some random segfaults (I
 got lucky the first time I tested).
 
 It looks like wxPython doesn't retain a reference to the wxApp PyObj for 
 you:
 
 [EMAIL PROTECTED]:~/Projects/matplotlib-svn$ pythonw
 Python 2.4.4 (#1, Oct 18 2006, 10:34:39)
 [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
 Type help, copyright, credits or license for more information.
  import wx
  app = wx.PySimpleApp()
  del app
  wx.GetApp()
 Segmentation fault

This qualifies as a wx bug, doesn't it?  If wx doesn't retain the 
reference, then instead of a segfault shouldn't it raise an exception?

Eric

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Ken McIvor

On Jul 5, 2007, at 3:48 PM, Eric Firing wrote:

 This qualifies as a wx bug, doesn't it?

I believe so.  I'll file it.

 If wx doesn't retain the reference, then instead of a segfault  
 shouldn't it raise an exception?

I'd expect wx.GetApp() to work like the rest of wxPython and always  
return the wx.App instance.

This has been fixed in revision 3463.

Ken

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-05 Thread Christopher Barker

Ken McIvor wrote:
 This qualifies as a wx bug, doesn't it?
 I believe so.  I'll file it.

I agree - a segfault is ALWAYS a bug.

 If wx doesn't retain the reference, then instead of a segfault  
 shouldn't it raise an exception?
 
 I'd expect wx.GetApp() to work like the rest of wxPython and always  
 return the wx.App instance.

If a wx.App has not been created, it returns None:

  import wx
  wx.GetApp()
  a = wx.GetApp()
  print a
None

Which is probably what it should do if the wxApp() has been deleted.

In any case, you can only create one wxApp per program instance, and it 
can not be destroyed and re-started, so keeping a global instance around 
is probably the way to go.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

[EMAIL PROTECTED]

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-03 Thread Michael Droettboom

Eric Firing wrote:
 I also made memleak_gui.py more flexible with arguments. For example, 
 here are tests with three backends, a generous number of loops, and 
 suppression of intermediate output:

Those changes are really helpful.  I just added code to display the 
total number of objects in the Python interpreter (len(gc.get_objects()) 
with each iteration as well, as that can be useful.  (It doesn't rule 
out memory leaks, but if it is increasing, that is definitely a problem.)

I also added a commandline option to print out any cycles involving 
uncollectable objects, and added the necessary function to do so to 
cbook.py.

Cheers,
Mike

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-03 Thread Eric Firing

Michael Droettboom wrote:
 Eric Firing wrote:
 I also made memleak_gui.py more flexible with arguments. For example, 
 here are tests with three backends, a generous number of loops, and 
 suppression of intermediate output:
 
 Those changes are really helpful.  I just added code to display the 
 total number of objects in the Python interpreter (len(gc.get_objects()) 
 with each iteration as well, as that can be useful.  (It doesn't rule 
 out memory leaks, but if it is increasing, that is definitely a problem.)
 
 I also added a commandline option to print out any cycles involving 
 uncollectable objects, and added the necessary function to do so to 
 cbook.py.
 
 Cheers,
 Mike

Mike,

Good, thank you.

I just committed a change to the output formatting of memleak_gui so 
that if you redirect it to a file, that file can be loaded with 
pylab.load() in case you want to plot the columns.  (At least this is 
true if you don't use the -c option.)

Yesterday, before your commits, I compared memleak_gui with stock Python 
2.4 versus stock 2.5 (both from ubuntu feisty) and found very little 
difference in the OS memory numbers.

Eric

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-03 Thread Michael Droettboom

Eric Firing wrote:

 I just committed a change to the output formatting of memleak_gui so 
 that if you redirect it to a file, that file can be loaded with 
 pylab.load() in case you want to plot the columns.  (At least this is 
 true if you don't use the -c option.)

Great.  Sorry for stomping on that ;)
 Yesterday, before your commits, I compared memleak_gui with stock 
 Python 2.4 versus stock 2.5 (both from ubuntu feisty) and found very 
 little difference in the OS memory numbers.
Are they still increasing linearly?  I'm still seeing some mystery leaks 
with Gtk, Qt4 and (much smaller) on Tk.  Qt and Wx seem fine here.  
Unfortunately Qt4 crashes valgrind, so it's not of much use. 

I'm curious whether your results match that.  I'm not terribly surprised 
that 2.4 isn't different from 2.5, since the case in which entire memory 
pools are freed in 2.5 is probably hard to trigger.

Cheers,
Mike

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-03 Thread Eric Firing


Michael Droettboom wrote:

Eric Firing wrote:


I just committed a change to the output formatting of memleak_gui so 
that if you redirect it to a file, that file can be loaded with 
pylab.load() in case you want to plot the columns.  (At least this is 
true if you don't use the -c option.)



Great.  Sorry for stomping on that ;)
Yesterday, before your commits, I compared memleak_gui with stock 
Python 2.4 versus stock 2.5 (both from ubuntu feisty) and found very 
little difference in the OS memory numbers.
Are they still increasing linearly?  I'm still seeing some mystery leaks 
with Gtk, Qt4 and (much smaller) on Tk.  Qt and Wx seem fine here.  


Attached are runs with gtk, wx, qtagg, and tkagg.  Quite a variety of 
results: tkagg is best, with only slow memory growth and a constant 
number of python objects; qtagg grows by 2.2k per loop, with no increase 
in python object count; wx (which is built on gtk) consumes 3.5k per 
loop, with an increasing object count; gtk consumes 1.8k per loop with 
an increasing object count.


All runs are on stock ubuntu feisty python 2.5.

Eric


Unfortunately Qt4 crashes valgrind, so it's not of much use.
I'm curious whether your results match that.  I'm not terribly surprised 
that 2.4 isn't different from 2.5, since the case in which entire memory 
pools are freed in 2.5 is probably hard to trigger.


Cheers,
Mike


# columns are: iteration, OS memory (k), number of python objects
#
   02742355194
  102744455204
  202744455214
  302744455224
  402748955234
  502756655244
  602756655254
  702756655264
  802756655274
  902756655284
 1002756655294
 1102756655304
 1202782255314
 1302782255324
 1402782255334
 1502782255344
 1602782255354
 1702782255364
 1802789755374
 1902789755384
 2002789755394
 2102789755404
 2202789755414
 2302789755424
 2402789755434
 2502789755444
 2602789755454
 2702797255464
 2802797255474
 2902797255484
 3002797255494
 3102797255504
 3202797255514
 3302797255524
 3402797255534
 3502797255544
 360279524
 3702800955564
 3802800955574
 3902800955584
 4002808755594
 4102808755604
 4202808755614
 4302834355624
 4402834355634
 4502834355644
 4602834355654
 4702834355664
 4802842455674
 4902842455684
 5002842455694
 5102842455704
 5202842455714
 5302842455724
 5402850555734
 5502850555744
 5602850555754
 5702850555764
 5802850555774
 5902850555784
 6002850555794
 6102850555804
 6202850555814
 6302850555824
 6402850555834
 6502850555844
 6602858055854
 6702858055864
 6802858055874
 6902858055884
 7002858055894
 7102858055904
 7202858055914
 7302858055924
 7402858055934
 7502883655944
 7602883655954
 7702883655964
 7802890055974
 7902890055984
 8002890055994
 8102890056004
 8202890056014
 8302890056024
 8402890056034
 8502890056044
 8602897056054
 8702897056064
 8802897056074
 8902897056084
 9002897056094
 9102897056104
 9202897056114
 9302897056124
 9402905256134
 9502905256144
 9602905256154
 9702905256164
 9802905256174
 9902905256184
10002905256194
10102905256204
10202930856214
10302930856224
10402930856234
10502928656244
10602934056254
10702934056264
10802934056274
10902934056284
11002934056294
11102934056304
11202934056314
11302934056324
11402934056334
11502937356344
11602944156354
11702944156364
11802944156374
11902944156384
12002944156394
12102944156404
12202952056414
12302952056424
12402952056434
12502952056444
12602952056454
12702952056464
12802952056474
12902952056484
13002952056494
13102950356504
13202955356514
13302953256524
13402958256534
13502983856544
13602991856554
13702991856564
13802991856574
13902991856584
14002991856594
14102991856604
14202991856614
14302991856624
14402991856634
14502991856644
14602991856654
14702991856664
14802991856674
14902998856684
15002998856694
15102998856704
15202998856714
15302996656724
1540

Re: [matplotlib-devel] Memory leaks

2007-07-03 Thread Darren Dale

On Tuesday 03 July 2007 04:33:46 pm Eric Firing wrote:
 Michael Droettboom wrote:
  Eric Firing wrote:
  I just committed a change to the output formatting of memleak_gui so
  that if you redirect it to a file, that file can be loaded with
  pylab.load() in case you want to plot the columns.  (At least this is
  true if you don't use the -c option.)
 
  Great.  Sorry for stomping on that ;)
 
  Yesterday, before your commits, I compared memleak_gui with stock
  Python 2.4 versus stock 2.5 (both from ubuntu feisty) and found very
  little difference in the OS memory numbers.
 
  Are they still increasing linearly?  I'm still seeing some mystery leaks
  with Gtk, Qt4 and (much smaller) on Tk.  Qt and Wx seem fine here.

 Attached are runs with gtk, wx, qtagg, and tkagg.  Quite a variety of
 results: tkagg is best, with only slow memory growth and a constant
 number of python objects; qtagg grows by 2.2k per loop, with no increase
 in python object count; wx (which is built on gtk) consumes 3.5k per
 loop, with an increasing object count; gtk consumes 1.8k per loop with
 an increasing object count.

 All runs are on stock ubuntu feisty python 2.5.

 Eric

  Unfortunately Qt4 crashes valgrind, so it's not of much use.
  I'm curious whether your results match that.  I'm not terribly surprised
  that 2.4 isn't different from 2.5, since the case in which entire memory
  pools are freed in 2.5 is probably hard to trigger.

I am swamped at work, and have not been able to follow this thread closely. 
But I just updated from svn and ran memleak_gui.py with qt4:

# columns are: iteration, OS memory (k), number of python objects
#
   03736453792
  103744153792
  203744153792
  303752553792
  403748353792
  503751153792
  603753953792
  703756853792
  803759653792
  903762453792
 1003765353792
# columns above are: iteration, OS memory (k), number of python objects
#
# uncollectable list: []
#
# Backend Qt4Agg, toolbar toolbar2
# Averaging over loops 30 to 100
# Memory went from 37525k to 37653k
# Average memory consumed per loop: 1.8286k bytes

Darren

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-02 Thread Michael Droettboom

Eric Firing wrote:
 So, this test is still showing problems, with similar memory 
 consumption in these three backends.
Not necessarily.  By default, Python allocates large pools from the 
operating system and then manages those pools itself (though its 
PyMalloc call).  Prior to Python 2.5, those pools were never freed.  
With Python 2.5, empty pools, when they occur, are freed back to the 
OS.  Due to fragmentation issues, even if there is enough free space in 
those pools for new objects, new pools may need to be created anyway, 
since Python objects can't be moved once they are created.  So seeing 
modest increases in memory usage during a long-running Python 
application is typical, and not something that can be avoided 
wiinaccurate at finding memory leaksthout micro-optimizing for pool 
performance (something that may be very difficult).  If memory usage is 
truly increasing in an unbounded way, then, yes, there may be problems, 
but it should eventually stabilize (though in a test such as memleak_gui 
that may take many iterations).  It's more interesting to see the curve 
of memory usage over time than the average over a number of iterations.

For further reading, see:
http://evanjones.ca/python-memory.html
README.valgrind in the Python source
http://mail.python.org/pipermail/python-dev/2006-March/061991.html

Because of this, using the total memory allocated by the Python process 
to track memory leaks is pretty blunt tool.  More important metrics are 
the total number of GC objects (gc.get_objects()), GC garbage 
(gc.garbage), and using a tool like Valgrind or Purify to find 
mismatched malloc/frees.  Another useful tool (but I didn't resort to 
yet with matplotlib testing) is to build Python with COUNT_ALLOCS, which 
then gives access to the total number of mallocs and frees in the Python 
interpreter at runtime.

IMO, the only reasonable way to use the total memory usage of Python to 
debug memory leaks is if you build Python without pool allocation 
(--without-pymalloc).  That was how I was debugging memory leaks last 
week (in conjunction with valgrind, and the gc module), and with that 
configuration, I was only seeing memory leakage with Pygtk 2.4, and a 
very small amount with Tk.  Are your numbers from a default build?  If 
so, I'll rebuild my Python and check my numbers against yours.  If they 
match, I suspect there's little we can do.

Cheers,
Mike


-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-02 Thread Eric Firing

Michael Droettboom wrote:
 Eric Firing wrote:
 So, this test is still showing problems, with similar memory 
 consumption in these three backends.
 Not necessarily.  By default, Python allocates large pools from the 
 operating system and then manages those pools itself (though its 
 PyMalloc call).  Prior to Python 2.5, those pools were never freed.  
 With Python 2.5, empty pools, when they occur, are freed back to the 
 OS.  Due to fragmentation issues, even if there is enough free space in 
 those pools for new objects, new pools may need to be created anyway, 
 since Python objects can't be moved once they are created.  So seeing 
 modest increases in memory usage during a long-running Python 
 application is typical, and not something that can be avoided 
 wiinaccurate at finding memory leaksthout micro-optimizing for pool 
 performance (something that may be very difficult).  If memory usage is 
 truly increasing in an unbounded way, then, yes, there may be problems, 
 but it should eventually stabilize (though in a test such as memleak_gui 
 that may take many iterations).  It's more interesting to see the curve 
 of memory usage over time than the average over a number of iterations.

I agree.  I just ran 2000 iterations with GtkAgg, plotted every 10th 
point, and the increase is linear (apart from a little bumpiness) over 
the entire range (not just the last 1000 iterations reported below):

Backend GTKAgg, toolbar toolbar2
Averaging over loops 1000 to 2000
Memory went from 31248k to 35040k
Average memory consumed per loop: 3.7920k bytes

Maybe this is just the behavior of pymalloc in 2.5?


 For further reading, see:
 http://evanjones.ca/python-memory.html
 README.valgrind in the Python source
 http://mail.python.org/pipermail/python-dev/2006-March/061991.html
 
 Because of this, using the total memory allocated by the Python process 
 to track memory leaks is pretty blunt tool.  More important metrics are 
 the total number of GC objects (gc.get_objects()), GC garbage 
 (gc.garbage), and using a tool like Valgrind or Purify to find 
 mismatched malloc/frees.  Another useful tool (but I didn't resort to 
 yet with matplotlib testing) is to build Python with COUNT_ALLOCS, which 
 then gives access to the total number of mallocs and frees in the Python 
 interpreter at runtime.
 
 IMO, the only reasonable way to use the total memory usage of Python to 
 debug memory leaks is if you build Python without pool allocation 
 (--without-pymalloc).  That was how I was debugging memory leaks last 
 week (in conjunction with valgrind, and the gc module), and with that 
 configuration, I was only seeing memory leakage with Pygtk 2.4, and a 
 very small amount with Tk.  Are your numbers from a default build?  If 
 so, I'll rebuild my Python and check my numbers against yours.  If they 
 match, I suspect there's little we can do.

I used stock Python 2.5 from ubuntu Feisty.  I should compile a version 
as you suggest, but I haven't done it yet.

Eric

 
 Cheers,
 Mike
 


-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-07-02 Thread Michael Droettboom

More results:

I've built and tested a more recent pygtk+ stack.  (glib-2.12, 
gtk+-2.10.9, librsvg-2.16.1, libxml2-2.6.29, pygobject-2.13.1, 
pygtk-2.10.4...).  The good news is that the C-level leaks I was seeing 
in pygtk 2.2 and 2.4 are resolved.  In particular, using an SVG icon and 
Gdk rendering no longer seems problematic.  I would suggest that anyone 
using old versions of pygtk should upgrade, rather than spending time on 
workarounds for matplotlib -- do you all agree?  And my Gtk patch should 
probably be reverted to use an SVG icon for the window again (or to only 
do it on versions of pygtk  2.xx).  I don't know what percentage of 
users are still using pygtk-2.4 and earlier...

There is, however, a new patch (attached) to fix a leak of 
FileChooserDialog objects that I didn't see in earlier pygtk versions.  
I have to admit that I'm a bit puzzled by the solution -- it seems that 
the FileChooserDialog object refuses to destruct whenever any custom 
Python attributes have been added to the object.  It doesn't really need 
them in this case so it's an easy fix, but I'm not sure why that was 
broken -- other classes do this and don't have problems (e.g. 
NavigationToolbar2GTK).  Maybe a pygtk expert out there knows what this 
is about.  It would be great if this resolved the linear memory growth 
that Eric is seeing with the Gtk backend.

GtkCairo seems to be free of leaks.

QtAgg (qt-3.3) was leaking because of a cyclical reference in the 
signals between the toolbar and its buttons.  (Patch attached).

Qt4 is forthcoming (I'm still trying to compile something that runs the 
demos cleanly). 

I tried the FltkAgg backend, but it doesn't seem to close the window at 
all when the figure is closed -- instead I get dozens of windows open at 
once.  Is that a known bug or correct behavior?

Cheers,
Mike

Eric Firing wrote:
 Michael Droettboom wrote:
 Eric Firing wrote:
 So, this test is still showing problems, with similar memory 
 consumption in these three backends.
 Not necessarily.  By default, Python allocates large pools from the 
 operating system and then manages those pools itself (though its 
 PyMalloc call).  Prior to Python 2.5, those pools were never freed.  
 With Python 2.5, empty pools, when they occur, are freed back to the 
 OS.  Due to fragmentation issues, even if there is enough free space 
 in those pools for new objects, new pools may need to be created 
 anyway, since Python objects can't be moved once they are created.  
 So seeing modest increases in memory usage during a long-running 
 Python application is typical, and not something that can be avoided 
 wiinaccurate at finding memory leaksthout micro-optimizing for pool 
 performance (something that may be very difficult).  If memory usage 
 is truly increasing in an unbounded way, then, yes, there may be 
 problems, but it should eventually stabilize (though in a test such 
 as memleak_gui that may take many iterations).  It's more interesting 
 to see the curve of memory usage over time than the average over a 
 number of iterations.

 I agree.  I just ran 2000 iterations with GtkAgg, plotted every 10th 
 point, and the increase is linear (apart from a little bumpiness) over 
 the entire range (not just the last 1000 iterations reported below):

 Backend GTKAgg, toolbar toolbar2
 Averaging over loops 1000 to 2000
 Memory went from 31248k to 35040k
 Average memory consumed per loop: 3.7920k bytes

 Maybe this is just the behavior of pymalloc in 2.5?


 For further reading, see:
 http://evanjones.ca/python-memory.html
 README.valgrind in the Python source
 http://mail.python.org/pipermail/python-dev/2006-March/061991.html

 Because of this, using the total memory allocated by the Python 
 process to track memory leaks is pretty blunt tool.  More important 
 metrics are the total number of GC objects (gc.get_objects()), GC 
 garbage (gc.garbage), and using a tool like Valgrind or Purify to 
 find mismatched malloc/frees.  Another useful tool (but I didn't 
 resort to yet with matplotlib testing) is to build Python with 
 COUNT_ALLOCS, which then gives access to the total number of mallocs 
 and frees in the Python interpreter at runtime.

 IMO, the only reasonable way to use the total memory usage of Python 
 to debug memory leaks is if you build Python without pool allocation 
 (--without-pymalloc).  That was how I was debugging memory leaks last 
 week (in conjunction with valgrind, and the gc module), and with that 
 configuration, I was only seeing memory leakage with Pygtk 2.4, and a 
 very small amount with Tk.  Are your numbers from a default build?  
 If so, I'll rebuild my Python and check my numbers against yours.  If 
 they match, I suspect there's little we can do.

 I used stock Python 2.5 from ubuntu Feisty.  I should compile a 
 version as you suggest, but I haven't done it yet.

 Eric


 Cheers,
 Mike





-
This SF.net email is

Re: [matplotlib-devel] Memory leaks

2007-07-02 Thread Michael Droettboom


Forgot to attach the patches.

Oops,
Mike

Michael Droettboom wrote:

More results:

I've built and tested a more recent pygtk+ stack.  (glib-2.12, 
gtk+-2.10.9, librsvg-2.16.1, libxml2-2.6.29, pygobject-2.13.1, 
pygtk-2.10.4...).  The good news is that the C-level leaks I was seeing 
in pygtk 2.2 and 2.4 are resolved.  In particular, using an SVG icon and 
Gdk rendering no longer seems problematic.  I would suggest that anyone 
using old versions of pygtk should upgrade, rather than spending time on 
workarounds for matplotlib -- do you all agree?  And my Gtk patch should 
probably be reverted to use an SVG icon for the window again (or to only 
do it on versions of pygtk  2.xx).  I don't know what percentage of 
users are still using pygtk-2.4 and earlier...


There is, however, a new patch (attached) to fix a leak of 
FileChooserDialog objects that I didn't see in earlier pygtk versions.  
I have to admit that I'm a bit puzzled by the solution -- it seems that 
the FileChooserDialog object refuses to destruct whenever any custom 
Python attributes have been added to the object.  It doesn't really need 
them in this case so it's an easy fix, but I'm not sure why that was 
broken -- other classes do this and don't have problems (e.g. 
NavigationToolbar2GTK).  Maybe a pygtk expert out there knows what this 
is about.  It would be great if this resolved the linear memory growth 
that Eric is seeing with the Gtk backend.


GtkCairo seems to be free of leaks.

QtAgg (qt-3.3) was leaking because of a cyclical reference in the 
signals between the toolbar and its buttons.  (Patch attached).


Qt4 is forthcoming (I'm still trying to compile something that runs the 
demos cleanly). 

I tried the FltkAgg backend, but it doesn't seem to close the window at 
all when the figure is closed -- instead I get dozens of windows open at 
once.  Is that a known bug or correct behavior?


Cheers,
Mike

Eric Firing wrote:
  

Michael Droettboom wrote:


Eric Firing wrote:
  
So, this test is still showing problems, with similar memory 
consumption in these three backends.

Not necessarily.  By default, Python allocates large pools from the 
operating system and then manages those pools itself (though its 
PyMalloc call).  Prior to Python 2.5, those pools were never freed.  
With Python 2.5, empty pools, when they occur, are freed back to the 
OS.  Due to fragmentation issues, even if there is enough free space 
in those pools for new objects, new pools may need to be created 
anyway, since Python objects can't be moved once they are created.  
So seeing modest increases in memory usage during a long-running 
Python application is typical, and not something that can be avoided 
wiinaccurate at finding memory leaksthout micro-optimizing for pool 
performance (something that may be very difficult).  If memory usage 
is truly increasing in an unbounded way, then, yes, there may be 
problems, but it should eventually stabilize (though in a test such 
as memleak_gui that may take many iterations).  It's more interesting 
to see the curve of memory usage over time than the average over a 
number of iterations.
  
I agree.  I just ran 2000 iterations with GtkAgg, plotted every 10th 
point, and the increase is linear (apart from a little bumpiness) over 
the entire range (not just the last 1000 iterations reported below):


Backend GTKAgg, toolbar toolbar2
Averaging over loops 1000 to 2000
Memory went from 31248k to 35040k
Average memory consumed per loop: 3.7920k bytes

Maybe this is just the behavior of pymalloc in 2.5?




For further reading, see:
http://evanjones.ca/python-memory.html
README.valgrind in the Python source
http://mail.python.org/pipermail/python-dev/2006-March/061991.html

Because of this, using the total memory allocated by the Python 
process to track memory leaks is pretty blunt tool.  More important 
metrics are the total number of GC objects (gc.get_objects()), GC 
garbage (gc.garbage), and using a tool like Valgrind or Purify to 
find mismatched malloc/frees.  Another useful tool (but I didn't 
resort to yet with matplotlib testing) is to build Python with 
COUNT_ALLOCS, which then gives access to the total number of mallocs 
and frees in the Python interpreter at runtime.


IMO, the only reasonable way to use the total memory usage of Python 
to debug memory leaks is if you build Python without pool allocation 
(--without-pymalloc).  That was how I was debugging memory leaks last 
week (in conjunction with valgrind, and the gc module), and with that 
configuration, I was only seeing memory leakage with Pygtk 2.4, and a 
very small amount with Tk.  Are your numbers from a default build?  
If so, I'll rebuild my Python and check my numbers against yours.  If 
they match, I suspect there's little we can do.
  
I used stock Python 2.5 from ubuntu Feisty.  I should compile a 
version as you suggest, but I haven't done it yet.


Eric



Cheers,
Mike

Re: [matplotlib-devel] Memory leaks

2007-07-02 Thread John Hunter

On 7/2/07, Michael Droettboom [EMAIL PROTECTED] wrote:
 Forgot to attach the patches.

Michael -- if you send me your sf ID I'll add you to the committers
list and you can check these in directly.

Vis-a-vis the gtk question, I agree that we should encourage people to
upgrade who are suffering from the leak rather than work around it.  I
would like to summarize the status of known leaks for the FAQ so
perhaps you could summarize across the backends what kind of leaks
remain in the --without-pymalloc with the known problems fixed (eg the
gtk upgrade).  If you could simply send me an update for the memory
leak FAQ (don't worry about the formatting, I can take care of that)
that would be great.  Or if you are feeling doubly adventurous, you
can simply update the FAQ in the htdocs/faq.html.template svn document
and commit it along with your other changes.

Thanks for all the very useful and detailed work!

JDH

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-06-30 Thread John Hunter

On 6/30/07, Eric Firing [EMAIL PROTECTED] wrote:
 Mike,

 All this sounds like great progress--thanks!  I particularly appreciate
 the descriptions of what problems you found and how you found them.

 John et al.: is there a maintainer for each of these backends?  I think

gtk: Steve Chaplin or me
wx: Ken McIvor
qt: Darren?
tk: Charlie?

After we get these patches in, we can just give Michael commit
privileges :-)  I can probably look at this Monday, but if you want to
commit and test some of these before then, please do so.

JDH

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks

2007-06-30 Thread Eric Firing

John Hunter wrote:
 On 6/30/07, Eric Firing [EMAIL PROTECTED] wrote:
 Mike,

 All this sounds like great progress--thanks!  I particularly appreciate
 the descriptions of what problems you found and how you found them.

 John et al.: is there a maintainer for each of these backends?  I think
 
 gtk: Steve Chaplin or me
 wx: Ken McIvor
 qt: Darren?
 tk: Charlie?
 
 After we get these patches in, we can just give Michael commit
 privileges :-)  I can probably look at this Monday, but if you want to
 commit and test some of these before then, please do so.

Done.  It looks like there is still plenty of memory leakage, but there 
are improvements, and the huge list of uncollectable garbage with tkAgg 
is gone.

I also made memleak_gui.py more flexible with arguments. For example, 
here are tests with three backends, a generous number of loops, and 
suppression of intermediate output:

python ../unit/memleak_gui.py -d wx -s 500 -e 1000 -q

uncollectable list: []

Backend WX, toolbar toolbar2
Averaging over loops 500 to 1000
Memory went from 29316k to 31211k
Average memory consumed per loop: 3.7900k bytes

python ../unit/memleak_gui.py -d tkagg -s 500 -e 1000 -q

uncollectable list: []

Backend TkAgg, toolbar toolbar2
Averaging over loops 500 to 1000
Memory went from 29202k to 31271k
Average memory consumed per loop: 4.1380k bytes

python ../unit/memleak_gui.py -d gtkagg -s 500 -e 1000 -q

uncollectable list: []

Backend GTKAgg, toolbar toolbar2
Averaging over loops 500 to 1000
Memory went from 29324k to 31131k
Average memory consumed per loop: 3.6140k bytes


So, this test is still showing problems, with similar memory consumption 
in these three backends.

Eric


 
 JDH

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] memory leaks

2007-03-28 Thread John Hunter

On 3/27/07, Eric Firing [EMAIL PROTECTED] wrote:
 I can add a couple of things to item (1) below.  First, the problem
 occurs only with toolbar2, not with classic or None.  Second, a script
 that illustrates it is attached.

I defintely agree that this is important -- and it is a big help to
have a script and the info that you narrowed the problem down to the
presence of the toolbar.  report_memory is platform dependent since ps
is.  I added a report_memory function to cbook so we could have some
common functionality to rely on.  So far it checks for linux or sunos5
and we should add to this and fix it up as necessary.  I also stripped
the script down to the bare essentials (the memory report) and added
it to unit/memleak_gui.py so others can use it for testing.

It turns out if you add a savefig call, TkAgg is terribly (1MB per
figure) and I can reproduce the smaller toolbar induced leak on my
platform with TkAgg and GTKAgg.  I tried adding some code to
figure.clf to help, but it didn't.  I also spent some time trying to
figure out what was going wrong with TkAgg but unfortunately did not
succeed.  I don't know anything about Tk, really.  One interesting
thing: the enormous filesave leak in TkAgg also only occurs if the
toolbar is present.  w/o the toolbar, neither gtkagg nor tkagg leak w/
or w/o the filesave.  With the toolbar, both leak a 20-80k w/o the
file save.

Developers: if you know something about a particular GUI, try this
script with -dYourGUIBackend and see if you can isolate the problem!

JDH

# in svn as unit/memleak_gui.py

#!/usr/bin/env python
'''
This illustrates a leak that occurs with any interactive backend.
Run with

  python memleak_gui.py -dGTKAgg   # or TkAgg, etc..

You may need to edit cbook.report_memory to support your platform

'''
import os, sys, time
import gc
import matplotlib

#matplotlib.use('TkAgg') # or TkAgg or WxAgg or QtAgg or Gtk
matplotlib.rcParams['toolbar'] = 'toolbar2'   # None, classic, toolbar2
#matplotlib.rcParams['toolbar'] = None   # None, classic, toolbar2

import pylab
from matplotlib import _pylab_helpers as ph
import matplotlib.cbook as cbook

indStart, indEnd = 30, 50
for i in range(indEnd):

fig = pylab.figure()
fig.savefig('test')
fig.clf()
pylab.close(fig)
val = cbook.report_memory(i)
print i, val
gc.collect()
if i==indStart: start = val # wait a few cycles for memory usage
to stabilize

gc.collect()
print
print 'uncollectable list:', gc.garbage
print
end = val
if i  indStart:
print 'Average memory consumed per loop: %1.4fk bytes\n' %
((end-start)/float(indEnd-indStart))

-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT  business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] memory leaks

2007-03-28 Thread Jeff Whitaker

John Hunter wrote:
 On 3/28/07, Jeff Whitaker [EMAIL PROTECTED] wrote:

 John:  I just added macos x support in the report_memory function.
 Regarding Eric's memory leak #2 (which occurs even for non-gui
 backends), here's a simple script to trigger it:


 Thanks Jeff, could you add this to the unit dir as well?

 JDH
Done - added as memleak_nongui.py

-Jeff

-- 
Jeffrey S. Whitaker Phone  : (303)497-6313
Meteorologist   FAX: (303)497-6449
NOAA/OAR/PSD  R/PSD1Email  : [EMAIL PROTECTED]
325 BroadwayOffice : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web: http://tinyurl.com/5telg


-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT  business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] memory leaks

2007-03-28 Thread Jeff Whitaker

Tom Holroyd (NIH/NIMH) [E] wrote:
 import os,matplotlib
 matplotlib.use('Agg')
 from matplotlib.figure import Figure
 from matplotlib.cbook import report_memory

 def plot():
 fig = Figure()
 i = 0
 while True:
 print report_memory(i)
 fig.clf()
 ax = fig.add_axes([0.1,0.1,0.7,0.7])
 ax.plot([1,2,3])
 i += 1

 if __name__ == '__main__': plot()

 I have matplotlib-0.90.0 installed, and this script doesn't leak for 
 me. It grows a bit as shown in the graph, then stabilizes. I'm on FC4 
 with Python 2.4.3.

 



Right - here too (on macos x), it levels off about about 15 times the 
initial memory usage.  I just didn't run it long enough to notice that 
before.

-Jeff

-- 
Jeffrey S. Whitaker Phone  : (303)497-6313
Meteorologist   FAX: (303)497-6449
NOAA/OAR/PSD  R/PSD1Email  : [EMAIL PROTECTED]
325 BroadwayOffice : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web: http://tinyurl.com/5telg


-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT  business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] memory leaks

2007-03-27 Thread Eric Firing

I can add a couple of things to item (1) below.  First, the problem 
occurs only with toolbar2, not with classic or None.  Second, a script 
that illustrates it is attached.


Eric

Eric Firing wrote:

In 2007, two different major memory leaks have been identified:

1) Eric Pellegrini showed that a loop over figure(); close() leaks.  I 
have verified that this occurs with any interactive backend, but not 
with non-interactive backends.  This may be the same problem that was 
reported in other messages, such as one by Dylan Passmore in January.


2) There is a recent thread Re: Memory leak in basemap or matplotlib 
showing that even with a non-interactive backend, a seemingly-pointless 
call to cla() is needed to prevent a leak.


I would like to suggest that we try harder to solve these problems ASAP. 
  This kind of malfunctioning at the core of mpl worries me.


I have spent quite a bit of time trying to figure out (1), and I have 
tracked it down to the NavigationToolbar2.  Eliminate the toolbar by 
putting None in the rc slot, and the memory leak vanishes.  It looks to 
me like some explicit call to a destroy method may be needed to 
dismantle the toolbar when a figure is closed and/or deleted.  I suspect 
that each gui toolkit is keeping references to components, and the 
toolbar is not getting the word when the window is destroyed. 
gc.garbage verifies that the toolbar components are what get left behind.


So, I hope a gui toolkit backend wizard can step forward and say, Of 
course, we just need to put a __del__ method here with a call to 
destroy(), or something like that.


I have spent much less time on (2), and made no progress.

We are relying very heavily on the gc--mpl has cyclic references all 
over the place.  Is anyone sure that we don't need explicit gc support 
in any of the extension code?  Can *everything* in the extension code be 
handled correctly with reference counting?  Is this independent of how 
things defined in extension code are used at the python level?


It is not clear to me that gc debugging methods even allow one to see 
problems in extension code that do not have some degree of gc support. 
The standard documentation of the gc module and the gc C API is minimal.


Eric

-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT  business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel


#!/usr/bin/env python
'''
This illustrates a leak that occurs with any interactive backend.
'''
import os, sys, time
import gc
import matplotlib

matplotlib.use('TkAgg') # or TkAgg or WxAgg or QtAgg or Gtk
matplotlib.rcParams['toolbar'] = 'toolbar2'   # None, classic, toolbar2

import pylab
from matplotlib import _pylab_helpers as ph

def report_memory(i):
pid = os.getpid()
a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
print i, '  ', a2[1],
return int(a2[1].split()[1])

gc.disable()
#gc.set_debug(gc.DEBUG_LEAK)
gc.set_debug(gc.DEBUG_UNCOLLECTABLE
| gc.DEBUG_INSTANCES | gc.DEBUG_OBJECTS)

# take a memory snapshot on indStart and compare it with indEnd
indStart, indEnd = 30, 50
for i in range(indEnd):

fig = pylab.figure()
print 'after figure, unreachable:', gc.collect()
print 'total uncollectable:', len(gc.garbage)
#report_memory(i)
#fig.clear()
#pylab.cla()  #no effect
#pylab.clf()   # no effect
#pylab.show()   # makes it worse
pylab.close()
#del(fig)
#gc.collect()  #no effect
val = report_memory(i)
print 'after close, unreachable:', gc.collect()
print 'total uncollectable:', len(gc.garbage)
del(fig)
print 'after del(fig), unreachable:', gc.collect()
print 'total uncollectable:',len(gc.garbage)
if i==indStart: start = val # wait a few cycles for memory usage to stabilize
gc.collect()
print
print 'uncollectable list:', gc.garbage
print
end = val
if i  indStart:
print 'Average memory consumed per loop: %1.4fk bytes\n' % ((end-start)/float(indEnd-indStart))

-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT  business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Memory leaks in Basemap

2006-12-01 Thread Jeff Whitaker

Aalok kapoor wrote:
 Thanks for reply,

 This was just an example for mem leak  in my application. Actually i 
 want maps with different data to be loaded each time and the map 
 objects taken once changees after ploting so i have to take different 
 object each time. After around 20 maps generation it gives memory 
 error to me.

 thanks,
 -Aalok

 */Jeff Whitaker [EMAIL PROTECTED]/* wrote:

 Aalok kapoor wrote:
  Hi all,
 
  I am using matplotlib-0.87.7, Basemap-0.9.3, Numpy-1.1 and agg
 backend.
  I am facing memory leak problems after running following script -
 
 
  import os, sys, time
  import cStringIO
  import Image
  import matplotlib
  matplotlib.use('Agg')
 
  import matplotlib.pylab as p
  import matplotlib.numerix as nx
  from matplotlib.toolkits.basemap import Basemap
 
  def get_image_bytes(width_fig, height_fig, str_img):
  Returns the bytes representing the image generated
 
  The createImage method should be called before calling this method.
 
  :Returns:
  Byte data.
  
  # Get the actual image data
  f = cStringIO.StringIO()
  dims = (width_fig, height_fig)
  im = Image.fromstring(RGB, dims, str_img)
  im.save(f, JPEG)
  return f.getvalue()
 
  def report_memory(i):
  pid = os.getpid()
  a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
  print i, ' ', a2[1],
  return int(a2[1].split()[1])
 
 
 
  # take a memory snapshot on indStart and compare it with indEnd
  indStart, indEnd = 30, 150
  print ' rss vsz %mem'
  for i in range(indEnd):
  p.figure(figsize=(float(640)/80.0,float(480)/80.0),
  facecolor='w', edgecolor='w')
  p.axes([0.0, 0.0, 1.0, 1.0])
  map1 = Basemap(projection='cyl', lat_0=50, lon_0=-100,
  area_thresh=1000.)
  # draw coastlines, country boundaries, fill continents.
  map1.drawcoastlines(linewidth=.5)
  map1.drawcountries()
  map1.fillcontinents(color=#CEFFCE)
  # draw the edge of the map projection region (the projection limb)
  map1.drawmapboundary()
  # compute the native map projection coordinates for cities.
  lats, lons = ([18.728], [20.890])
  x,y = map1(lons,lats)
  # plot filled circles at the locations of the cities.
  map1.plot(x,y,'bo')
  canvas = p.get_current_fig_manager().canvas
  agg = canvas.switch_backends \
  (matplotlib.backends.backend_agg.FigureCanvasAgg)
  agg.draw() # Takes ~.9 seconds
  # get the image data
  str_img = agg.tostring_rgb()
  width_fig, height_fig = map(int, agg.figure.bbox.get_bounds()[2:])
  p.cla()
  p.close()
  bytes = get_image_bytes(width_fig, height_fig, str_img)
 
 
  val = report_memory(i)
  # wait a few cycles for memory usage to stabilize
  if i==indStart: start = val
 
  end = val
  print 'Average memory consumed per loop: %1.4fk bytes' % \
  ((end-start)/float(indEnd-indStart))
 
 
 
 
  Here is the O/P
 
  $python memory_leak_map.py
  rss vsz %mem
  0 47272 50632 9.7
  1 74412 77700 15.3
  2 93960 97380 19.3
  3 113308 116776 23.3
  4 132824 136416 27.3
  5 152352 155828 31.3
  6 171860 175216 35.3
  7 191372 194868 39.3
  8 210872 214248 43.3
  9 230336 233916 47.3
  10 249732 253284 51.3
  11 269252 272692 55.3
  12 288680 292336 59.3
  13 308108 311724 63.2
  14 305160 331112 62.6
  15 301096 350764 61.8
  16 304884 370160 62.6
  17 298276 389804 61.2
  18 305876 409184 62.8
  19 298316 428596 61.2
  20 307856 448224 63.2
  21 308004 467640 63.2
  22 308844 487016 63.4
  23 306260 506656 62.9
  24 300612 526052 61.7
  Traceback (most recent call last):
  File memory_leak_map.py, line 41, in ?
  area_thresh=1000.)
  File
 
 
 /usr/local/lib/python2.4/site-packages/matplotlib/toolkits/basemap/basemap.py,

  line 815, in __init__
  self.riversegs = segments+segments2+segments3
  MemoryError
  Killed: 9
 
 
  Can someone please help me out solving this problem?
 
 
  Thanks
  -Aalok
 
 Aaolk: All your plots use the same map projection, yet you are
 initializing a new Basemap instance every time through the loop. Try
 moving the line

 map1 = Basemap(projection='cyl', lat_0=50, lon_0=-100,
 area_thresh=1000.)

 outside the loop.

 -Jeff


Aaolk:  Here's what I get with matplotlib 0.87.7, basemap 0.9.4, numpy 
1.0 on MacOS 10.4.9 with python 2.5 (python 2.4 gives the same result):

[mac28:~/matplotlib/toolkits/basemap] jsw% python memleak.py
 rss vsz %mem
028248   153200 -1.3
134028   157316 -1.6
234908   158084 -1.7
335688   158852 -1.7
436200   159364 -1.7
536712   159876 -1.8
636456   159620 -1.7
736712   159876 -1.8
836712

Re: [matplotlib-devel] Memory leaks in Basemap

2006-12-01 Thread Aalok kapoor

After using numpy-1.0 the results are more bad. After 15 maps it is reaching to 
71% memory usage. I am working on freeBSD box.

python memory_leak_map.py
 rss   vsz%mem
047940 50636  9.8
175080 77700 15.4
294636 97380 19.4
3114156 116776 23.4
4133672 136416 27.4
5153204 155832 31.4
6172708 175216 35.5
7192224 194872 39.5
8211720 214248 43.5
9231244 233916 47.5
10250732 253280 51.5
11270252 272688 55.5
12289756 292332 59.5
13309272 311728 63.5
14328764 331108 67.5
15347804 350760 71.4



thanks,
-Aalok
Aalok kapoor [EMAIL PROTECTED] wrote: Hi,

Thanks for this direction. I am using Numpy-1.1. Do I not supposed to use this 
version? I will try to use older version and see what happens. I hope this will 
solve the problem.

thanks
-Aalok

Jeff Whitaker [EMAIL PROTECTED] wrote: Aalok kapoor wrote:
 Thanks for reply,

 This was just an example for mem leak  in my application. Actually i 
 want maps with different data to be loaded each time and the map 
 objects taken once changees after ploting so i have to take different 
 object each time. After around 20 maps generation it gives memory 
 error to me.

 thanks,
 -Aalok

 */Jeff Whitaker /* wrote:

 Aalok kapoor wrote:
  Hi all,
 
   I am using matplotlib-0.87.7, Basemap-0.9.3, Numpy-1.1 and agg
 backend.
  I am facing memory leak problems after running following script -
 
 
  import os, sys, time
  import cStringIO
  import Image
  import matplotlib
  matplotlib.use('Agg')
 
  import matplotlib.pylab as p
  import matplotlib.numerix as nx
  from matplotlib.toolkits.basemap import Basemap
 
  def get_image_bytes(width_fig, height_fig, str_img):
  Returns the bytes representing the image generated
 
  The createImage method should be called before calling this method.
 
  :Returns:
  Byte data.
  
  # Get the actual image data
  f =  cStringIO.StringIO()
  dims = (width_fig, height_fig)
  im = Image.fromstring(RGB, dims, str_img)
  im.save(f, JPEG)
  return f.getvalue()
 
  def report_memory(i):
  pid = os.getpid()
  a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
  print i, ' ', a2[1],
  return int(a2[1].split()[1])
 
 
 
  # take a memory snapshot on indStart and compare it with indEnd
  indStart, indEnd = 30, 150
  print ' rss vsz %mem'
  for i in range(indEnd):
  p.figure(figsize=(float(640)/80.0,float(480)/80.0),
  facecolor='w', edgecolor='w')
  p.axes([0.0, 0.0, 1.0, 1.0])
  map1 = Basemap(projection='cyl', lat_0=50, lon_0=-100,
   area_thresh=1000.)
  # draw coastlines, country boundaries, fill continents.
  map1.drawcoastlines(linewidth=.5)
  map1.drawcountries()
  map1.fillcontinents(color=#CEFFCE)
  # draw the edge of the map projection region (the projection limb)
  map1.drawmapboundary()
  # compute the native map projection coordinates for cities.
  lats, lons = ([18.728], [20.890])
  x,y = map1(lons,lats)
  # plot filled circles at the locations of the cities.
  map1.plot(x,y,'bo')
  canvas = p.get_current_fig_manager().canvas
  agg = canvas.switch_backends \
  (matplotlib.backends.backend_agg.FigureCanvasAgg)
  agg.draw() # Takes ~.9 seconds
  # get the image data
  str_img = agg.tostring_rgb()
  width_fig, height_fig =  map(int, agg.figure.bbox.get_bounds()[2:])
  p.cla()
  p.close()
  bytes = get_image_bytes(width_fig, height_fig, str_img)
 
 
  val = report_memory(i)
  # wait a few cycles for memory usage to stabilize
  if i==indStart: start = val
 
  end = val
  print 'Average memory consumed per loop: %1.4fk bytes' % \
  ((end-start)/float(indEnd-indStart))
 
 
 
 
  Here is the O/P
 
  $python memory_leak_map.py
  rss vsz %mem
  0 47272 50632 9.7
  1 74412 77700 15.3
  2 93960 97380 19.3
  3 113308 116776 23.3
  4 132824 136416 27.3
  5 152352 155828 31.3
  6 171860 175216 35.3
   7 191372 194868 39.3
  8 210872 214248 43.3
  9 230336 233916 47.3
  10 249732 253284 51.3
  11 269252 272692 55.3
  12 288680 292336 59.3
  13 308108 311724 63.2
  14 305160 331112 62.6
  15 301096 350764 61.8
  16 304884 370160 62.6
  17 298276 389804 61.2
  18 305876 409184 62.8
  19 298316 428596 61.2
  20 307856 448224 63.2
  21 308004 467640 63.2
  22 308844 487016 63.4
  23 306260 506656 62.9
  24 300612 526052 61.7
  Traceback (most recent call last):
  File memory_leak_map.py, line 41, in ?
  area_thresh=1000.)
  File
 
 
 /usr/local/lib/python2.4/site-packages/matplotlib/toolkits/basemap/basemap.py,

   line 815, in __init__
  self.riversegs = segments+segments2+segments3
  MemoryError

Re: [matplotlib-devel] Memory leaks in Basemap

2006-12-01 Thread Curt Bridges

Jeff and Aalok,
  I am in central MO and it doesn't bother me but, for some unknown reason I'm 
recieving correspondense between you two. You might want to start a new email 
instead of replying  to each other.  Have a nice day. Curt Bridges  

Aalok kapoor [EMAIL PROTECTED] wrote:
  After using numpy-1.0 the results are more bad. After 15 maps it is reaching 
to 71% memory usage. I am working on freeBSD box.

python memory_leak_map.py
 rss   vsz%mem
047940 50636  9.8
175080 77700 15.4
294636 97380 19.4
3114156 116776 23.4
4133672 136416 27.4
5153204 155832 31.4
6172708 175216 35.5
7192224 194872 39.5
8211720 214248 43.5
9231244 233916 47.5
10250732 253280 51.5
11270252 272688 55.5
12289756 292332 59.5
13309272 311728 63.5
14328764 331108 67.5
15347804 350760 71.4



thanks,
-Aalok
Aalok kapoor [EMAIL PROTECTED] wrote:  Hi,

Thanks for this direction. I am using Numpy-1.1. Do I not supposed to use this 
version? I will try to use older version and see what happens. I hope this will 
solve the problem.

thanks
-Aalok

Jeff Whitaker [EMAIL PROTECTED] wrote:  Aalok kapoor wrote:
 Thanks for reply,

 This was just an example for mem leak in my application. Actually i 
 want maps with different data to be loaded each time and the map 
 objects taken once changees after ploting so i have to take different 
 object each time. After around 20 maps generation it gives memory 
 error to me.

 thanks,
 -Aalok

 */Jeff Whitaker /* wrote:

 Aalok kapoor wrote:
  Hi all,
 
  I am using matplotlib-0.87.7, Basemap-0.9.3, Numpy-1.1 and agg
 backend.
  I am facing memory leak problems after running following script -
 
 
  import os, sys, time
  import cStringIO
  import Image
  import matplotlib
  matplotlib.use('Agg')
 
  import matplotlib.pylab as p
  import matplotlib.numerix as nx
  from matplotlib.toolkits.basemap import Basemap
 
  def get_image_bytes(width_fig, height_fig, str_img):
  Returns the bytes representing the image generated
 
  The createImage method should be called before calling this method.
 
  :Returns:
  Byte data.
  
  # Get the actual image data
  f = cStringIO.StringIO()
  dims = (width_fig, height_fig)
  im = Image.fromstring(RGB, dims, str_img)
  im.save(f, JPEG)
  return f.getvalue()
 
  def report_memory(i):
  pid = os.getpid()
  a2 = os.popen('ps -p %d -o rss,vsz,%%mem' % pid).readlines()
  print i, ' ', a2[1],
  return int(a2[1].split()[1])
 
 
 
  # take a memory snapshot on indStart and compare it with indEnd
  indStart, indEnd = 30, 150
  print ' rss vsz %mem'
  for i in range(indEnd):
  p.figure(figsize=(float(640)/80.0,float(480)/80.0),
  facecolor='w', edgecolor='w')
  p.axes([0.0, 0.0, 1.0, 1.0])
  map1 = Basemap(projection='cyl', lat_0=50, lon_0=-100,
  area_thresh=1000.)
  # draw coastlines, country boundaries, fill continents.
  map1.drawcoastlines(linewidth=.5)
  map1.drawcountries()
  map1.fillcontinents(color=#CEFFCE)
  # draw the edge of the map projection region (the projection limb)
  map1.drawmapboundary()
  # compute the native map projection coordinates for cities.
  lats, lons = ([18.728], [20.890])
  x,y = map1(lons,lats)
  # plot filled circles at the locations of the cities.
  map1.plot(x,y,'bo')
  canvas = p.get_current_fig_manager().canvas
  agg = canvas.switch_backends \
  (matplotlib.backends.backend_agg.FigureCanvasAgg)
  agg.draw() # Takes ~.9 seconds
  # get the image data
  str_img = agg.tostring_rgb()
  width_fig, height_fig = map(int, agg.figure.bbox.get_bounds()[2:])
  p.cla()
  p.close()
  bytes = get_image_bytes(width_fig, height_fig, str_img)
 
 
  val = report_memory(i)
  # wait a few cycles for memory usage to stabilize
  if i==indStart: start = val
 
  end = val
  print 'Average memory consumed per loop: %1.4fk bytes' % \
  ((end-start)/float(indEnd-indStart))
 
 
 
 
  Here is the O/P
 
  $python memory_leak_map.py
  rss vsz %mem
  0 47272 50632 9.7
  1 74412 77700 15.3
  2 93960 97380 19.3
  3 113308 116776 23.3
  4 132824 136416 27.3
  5 152352 155828 31.3
  6 171860 175216 35.3
  7 191372 194868 39.3
  8 210872 214248 43.3
  9 230336 233916 47.3
  10 249732 253284 51.3
  11 269252 272692 55.3
  12 288680 292336 59.3
  13 308108 311724 63.2
  14 305160 331112 62.6
  15 301096 350764 61.8
  16 304884 370160 62.6
  17 298276 389804 61.2
  18 305876 409184 62.8
  19 298316 428596 61.2
  20 307856 448224 63.2
  21 308004 467640 63.2
  22 308844 487016 63.4
  23 306260 506656 62.9
  24 300612 526052 61.7
  Traceback (most recent call last):
  File memory_leak_map.py, line 41, in ?
  area_thresh=1000.)
  File
 
 /usr/local/lib/python2.4/site-packages/matplotlib/toolkits/basemap/basemap.py,

  line 815, in __init__
  self.riversegs = segments+segments2+segments3
  MemoryError
  Killed: 9
 
 
  Can someone please help me out solving this problem?
 
 
  Thanks
  -Aalok
 
 Aaolk: All your plots use the same map projection, yet you are
 initializing a new Basemap instance every time

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] Memory leaks

Re: [matplotlib-devel] memory leaks

Re: [matplotlib-devel] memory leaks

Re: [matplotlib-devel] memory leaks

Re: [matplotlib-devel] memory leaks

Re: [matplotlib-devel] Memory leaks in Basemap

Re: [matplotlib-devel] Memory leaks in Basemap

Re: [matplotlib-devel] Memory leaks in Basemap

34 matches

Site Navigation

Mail list logo

Footer information