[Matplotlib-users] Possible memory leak?
Matplotlib Users:
It seems matplotlib plotting has a relatively small memory leak. My
experiments suggest it leaks between 5K and 8K bytes of RAM for ever plot
redraw. For example, in one experiment, plotting the same buffer (so as to
not
allocate new memory) every second for a period of about 12 hours resulted in
memory usage (physical RAM) increasing by approximately 223MB, which is
about
5.3K per replot. The plotting code is:
class PlotPanel(wx.Panel):
def __init__(self, parent):
wx.Panel.__init__(self, parent, wx.ID_ANY,
style=wx.BORDER_THEME|wx.TAB_TRAVERSAL)
self._figure = MplFigure(dpi=None)
self._canvas = MplCanvas(self, -1, self._figure)
self._axes = self._figure.add_subplot(1,1,1)
sizer = wx.BoxSizer(wx.VERTICAL)
sizer.Add(self._canvas, 1, wx.EXPAND|wx.TOP, 5)
self.SetSizer(sizer)
def draw(self, channel, seconds):
self._axes.clear()
self._axes.plot(channel, seconds)
self._canvas.draw()
`draw()` is called every second with the same `channels` and `seconds`
numpy.array buffers.
In my case, this leak, though relatively small, becomes a serious issue
since
my software often runs for long periods of time (days) plotting data
streamed
from a data acquisition unit.
Any suggestions will help. Am I miss understanding something here? Maybe I
need to call some obscure function to free memory, or something?
My testing environment:
* Windws XP SP3, Intel Core 2 Duo @ 2.33GHz, 1.96 GB RAM
* Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit
(Intel)] on win32
* matplotlib version 1.0.0
* numpy 1.4.1
* wxPython version 2.8.11.0
The complete test program follows.
Thanks,
Caleb
from random import random
from datetime import datetime
import os
import time
import win32api
import win32con
import win32process
import wx
import numpy
import matplotlib as mpl
from matplotlib.figure import Figure as MplFigure
from matplotlib.backends.backend_wxagg import FigureCanvasWxAgg as MplCanvas
def get_process_memory_info(process_id):
memory = {}
process = None
try:
process = win32api.OpenProcess(
win32con.PROCESS_QUERY_INFORMATION|win32con.PROCESS_VM_READ,
False, process_id);
if process is not None:
return win32process.GetProcessMemoryInfo(process)
finally:
if process:
win32api.CloseHandle(process)
return memory
meg = 1024.0 * 1024.0
class PlotPanel(wx.Panel):
def __init__(self, parent):
wx.Panel.__init__(self, parent, wx.ID_ANY,
style=wx.BORDER_THEME|wx.TAB_TRAVERSAL)
self._figure = MplFigure(dpi=None)
self._canvas = MplCanvas(self, -1, self._figure)
self._axes = self._figure.add_subplot(1,1,1)
sizer = wx.BoxSizer(wx.VERTICAL)
sizer.Add(self._canvas, 1, wx.EXPAND|wx.TOP, 5)
self.SetSizer(sizer)
def draw(self, channel, seconds):
self._axes.clear()
self._axes.plot(channel, seconds)
self._canvas.draw()
class TestFrame(wx.Frame):
def __init__(self, parent, id, title):
wx.Frame.__init__(
self, parent, id, title, wx.DefaultPosition, (600, 400))
self.testDuration = 60 * 60 * 24
self.startTime = 0
self.channel = numpy.sin(numpy.arange(1000) * random())
self.seconds = numpy.arange(len(self.channel))
self.plotPanel = PlotPanel(self)
sizer = wx.BoxSizer(wx.VERTICAL)
sizer.Add(self.plotPanel, 1 ,wx.EXPAND)
self.SetSizer(sizer)
self._timer = wx.Timer(self)
self.Bind(wx.EVT_TIMER, self._onTimer, self._timer)
self._timer.Start(1000)
print "starting memory: ",\
get_process_memory_info(os.getpid())["WorkingSetSize"]/meg
def _onTimer(self, evt):
if self.startTime == 0:
self.startTime = time.time()
if (time.time() - self.startTime) >= self.testDuration:
self._timer.Stop()
self.plotPanel.draw(self.channel, self.seconds)
t = datetime.now()
memory = get_process_memory_info(os.getpid())
print "time: {0}, working: {1:f}".format(
t, memory["WorkingSetSize"]/meg)
class MyApp(wx.App):
def OnInit(self):
frame = TestFrame(None, wx.ID_ANY, "Memory Leak")
self.SetTopWindow(frame)
frame.Show(True)
return True
if __name__ == '__main__':
app = MyApp(0)
app.MainLoop()
--
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today
http://p.sf.net/sfu/msIE9-sfdev2dev___
Matplotlib-users maili
Re: [Matplotlib-users] Possible memory leak?
This picks up from a thread of the same name between 18 Nov 2010 and 22 Nov 2010. Release 1.0.1 of matplotlib has made significant gains in reducing the memory leak (thanks!!), but it did not eliminate the problem entirely. Recall, the TkAgg back-end does not have any leak, so we know this particular leak is in matplotlib or wxPython. Here are the results of some tests. Matplotlib 1.0.0 - 1 hour - Plotted 3595 times, about 1Hz - Memory usage increased by about 18.7MB (59.96 - 41.25), or about 5.3K per redraw. Matplotlib 1.0.1 - 1 hour - Plotted 3601 times, about 1Hz - Memory usage increased by about 1.4MB (42.98 - 41.59), or about 0.40K per redraw. - 12 hour - Plotted 43201 times, about 1Hz - Memory usage increased by about 13.3MB (54.32 - 41.01), or about 0.32K per redraw. As stated before, for a process plotting data for long periods of time, this becomes an issue. Caleb -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Matplotlib-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Possible memory leak?
On Tue, Apr 19, 2011 at 1:01 PM, Michael Droettboom wrote: > There's a lot of moving parts here. Running your script again is > showing some leaks in valgrind that weren't there before, but a number > of the underlying libraries have changed on my system since then (memory > leaks tend to be Whac-a-mole sometimes...) > > Which versions of the following are you running, and on what platform -- > some variant of MS-Windows if I recall correctly? > > Python > Numpy > wxPython > Tkinter Windows XP SP 3 Python - 2.6.6 Numpy - 1.4.1 wxPython - 2.8.11.0 Tkinter - $Revision: 73770 $ I'll install new versions of Numpy and wxPython (and maybe Python) and try again. -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Matplotlib-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Possible memory leak?
On Tue, Apr 19, 2011 at 2:25 PM, Michael Droettboom wrote: > Ok. I have a RHEL5 Linux box with Python 2.7.1. > > With Numpy 1.4.1 and 1.5.1 I don't see any leaks. With Numpy git HEAD, > I did see a leak -- I submitted a pull request to Numpy here: > > https://github.com/numpy/numpy/pull/76 > > I get the same results (no leaks) running your wx, tk and agg scripts > (with the Windows-specific stuff removed). > > FWIW, I have wxPython 2.8.11.0 and Tkinter rev 81008. > > So the variables are the platform and the version of Python. Perhaps > it's one of those two things? > > Mike Consider the following: matplotlib 1.0.1, numpy 1.5.1, python 2.7.1, wxPython 2.8.11.0, Windows XP SP3 - 1 hour - Plotted 3601 times, about 1Hz - Memory usage increased by about 1.16MB (41.39 - 40.23), or about 0.33K per redraw It seems the same memory leak exists. Given you don't have this issue on Linux with the same Python configuration, I can only assume it is related to some Windows specific code somewhere. I'll run for a longer period of time just in case, but I don't expect the results to be different. -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Matplotlib-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Possible memory leak?
On Wed, Apr 20, 2011 at 9:29 AM, Michael Droettboom wrote: > On 04/20/2011 07:48 AM, Caleb Constantine wrote: >> On Tue, Apr 19, 2011 at 2:25 PM, Michael Droettboom wrote: >>> Ok. I have a RHEL5 Linux box with Python 2.7.1. >>> >>> With Numpy 1.4.1 and 1.5.1 I don't see any leaks. With Numpy git HEAD, >>> I did see a leak -- I submitted a pull request to Numpy here: >>> >>> https://github.com/numpy/numpy/pull/76 >>> >>> I get the same results (no leaks) running your wx, tk and agg scripts >>> (with the Windows-specific stuff removed). >>> >>> FWIW, I have wxPython 2.8.11.0 and Tkinter rev 81008. >>> >>> So the variables are the platform and the version of Python. Perhaps >>> it's one of those two things? >>> >>> Mike >> Consider the following: >> >> matplotlib 1.0.1, numpy 1.5.1, python 2.7.1, wxPython 2.8.11.0, >> Windows XP SP3 >> >> - 1 hour >> - Plotted 3601 times, about 1Hz >> - Memory usage increased by about 1.16MB (41.39 - 40.23), or >> about 0.33K per redraw >> >> It seems the same memory leak exists. Given you don't have this issue >> on Linux with the same Python configuration, I can only assume it is >> related to some Windows specific code somewhere. I'll run for a longer >> period of time just in case, but I don't expect the results to be >> different. > One way to rule out Windows-specific code may be to run with the Agg > backend only (without wx). Have you plotted the memory growth? This > amount of memory growth is well within the pool allocation sizes that > Python routinely uses. Does the value of len(gc.get_objects()) grow > over time? > New results follows. matplotlib 1.0.1, numpy 1.5.1, python 2.7.1, wxPython 2.8.11.0, Windows XP SP3 agg - 3601 redraws (1 hour), about 1Hz - Memory usage: 28.79 - 27.57 = 1.22 MB - len(gc.get_objects()): 23424 at beginning and end - Plot of memory growth: roughly linear, increasing with slope of 0.26KB tkagg - 3601 redraws (1 hour), about 1Hz - Memory usage: 33.22 - 33.32 = -0.1 MB - len(gc.get_objects()): 24182 at beginning and end - Plot of memory growth: very irregular (up and down), but a line fit has a slope of about 0.025KB (I could run longer and see if slope approaches 0) wxagg - 3601 redraws (1 hour), about 1Hz - Memory usage: 43.28 - 41.80 = 1.5 MB - len(gc.get_objects()): 41473 at beginning and end - Plot of memory growth: roughly linear, increasing with slope of 0.32KB -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Matplotlib-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Possible memory leak?
On Wed, Apr 20, 2011 at 1:04 PM, Michael Droettboom wrote: > On 04/20/2011 11:27 AM, Caleb Constantine wrote: >> On Wed, Apr 20, 2011 at 9:29 AM, Michael Droettboom wrote: >>> On 04/20/2011 07:48 AM, Caleb Constantine wrote: >>>> On Tue, Apr 19, 2011 at 2:25 PM, Michael Droettboom >>>> wrote: >>>>> Ok. I have a RHEL5 Linux box with Python 2.7.1. >>>>> >>>>> With Numpy 1.4.1 and 1.5.1 I don't see any leaks. With Numpy git HEAD, >>>>> I did see a leak -- I submitted a pull request to Numpy here: >>>>> >>>>> https://github.com/numpy/numpy/pull/76 >>>>> >>>>> I get the same results (no leaks) running your wx, tk and agg scripts >>>>> (with the Windows-specific stuff removed). >>>>> >>>>> FWIW, I have wxPython 2.8.11.0 and Tkinter rev 81008. >>>>> >>>>> So the variables are the platform and the version of Python. Perhaps >>>>> it's one of those two things? >>>>> >>>>> Mike >>>> Consider the following: >>>> >>>> matplotlib 1.0.1, numpy 1.5.1, python 2.7.1, wxPython 2.8.11.0, >>>> Windows XP SP3 >>>> >>>> - 1 hour >>>> - Plotted 3601 times, about 1Hz >>>> - Memory usage increased by about 1.16MB (41.39 - 40.23), or >>>> about 0.33K per redraw >>>> >>>> It seems the same memory leak exists. Given you don't have this issue >>>> on Linux with the same Python configuration, I can only assume it is >>>> related to some Windows specific code somewhere. I'll run for a longer >>>> period of time just in case, but I don't expect the results to be >>>> different. >>> One way to rule out Windows-specific code may be to run with the Agg >>> backend only (without wx). Have you plotted the memory growth? This >>> amount of memory growth is well within the pool allocation sizes that >>> Python routinely uses. Does the value of len(gc.get_objects()) grow >>> over time? >>> >> New results follows. >> >> matplotlib 1.0.1, numpy 1.5.1, python 2.7.1, wxPython 2.8.11.0, Windows XP >> SP3 >> >> agg >> - 3601 redraws (1 hour), about 1Hz >> - Memory usage: 28.79 - 27.57 = 1.22 MB >> - len(gc.get_objects()): 23424 at beginning and end >> - Plot of memory growth: roughly linear, increasing with slope of 0.26KB >> >> tkagg >> - 3601 redraws (1 hour), about 1Hz >> - Memory usage: 33.22 - 33.32 = -0.1 MB >> - len(gc.get_objects()): 24182 at beginning and end >> - Plot of memory growth: very irregular (up and down), but a line fit >> has a slope of about 0.025KB (I could run longer and see if slope >> approaches 0) >> >> wxagg >> - 3601 redraws (1 hour), about 1Hz >> - Memory usage: 43.28 - 41.80 = 1.5 MB >> - len(gc.get_objects()): 41473 at beginning and end >> - Plot of memory growth: roughly linear, increasing with slope of 0.32KB > Thanks. These are very useful results. > > The fact that gc.get_objects() remains constant suggests to me that this > is not a simple case of holding on to a Python reference longer than we > intend to. Instead, this is either a C-side reference counting bug, or > a genuine C malloc-and-never-free bug. Puzzlingly, valgrind usually > does a very good job of finding such bugs, but is turning up nothing for > me. Will have to scratch my head a little bit longer and see if I can > come up with a proper experiment that will help me get to the bottom of > this. > For completeness, I ran more tests over a 10 hour period at an increased redraw rate. Details follows. Note tkagg memory usage is flat, agg and wxagg are not. matplotlib 1.0.1, numpy 1.5.1, python 2.7.1, wxPython 2.8.11.0, Windows XP SP3 agg - 52214 redraws - Memory usage: 27.55 - 43.46 = 15.22 MB - len(gc.get_objects()): 23424 at beginning and end - Plot of memory growth: linear, increasing with slope of 0.31KB tkagg - 71379 redraws - Memory usage: 30.47 - 30.25 = 0.22 MB - len(gc.get_objects()): 24171 at beginning, 24182 at end, but mostly constant at 24182 - Plot of memory growth: very irregular (up and down), but a line fit has a slope of about 0.0002KB. wxagg - 72001 redraws - Memory usage: 62.08 - 40.10 = 21.98 MB - len(gc.get_objects()): 41473 at beginning and end - Plot of memory growth: linear, increasing with slope of 0.31KB -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Matplotlib-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Possible memory leak?
On Thu, Apr 21, 2011 at 2:07 PM, Michael Droettboom wrote: > Ok. I think I've found a leak in the way the spines' paths were being > updated. > > https://github.com/matplotlib/matplotlib/pull/89 > > Can you apply the patch there and let me know how it improves things for > you? > > Cheers, > Mike > I applied the patch and (at least for a short run), it seems the leak for wxagg has been fixed. The new results for wxagg is pretty much the same as the old results for tkagg (which does not have the leak): a plot of memory usage is very irregular (up and down), but a line fit has a slope of about 0.009KB. I suspect if I ran this for a few hours the slope would approach 0 KB. Thanks so much!! This is a life saver for my project! Caleb -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ Matplotlib-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/matplotlib-users
