[Matplotlib-users] hist doesn't like 2d arrays
Dear Folks, I'm finding that hist has problems computing on 2d arrays. import numpy import pylab mu, sigma = 2, 0.5 v = numpy.random.normal(mu,sigma,16) pylab.hist(v, bins=1000, normed=1) This works without any problems. But if you try this: w=v.reshape(400,400) pylab.hist(w, bins=1000, normed=1) it doesn't come back on my machine until all of memory is used up. However: n,bins = numpy.histogram(w,bins=1000,normed=1) works just fine. -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] memory usage with repeated imshow
Tom, I just went through this, though with version 1.01 of mpl, so it may be different. You can read the very long thread at: http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg20031.html Those who maintain mpl don't think there is a memory leak. What I found was that imshow() does consume a lot of memory (now fixed in the development version) and that the first 2 or so uses build on each other, but after that it levels off giving back memory after close(). There is a discrepancy between what python reports it's using and what the OS reports (I had 500MB from the OS, but only 150MB from python). There is a chance that ipython is caching your results (try ipython -pylab -cs 0), but when I ran without ipython, python still had a large portion of memory. -robert On 2/9/2011 3:52 PM, Tom Dimiduk wrote: I am using matplotlib pylab in association with ipython -pylab to show many large (~2000x2000 or larger) images. Each time I show another image it consumes more memory until eventually exhausting all system memory and making my whole system unresponsive. The easiest way to replicate this behaviour is with a = ones((,)) imshow(a) optionally close() and then imshow(a) again. I am using ipython .10.1 and matplotlib 0.99.3. Is there something I should be doing differently to avoid this problem? Is it fixed in a later version? Thanks, Tom -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Trouble with imshow
On 2/3/2011 10:06 AM, Eric Firing wrote:
On 02/02/2011 10:17 PM, Eric Firing wrote:
On 02/02/2011 08:38 PM, Robert Abiad wrote:
[...]
I'll put it in as an enhancement, but I'm still unsure if there is a
bug in
there as well. Is there something I should be doing to clear memory
after the
first figure is closed other than close()? I don't understand why
memory usage
grows each time I replot, but I'm pretty sure it isn't desireable
behavior. As
I mentioned, this effect is worse with plot.
So is this a bug or improper usage?
I'm not quite sure, but I don't think there is a specifically matplotlib
memory leak bug at work here. Are you using ipython, and if so, have you
turned off the caching? In its default mode, ipython keeps lots of
references, thereby keeping memory in use. Also, memory management and
reporting can be a bit tricky and misleading.
Nevertheless, the attached script may be illustrating the problem. Try
running it from the command line as-is (maybe shorten the loop--it
doesn't take 100 iterations to show the pattern) and then commenting out
the line as indicated in the comment. It seems that if anything is done
that adds ever so slightly to memory use while the figure is displayed,
then when the figure is closed, its memory is not reused. I'm puzzled.
I wasn't thinking straight--there is no mystery and no memory leak.
Ignore my example script referred to above. It was saving rows of the z
array, not single elements as I had intended, so of course memory use
was growing substantially.
Eric
You may not see a memory leak, but I still can't get my memory back without
killing python. I
turned off the ipython caching and even ran without iPython on both Windows and
Ubuntu, but when I
use imshow(), followed by close('all') and another imshow(), I run out of
memory. I can see from
the OS that the memory does not come back after close() and that it grows after
the second imshow().
Any other ideas? Looks like a bug to me otherwise.
-robert
--
The modern datacenter depends on network connectivity to access resources
and provide services. The best practices for maximizing a physical server's
connectivity to a physical network are well understood - see how these
rules translate into the virtual world?
http://p.sf.net/sfu/oracle-sfdevnlfb
___
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users
[Matplotlib-users] Trouble with imshow
Hello All, I'm very new to python, so bear with me. I'd like to use python to do my image processing, but I'm running into behavior that doesn't make sense to me. I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy. I'm loading in a fits file that's 26 MB (~16 Mpixels). When I load my image in ImageJ, I can see memory usage go up by 50MB, but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each time. If I close the figure and replot it, imshow() crashes. I don't know if I'm doing something wrong, or if it's a new or known bug. I tried the same thing on Linux and got the same result. Here's a transcript. Welcome to pylab, a matplotlib-based Python environment. For more information, type 'help(pylab)'. In [1]: import pyfits In [2]: from Tkinter import * In [3]: import tkFileDialog In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename()) In [5]: imshow(image) Out[5]: matplotlib.image.AxesImage object at 0x03BCA170 In [6]: close() In [7]: imshow(image,origin='lower') Out[7]: matplotlib.image.AxesImage object at 0x0440E170 In [8]: close() In [9]: imshow(image[100:3600,100:3600],origin='lower') Out[9]: matplotlib.image.AxesImage object at 0x045D9FB0 In [10]: Exception in Tkinter callback Traceback (most recent call last): File C:\app\Python2.6\lib\lib-tk\Tkinter.py, line 1410, in __call__ return self.func(*args) File C:\app\Python2.6\lib\lib-tk\Tkinter.py, line 495, in callit func(*args) File C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py, line 263, in idle_draw self.draw() File C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py, line 248, in draw FigureCanvasAgg.draw(self) File C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py, line 394, in draw self.figure.draw(self.renderer) File C:\app\Python2.6\lib\site-packages\matplotlib\artist.py, line 55, in draw_wrapper draw(artist, renderer, *args, **kwargs) File C:\app\Python2.6\lib\site-packages\matplotlib\figure.py, line 798, in draw func(*args) File C:\app\Python2.6\lib\site-packages\matplotlib\artist.py, line 55, in draw_wrapper draw(artist, renderer, *args, **kwargs) File C:\app\Python2.6\lib\site-packages\matplotlib\axes.py, line 1946, in draw a.draw(renderer) File C:\app\Python2.6\lib\site-packages\matplotlib\artist.py, line 55, in draw_wrapper draw(artist, renderer, *args, **kwargs) File C:\app\Python2.6\lib\site-packages\matplotlib\image.py, line 354, in draw im = self.make_image(renderer.get_image_magnification()) File C:\app\Python2.6\lib\site-packages\matplotlib\image.py, line 569, in make_image transformed_viewLim) File C:\app\Python2.6\lib\site-packages\matplotlib\image.py, line 201, in _get_unsampled_image x = self.to_rgba(self._A, self._alpha) File C:\app\Python2.6\lib\site-packages\matplotlib\cm.py, line 193, in to_rgba x = self.norm(x) File C:\app\Python2.6\lib\site-packages\matplotlib\colors.py, line 820, in __call__ result = (val-vmin) / (vmax-vmin) File C:\app\Python2.6\lib\site-packages\numpy\ma\core.py, line 3673, in __div__ return divide(self, other) File C:\app\Python2.6\lib\site-packages\numpy\ma\core.py, line 1077, in __call__ m |= filled(domain(da, db), True) File C:\app\Python2.6\lib\site-packages\numpy\ma\core.py, line 772, in __call__ return umath.absolute(a) * self.tolerance = umath.absolute(b) MemoryError Thanks for any help, -robert -- Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)! Finally, a world-class log management solution at an even better price-free! Download using promo code Free_Logger_4_Dev2Dev. Offer expires February 28th, so secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsight-sfd2d ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Trouble with imshow
On 2/2/2011 3:59 PM, Christoph Gohlke wrote: On 2/2/2011 3:33 PM, Robert Abiad wrote: Hello All, I'm very new to python, so bear with me. I'd like to use python to do my image processing, but I'm running into behavior that doesn't make sense to me. I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy. I'm loading in a fits file that's 26 MB (~16 Mpixels). When I load my image in ImageJ, I can see memory usage go up by 50MB, but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each time. If I close the figure and replot it, imshow() crashes. I don't know if I'm doing something wrong, or if it's a new or known bug. I tried the same thing on Linux and got the same result. Here's a transcript. Welcome to pylab, a matplotlib-based Python environment. For more information, type 'help(pylab)'. In [1]: import pyfits In [2]: from Tkinter import * In [3]: import tkFileDialog In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename()) In [5]: imshow(image) Out[5]:matplotlib.image.AxesImage object at 0x03BCA170 In [6]: close() In [7]: imshow(image,origin='lower') Out[7]:matplotlib.image.AxesImage object at 0x0440E170 In [8]: close() In [9]: imshow(image[100:3600,100:3600],origin='lower') Out[9]:matplotlib.image.AxesImage object at 0x045D9FB0 In [10]: Exception in Tkinter callback Traceback (most recent call last): File C:\app\Python2.6\lib\lib-tk\Tkinter.py, line 1410, in __call__ return self.func(*args) File C:\app\Python2.6\lib\lib-tk\Tkinter.py, line 495, in callit func(*args) File C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py, line 263, in idle_draw self.draw() File C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py, line 248, in draw FigureCanvasAgg.draw(self) File C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py, line 394, in draw self.figure.draw(self.renderer) File C:\app\Python2.6\lib\site-packages\matplotlib\artist.py, line 55, in draw_wrapper draw(artist, renderer, *args, **kwargs) File C:\app\Python2.6\lib\site-packages\matplotlib\figure.py, line 798, in draw func(*args) File C:\app\Python2.6\lib\site-packages\matplotlib\artist.py, line 55, in draw_wrapper draw(artist, renderer, *args, **kwargs) File C:\app\Python2.6\lib\site-packages\matplotlib\axes.py, line 1946, in draw a.draw(renderer) File C:\app\Python2.6\lib\site-packages\matplotlib\artist.py, line 55, in draw_wrapper draw(artist, renderer, *args, **kwargs) File C:\app\Python2.6\lib\site-packages\matplotlib\image.py, line 354, in draw im = self.make_image(renderer.get_image_magnification()) File C:\app\Python2.6\lib\site-packages\matplotlib\image.py, line 569, in make_image transformed_viewLim) File C:\app\Python2.6\lib\site-packages\matplotlib\image.py, line 201, in _get_unsampled_image x = self.to_rgba(self._A, self._alpha) File C:\app\Python2.6\lib\site-packages\matplotlib\cm.py, line 193, in to_rgba x = self.norm(x) File C:\app\Python2.6\lib\site-packages\matplotlib\colors.py, line 820, in __call__ result = (val-vmin) / (vmax-vmin) File C:\app\Python2.6\lib\site-packages\numpy\ma\core.py, line 3673, in __div__ return divide(self, other) File C:\app\Python2.6\lib\site-packages\numpy\ma\core.py, line 1077, in __call__ m |= filled(domain(da, db), True) File C:\app\Python2.6\lib\site-packages\numpy\ma\core.py, line 772, in __call__ return umath.absolute(a) * self.tolerance= umath.absolute(b) MemoryError Thanks for any help, -robert These are previous discussions on the issue: http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg14727.html http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg19815.html http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg19614.html Christoph The first 2 discussions lead to suggestions of more memory on a 64-bit installation, but that doesn't seem like a great solution. I use other image processing software (ImageJ and IDL) and neither has any trouble with my images. As I mentioned ImageJ uses 1/10th the memory for the same display, or about 1 byte of memory for 1 byte of image. I think matplotlib should aim for the same. I also think it should free up memory when the image is closed, but maybe I'm not doing the right thing. Is there something else I should be doing to free up memory? Things are even worse with plot. I'll file a bug report as Benjamin suggests. -robert -- Special Offer
Re: [Matplotlib-users] Trouble with imshow
On 2/2/2011 6:06 PM, Eric Firing wrote: On 02/02/2011 03:08 PM, Robert Abiad wrote: On 2/2/2011 3:59 PM, Christoph Gohlke wrote: On 2/2/2011 3:33 PM, Robert Abiad wrote: Hello All, I'm very new to python, so bear with me. I'd like to use python to do my image processing, but I'm running into behavior that doesn't make sense to me. I'm using Windows 7 Pro (64-bit) with 4 gigs of memory, python 2.6.6, and the newest versions of ipython, pyfits, matplotlib (1.0.1), numpy (1.5.1), scipy. I'm loading in a fits file that's 26 MB (~16 Mpixels). When I load my image in ImageJ, I can see memory usage go up by 50MB, but when I try displaying the image using imshow(), my memory usage goes up by around 500MB, each time. If I close the figure and replot it, imshow() crashes. I don't know if I'm doing something wrong, or if it's a new or known bug. I tried the same thing on Linux and got the same result. Here's a transcript. Welcome to pylab, a matplotlib-based Python environment. For more information, type 'help(pylab)'. In [1]: import pyfits In [2]: from Tkinter import * In [3]: import tkFileDialog In [4]: image=pyfits.getdata(tkFileDialog.askopenfilename()) In [5]: imshow(image) Out[5]:matplotlib.image.AxesImage object at 0x03BCA170 In [6]: close() In [7]: imshow(image,origin='lower') Out[7]:matplotlib.image.AxesImage object at 0x0440E170 In [8]: close() In [9]: imshow(image[100:3600,100:3600],origin='lower') Out[9]:matplotlib.image.AxesImage object at 0x045D9FB0 In [10]: Exception in Tkinter callback Traceback (most recent call last): File C:\app\Python2.6\lib\lib-tk\Tkinter.py, line 1410, in __call__ return self.func(*args) File C:\app\Python2.6\lib\lib-tk\Tkinter.py, line 495, in callit func(*args) File C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py, line 263, in idle_draw self.draw() File C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_tkagg.py, line 248, in draw FigureCanvasAgg.draw(self) File C:\app\Python2.6\lib\site-packages\matplotlib\backends\backend_agg.py, line 394, in draw self.figure.draw(self.renderer) File C:\app\Python2.6\lib\site-packages\matplotlib\artist.py, line 55, in draw_wrapper draw(artist, renderer, *args, **kwargs) File C:\app\Python2.6\lib\site-packages\matplotlib\figure.py, line 798, in draw func(*args) File C:\app\Python2.6\lib\site-packages\matplotlib\artist.py, line 55, in draw_wrapper draw(artist, renderer, *args, **kwargs) File C:\app\Python2.6\lib\site-packages\matplotlib\axes.py, line 1946, in draw a.draw(renderer) File C:\app\Python2.6\lib\site-packages\matplotlib\artist.py, line 55, in draw_wrapper draw(artist, renderer, *args, **kwargs) File C:\app\Python2.6\lib\site-packages\matplotlib\image.py, line 354, in draw im = self.make_image(renderer.get_image_magnification()) File C:\app\Python2.6\lib\site-packages\matplotlib\image.py, line 569, in make_image transformed_viewLim) File C:\app\Python2.6\lib\site-packages\matplotlib\image.py, line 201, in _get_unsampled_image x = self.to_rgba(self._A, self._alpha) File C:\app\Python2.6\lib\site-packages\matplotlib\cm.py, line 193, in to_rgba x = self.norm(x) File C:\app\Python2.6\lib\site-packages\matplotlib\colors.py, line 820, in __call__ result = (val-vmin) / (vmax-vmin) File C:\app\Python2.6\lib\site-packages\numpy\ma\core.py, line 3673, in __div__ return divide(self, other) File C:\app\Python2.6\lib\site-packages\numpy\ma\core.py, line 1077, in __call__ m |= filled(domain(da, db), True) File C:\app\Python2.6\lib\site-packages\numpy\ma\core.py, line 772, in __call__ return umath.absolute(a) * self.tolerance= umath.absolute(b) MemoryError Thanks for any help, -robert These are previous discussions on the issue: http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg14727.html http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg19815.html http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg19614.html Christoph The first 2 discussions lead to suggestions of more memory on a 64-bit installation, but that doesn't seem like a great solution. I use other image processing software (ImageJ and IDL) and neither has any trouble with my images. As I mentioned ImageJ uses 1/10th the memory for the same display, or about 1 byte of memory for 1 byte of image. I think matplotlib should aim for the same. I also think it should free up memory when the image is closed, but maybe I'm not doing the right thing. Is there something else I should be doing to free up memory? Things are even
