Re: [Matplotlib-users] Memory issues with imshow
Perry Greenfield wrote: > To give an idea, when you ask matplotlib to render an image, it > processes it (resamples, rescales, maps to colors, etc) in order to > actually display it. Since it may redo all that if you resize or > otherwise re-render the figure, it needs to keep a reference to the > original image. Even if you delete your reference to it, it still has > it, and thus it won't be deleted until the figure is cleared. So if the > input to the imshow call is the full size array, you will have that > around. You may want to downsample that image to lower resolution (and > make sure that the downsampled version is a copy, not a view of the > original array). This is what I already did, at least that was what I thought. I calculated a downsampled image, but ignored it ... :( So now I was able to do the stitching of >30 images with a resolution of 1224x1632, yielding a total picture of 6600 x 5450 pixels, with 600 dpi. Thanks for your help! Cheers, Gerd -- Dr. Gerd Wellenreuther beamline scientist P06 "Hard X-Ray Micro/Nano-Probe" Petra III project HASYLAB at DESY Notkestr. 85 22603 Hamburg Tel.: + 49 40 8998 5701 -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Memory issues with imshow
Hello, switching to 64-bit Python and OS might help. I can display 8bit images up to 8459x8459 with imshow on Windows 7 64-bit with 8GB RAM. Python then uses about 5.5 GB RAM according to task manager. A 8460x8460 or larger 8bit images crash Python (definitely a bug). The 32-bit interpreter starts throwing MemoryErrors around 4350x4350 pixels. The exact limits will depend on the application, OS, and free RAM. Christoph On 12/15/2009 9:15 AM, Wellenreuther, Gerd wrote: > Well, I am trying to create an overlay, *one* picture showing all 34 > images. So I am only trying to create a single figure. > > I just attached an example so you can get an idea (it was downsampled > for mailing, the original picture has ca. 5500 x 6500 pixels). In the > end, I just want to save the image to the disk, so I am using 'Agg' as > the backend - hope this also saves me some memory. > > And about "old" images: I am always starting a completely new > python-process for each stitching (one at a time). > > Cheers, Gerd > > Perry Greenfield wrote: >> Are you clearing the figure after each image display? The figure >> retains references to the image if you don't do a clf() and thus you >> will eventually run out of memory, even if you delete the images (they >> don't go away while matplotlib is using them). >> >> Perry >> >> On Dec 15, 2009, at 10:32 AM, Wellenreuther, Gerd wrote: >> >>> Dear all, >>> >>> I am trying to write a script to be used with our microscope, stitching >>> images of various magnifications together to yield a big picture of a >>> sample. The preprocessing involves operations like rotating the picture >>> etc., and finally those pictures are being plotted using imshow. >>> >>> Unfortunately, I am running into memory problems, e.g.: >>> C:\Python26\lib\site-packages\PIL\Image.py:1264: DeprecationWarning: integer argument expected, got float im = self.im.stretch(size, resample) Traceback (most recent call last): File "F:\Procs\Find_dendrites.py", line 1093, in file_type="PNG",do_stitching=do_stitching,do_dendrite_finding=do_dendrite_finding,down_sizing_factor=48,dpi=600) File "F:\Procs\Find_dendrites.py", line 1052, in process_images scale,aspect_ratio,dpi,left,right,bottom,top) File "F:\Procs\Find_dendrites.py", line 145, in stitch_images pylab.draw() File "C:\Python26\lib\site-packages\matplotlib\pyplot.py", line 352, in draw get_current_fig_manager().canvas.draw() File "C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py", line 313, in draw self.renderer = self.get_renderer() File "C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py", line 324, in get_renderer self.renderer = RendererAgg(w, h, self.figure.dpi) File "C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py", line 59, in __init__ self._renderer = _RendererAgg(int(width), int(height), dpi, debug=False) RuntimeError: Could not allocate memory for image >>> >>> or >>> Traceback (most recent call last): File "F:\Procs\Find_dendrites.py", line 1093, in file_type="PNG",do_stitching=do_stitching,do_dendrite_finding=do_dendrite_finding,down_sizing_factor=48,dpi=75) File "F:\Procs\Find_dendrites.py", line 1052, in process_images scale,aspect_ratio,dpi,left,right,bottom,top) File "F:\Procs\Find_dendrites.py", line 142, in stitch_images pylab.imshow(rotated_images[i],aspect='auto') File "C:\Python26\lib\site-packages\matplotlib\pyplot.py", line 2046, in imshow ret = ax.imshow(X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs) File "C:\Python26\lib\site-packages\matplotlib\axes.py", line 6275, in imshow im.set_data(X) File "C:\Python26\lib\site-packages\matplotlib\image.py", line 291, in set_data self._A = pil_to_array(A) File "C:\Python26\lib\site-packages\matplotlib\image.py", line 856, in pil_to_array x = toarray(im) File "C:\Python26\lib\site-packages\matplotlib\image.py", line 831, in toarray x = np.fromstring(x_str,np.uint8) MemoryError >>> >>> >>> I already implemented some downscaling of the original images (ca. 3200 >>> x 2400 pixels), to roughly match the figures dpi-setting. But this does >>> not seem to be the only issue. The script does work for dpi of 600 or >>> 150 for 11 individual images, yielding e.g. a 23 MB file with 600 dpi >>> and 36 Megapixels. But it fails for e.g. 35 images even for 75 dpi. >>> >>> I was trying to throw away any unneccessary data using del + triggering >>> the garbage collection, but this did not help beyond a certain point. >>> Maybe somebody could tell me what kind of limitations there are using >>> imshow to plot a lot of images together, and how to improve? >>> >>> Some more info: I am using Windows. Just by judging from t
Re: [Matplotlib-users] Memory issues with imshow
On Dec 15, 2009, at 12:30 PM, Wellenreuther, Gerd wrote: > Hi Perry, > > to clarify what I am doing - maybe the error lies in here: > > * First I am building up a list of the corrected+rotated images > > * After that is done I am creating the figure > > * Then looping over every image, creating proper axes for each > individual image and finally: > >>pylab.imshow(rotated_images[i],aspect='auto') >>rotated_images[i]=[] >>gc.collect(2) > > So I am trying to immediately delete the now obsolete image-data, by > removing the reference and forcing garbage collection. No idea whether > this is the proper/best way to do it ... but at least I hope my > intention is clear :). > > Anyone an idea how to improve? if the above code is in a loop, and there is no figure clearing in the loop, then rotated_images[i] = [] gc.collect(2) will have no effect since matplotlib will still have references to the array (and generally, you never need to call gc.collect by the way). What isn't clear to me in this is how you handle the offsetting and combining of images. Normally imshow will just display one image right over the other. Can you just insert the appropriate subsampled image into one output image, and then display that. After the insertion, you can delete the input image inside the loop. Perry -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Memory issues with imshow
On Dec 15, 2009, at 1:09 PM, Gerd Wellenreuther wrote: > > > Perry Greenfield schrieb: >> if the above code is in a loop, and there is no figure clearing in >> the loop, then >> >> rotated_images[i] = [] >> gc.collect(2) >> >> will have no effect since matplotlib will still have references to >> the array (and generally, you never need to call gc.collect by the >> way). > I do / did not know whether matplotlib is actually refering to my > array (in that context dereferencing it will not free memory), or > actually copying the data at one instance (in that case it should > help). So, as you can see I am lacking the inside-knowledge of > matplotlib, and was just trying some things which were not doing any > harm (at least this is what I suppose). > To give an idea, when you ask matplotlib to render an image, it processes it (resamples, rescales, maps to colors, etc) in order to actually display it. Since it may redo all that if you resize or otherwise re-render the figure, it needs to keep a reference to the original image. Even if you delete your reference to it, it still has it, and thus it won't be deleted until the figure is cleared. So if the input to the imshow call is the full size array, you will have that around. You may want to downsample that image to lower resolution (and make sure that the downsampled version is a copy, not a view of the original array). Then you can get rid of the original image, and instead display the smaller version. Keeping that around won't impact memory. > But I would expect that figure clearing would not only free memory, > but also erase the formerly inserted images, right? *That* would > harm ;). Yes, it would erase it :-) Perry -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Memory issues with imshow
Perry Greenfield schrieb: > if the above code is in a loop, and there is no figure clearing in the > loop, then > > rotated_images[i] = [] > gc.collect(2) > > will have no effect since matplotlib will still have references to the > array (and generally, you never need to call gc.collect by the way). I do / did not know whether matplotlib is actually refering to my array (in that context dereferencing it will not free memory), or actually copying the data at one instance (in that case it should help). So, as you can see I am lacking the inside-knowledge of matplotlib, and was just trying some things which were not doing any harm (at least this is what I suppose). But I would expect that figure clearing would not only free memory, but also erase the formerly inserted images, right? *That* would harm ;). > What isn't clear to me in this is how you handle the offsetting and > combining of images. Normally imshow will just display one image right > over the other. Can you just insert the appropriate subsampled image > into one output image, and then display that. After the insertion, you > can delete the input image inside the loop. I am doing the offsetting by calculating and setting proper axes in my coordinate-system, so each imshow is done in a new, tiny axes inside the same figure, all of them without any kind of border around them. The big axes giving the box with the x- and y-ticks and the labels is create before that, and not filled directly (it is empty so to say). In this fashion I can overlay pictures precisely (or that is what I am trying). Of course, I am completely open to any suggestions on how to achieve this goal with different methods ... Cheers, Gerd -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Memory issues with imshow
Hi Perry, to clarify what I am doing - maybe the error lies in here: * First I am building up a list of the corrected+rotated images * After that is done I am creating the figure * Then looping over every image, creating proper axes for each individual image and finally: > pylab.imshow(rotated_images[i],aspect='auto') > rotated_images[i]=[] > gc.collect(2) So I am trying to immediately delete the now obsolete image-data, by removing the reference and forcing garbage collection. No idea whether this is the proper/best way to do it ... but at least I hope my intention is clear :). Anyone an idea how to improve? Cheers, Gerd Perry Greenfield wrote: > Hi Gerd, > > It still hinges on how these are stitched together. E.g. if you created > the composite by using sliced and strided arrays, and keep those strided > arrays around, then the original images are still there. But it's hard > to know what's going on without the details. It sure sounds like the > original arrays are still around somewhere. > > Thanks, Perry > > On Dec 15, 2009, at 11:54 AM, Wellenreuther, Gerd wrote: > >> Well, I am trying to create an overlay, *one* picture showing all 34 >> images. So I am only trying to create a single figure. >> >> I just attached an example so you can get an idea (it was downsampled >> for mailing, the original picture has ca. 5500 x 6500 pixels). In the >> end, I just want to save the image to the disk, so I am using 'Agg' as >> the backend - hope this also saves me some memory. >> >> And about "old" images: I am always starting a completely new >> python-process for each stitching (one at a time). >> >> Cheers, Gerd >> >> Perry Greenfield wrote: >>> Are you clearing the figure after each image display? The figure >>> retains references to the image if you don't do a clf() and thus you >>> will eventually run out of memory, even if you delete the images >>> (they don't go away while matplotlib is using them). >>> Perry >>> On Dec 15, 2009, at 10:32 AM, Wellenreuther, Gerd wrote: Dear all, I am trying to write a script to be used with our microscope, stitching images of various magnifications together to yield a big picture of a sample. The preprocessing involves operations like rotating the picture etc., and finally those pictures are being plotted using imshow. Unfortunately, I am running into memory problems, e.g.: > C:\Python26\lib\site-packages\PIL\Image.py:1264: > DeprecationWarning: integer argument expected, got float > im = self.im.stretch(size, resample) > Traceback (most recent call last): > File "F:\Procs\Find_dendrites.py", line 1093, in > > file_type="PNG",do_stitching=do_stitching,do_dendrite_finding=do_dendrite_finding,down_sizing_factor=48,dpi=600) > > > File "F:\Procs\Find_dendrites.py", line 1052, in process_images > scale,aspect_ratio,dpi,left,right,bottom,top) > File "F:\Procs\Find_dendrites.py", line 145, in stitch_images > pylab.draw() > File "C:\Python26\lib\site-packages\matplotlib\pyplot.py", line > 352, in draw > get_current_fig_manager().canvas.draw() > File > "C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py", > line 313, in draw > self.renderer = self.get_renderer() > File > "C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py", > line 324, in get_renderer > self.renderer = RendererAgg(w, h, self.figure.dpi) > File > "C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py", > line 59, in __init__ > self._renderer = _RendererAgg(int(width), int(height), dpi, > debug=False) > RuntimeError: Could not allocate memory for image or > Traceback (most recent call last): > File "F:\Procs\Find_dendrites.py", line 1093, in > > file_type="PNG",do_stitching=do_stitching,do_dendrite_finding=do_dendrite_finding,down_sizing_factor=48,dpi=75) > > > File "F:\Procs\Find_dendrites.py", line 1052, in process_images > scale,aspect_ratio,dpi,left,right,bottom,top) > File "F:\Procs\Find_dendrites.py", line 142, in stitch_images > pylab.imshow(rotated_images[i],aspect='auto') > File "C:\Python26\lib\site-packages\matplotlib\pyplot.py", line > 2046, in imshow > ret = ax.imshow(X, cmap, norm, aspect, interpolation, alpha, > vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, > resample, url, **kwargs) > File "C:\Python26\lib\site-packages\matplotlib\axes.py", line 6275, > in imshow > im.set_data(X) > File "C:\Python26\lib\site-packages\matplotlib\image.py", line 291, > in set_data > self._A = pil_to_array(A) > File "C:\Python26\lib\site-packages\matplotlib\image.py", line 856, > in pil_to_array > x = toarray(im) > File "C:\Python26\lib\site-packages\matplotlib\image.py", line 831,
[Matplotlib-users] Memory issues with imshow
Dear all, I am trying to write a script to be used with our microscope, stitching images of various magnifications together to yield a big picture of a sample. The preprocessing involves operations like rotating the picture etc., and finally those pictures are being plotted using imshow. Unfortunately, I am running into memory problems, e.g.: > C:\Python26\lib\site-packages\PIL\Image.py:1264: DeprecationWarning: integer > argument expected, got float > im = self.im.stretch(size, resample) > Traceback (most recent call last): > File "F:\Procs\Find_dendrites.py", line 1093, in > > file_type="PNG",do_stitching=do_stitching,do_dendrite_finding=do_dendrite_finding,down_sizing_factor=48,dpi=600) > File "F:\Procs\Find_dendrites.py", line 1052, in process_images > scale,aspect_ratio,dpi,left,right,bottom,top) > File "F:\Procs\Find_dendrites.py", line 145, in stitch_images > pylab.draw() > File "C:\Python26\lib\site-packages\matplotlib\pyplot.py", line 352, in draw > get_current_fig_manager().canvas.draw() > File "C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py", > line 313, in draw > self.renderer = self.get_renderer() > File "C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py", > line 324, in get_renderer > self.renderer = RendererAgg(w, h, self.figure.dpi) > File "C:\Python26\lib\site-packages\matplotlib\backends\backend_agg.py", > line 59, in __init__ > self._renderer = _RendererAgg(int(width), int(height), dpi, debug=False) > RuntimeError: Could not allocate memory for image or > Traceback (most recent call last): > File "F:\Procs\Find_dendrites.py", line 1093, in > > file_type="PNG",do_stitching=do_stitching,do_dendrite_finding=do_dendrite_finding,down_sizing_factor=48,dpi=75) > File "F:\Procs\Find_dendrites.py", line 1052, in process_images > scale,aspect_ratio,dpi,left,right,bottom,top) > File "F:\Procs\Find_dendrites.py", line 142, in stitch_images > pylab.imshow(rotated_images[i],aspect='auto') > File "C:\Python26\lib\site-packages\matplotlib\pyplot.py", line 2046, in > imshow > ret = ax.imshow(X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, > origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs) > File "C:\Python26\lib\site-packages\matplotlib\axes.py", line 6275, in > imshow > im.set_data(X) > File "C:\Python26\lib\site-packages\matplotlib\image.py", line 291, in > set_data > self._A = pil_to_array(A) > File "C:\Python26\lib\site-packages\matplotlib\image.py", line 856, in > pil_to_array > x = toarray(im) > File "C:\Python26\lib\site-packages\matplotlib\image.py", line 831, in > toarray > x = np.fromstring(x_str,np.uint8) > MemoryError I already implemented some downscaling of the original images (ca. 3200 x 2400 pixels), to roughly match the figures dpi-setting. But this does not seem to be the only issue. The script does work for dpi of 600 or 150 for 11 individual images, yielding e.g. a 23 MB file with 600 dpi and 36 Megapixels. But it fails for e.g. 35 images even for 75 dpi. I was trying to throw away any unneccessary data using del + triggering the garbage collection, but this did not help beyond a certain point. Maybe somebody could tell me what kind of limitations there are using imshow to plot a lot of images together, and how to improve? Some more info: I am using Windows. Just by judging from the task-manager, the preprocessing is not the problem. But *plotting* the images using imshow seems to cause an increase of memory consumption of the task of 32-33 MB *each* time. Somewhere around a total of 1.3 - 1.5 Gigs the process dies ... Thanks in advance, Gerd -- Dr. Gerd Wellenreuther beamline scientist P06 "Hard X-Ray Micro/Nano-Probe" Petra III project HASYLAB at DESY Notkestr. 85 22603 Hamburg Tel.: + 49 40 8998 5701 -- Return on Information: Google Enterprise Search pays you back Get the facts. http://p.sf.net/sfu/google-dev2dev ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
