On 02/04/2011 10:28 AM, Christoph Gohlke wrote: > > > On 2/4/2011 11:54 AM, Eric Firing wrote: >> On 02/03/2011 05:35 PM, Christoph Gohlke wrote: >>> >>> >>> On 2/3/2011 6:50 PM, Eric Firing wrote: >>>> On 02/03/2011 03:04 PM, Benjamin Root wrote: >>>> >>>>> Also, not to sound too annoying, but has anyone considered the idea of >>>>> using compressed arrays for holding those rgba values? >>>> >>>> I don't see how that really helps; as far as I know, a full rgba array >>>> has to be passed into agg. What *does* help is using uint8 from start >>>> to finish. It might also be possible to use some smart downsampling >>>> before generating the rgba array, but the uint8 route seems to me the >>>> first thing to attack. >>>> >>>> Eric >>>> >>>>> >>>>> Ben Root >>>> >>> >>> Please review the attached patch. It avoids generating and storing >>> float64 rgba arrays and uses uint8 rgba instead. That's a huge memory >>> saving and also faster. I can't see any side effects as >>> _image.fromarray() converts the float64 input to uint8 anyway. >> >> Christoph, >> >> Thank you! I haven't found anything wrong with that delightfully simple >> patch, so I have committed it to the trunk. Back in 2007 I added the >> ability of colormapping to generate uint8 directly, precisely to enable >> this sort of optimization. Why it was not already being used in imshow, >> I don't know--maybe I was going to do it, got sidetracked, and never >> finished. >> >> I suspect it won't be as simple as for the plain image, but there may be >> opportunities for optimizing with uint8 in other image-like operations. >> >>> >>> So far other attempts to optimize memory usage were thwarted by >>> matplotlib's internal use of masked arrays. As mentioned before, users >>> can provide their own normalized rgba arrays to avoid all this processing. >>> >> >> Did you see other potential low-hanging fruit that might be harvested >> with some changes to the code associated with masked arrays? >> >> Eric >> > > The norm function currently converts the data to double precision > floating point and also creates temporary arrays that can be avoided. > For float32 and low precision integer images this seems overkill and one > could use float32. It might be possible to replace the norm function > with numpy.digitize if that works with masked arrays. Last, the > _image.frombyte function does a copy of 'strided arrays' (only relevant > when zooming/panning large images). I try to provide a patch for each.
masked arrays can be filled to create an ndarray before passing to digitize; whether that will be faster, remains to be seen. I've never used digitize. Regarding frombyte, I suspect you can't avoid the copy; the data structure being passed to agg is just a string of bytes, as far as I can see, so everything is based on having a simple contiguous array. Eric > > Christoph > > ------------------------------------------------------------------------------ > The modern datacenter depends on network connectivity to access resources > and provide services. The best practices for maximizing a physical server's > connectivity to a physical network are well understood - see how these > rules translate into the virtual world? > http://p.sf.net/sfu/oracle-sfdevnlfb > _______________________________________________ > Matplotlib-users mailing list > Matplotlib-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/matplotlib-users ------------------------------------------------------------------------------ The modern datacenter depends on network connectivity to access resources and provide services. The best practices for maximizing a physical server's connectivity to a physical network are well understood - see how these rules translate into the virtual world? http://p.sf.net/sfu/oracle-sfdevnlfb _______________________________________________ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
