On 2/4/2011 3:29 PM, Eric Firing wrote:
On 02/04/2011 12:33 PM, Christoph Gohlke wrote:
On 2/4/2011 2:14 PM, Eric Firing wrote:
On 02/04/2011 11:33 AM, Eric Firing wrote:
On 02/04/2011 10:28 AM, Christoph Gohlke wrote:
On 2/4/2011 11:54 AM, Eric Firing wrote:
On 02/03/2011 05:35 PM, Christoph Gohlke wrote:
On 2/3/2011 6:50 PM, Eric Firing wrote:
On 02/03/2011 03:04 PM, Benjamin Root wrote:
Also, not to sound too annoying, but has anyone considered the idea of
using compressed arrays for holding those rgba values?
I don't see how that really helps; as far as I know, a full rgba array
has to be passed into agg. What *does* help is using uint8 from start
to finish. It might also be possible to use some smart downsampling
before generating the rgba array, but the uint8 route seems to me the
first thing to attack.
Eric
Ben Root
Please review the attached patch. It avoids generating and storing
float64 rgba arrays and uses uint8 rgba instead. That's a huge memory
saving and also faster. I can't see any side effects as
_image.fromarray() converts the float64 input to uint8 anyway.
Christoph,
Thank you! I haven't found anything wrong with that delightfully simple
patch, so I have committed it to the trunk. Back in 2007 I added the
ability of colormapping to generate uint8 directly, precisely to enable
this sort of optimization. Why it was not already being used in imshow,
I don't know--maybe I was going to do it, got sidetracked, and never
finished.
I suspect it won't be as simple as for the plain image, but there may be
opportunities for optimizing with uint8 in other image-like operations.
So far other attempts to optimize memory usage were thwarted by
matplotlib's internal use of masked arrays. As mentioned before, users
can provide their own normalized rgba arrays to avoid all this processing.
Did you see other potential low-hanging fruit that might be harvested
with some changes to the code associated with masked arrays?
Eric
The norm function currently converts the data to double precision
floating point and also creates temporary arrays that can be avoided.
For float32 and low precision integer images this seems overkill and one
could use float32. It might be possible to replace the norm function
with numpy.digitize if that works with masked arrays. Last, the
_image.frombyte function does a copy of 'strided arrays' (only relevant
when zooming/panning large images). I try to provide a patch for each.
masked arrays can be filled to create an ndarray before passing to
digitize; whether that will be faster, remains to be seen. I've never
used digitize.
I didn't say that ("can be filled...") right. I think one would need to
use the mask to put in the i_bad index where appropriate. np.ma does
not have a digitize function. I suspect it won't help much if at all in
Normalize, but it would be a natural for use in BoundaryNorm.
It looks easy to allow Normalize.__call__ to use float32 if that is what
it receives.
I don't see any unnecessary temporary array creation apart from the
conversion to float64, except for the generation of a masked array
regardless of input. I don't think this costs much; if it gets an
ndarray it does not copy it, and it does not generate a full mask array.
Still, the function probably could be sped up a bit by handling
masking more explicitly instead of letting ma do the work.
In class Normalize:
result = 0.0 * val
and
result = (val-vmin) / (vmax-vmin)
Eric
Regarding frombyte, I suspect you can't avoid the copy; the data
structure being passed to agg is just a string of bytes, as far as I can
see, so everything is based on having a simple contiguous array.
The PyArray_ContiguousFromObject call will return a copy if the input
array is not already contiguous.
Exactly. I thought you were suggesting that this was not needed, but
maybe I misunderstood.
Eric
In fact I am suggesting that this is not needed. The copy-to-agg routine
could be made 'stride aware' and use PyArray_FromObject. Not a very low
hanging fruit but it seems fromarray() does this already. I'm not sure
it's worth.
How about these changes to color.py (attached). This avoids copies, uses
in-place operations, and calculates single precision when normalizing
small integer and float32 arrays. Similar could be done for LogNorm. Do
masked arrays support in-place operations?
Christoph
Index: colors.py
===================================================================
--- colors.py (revision 8944)
+++ colors.py (working copy)
@@ -514,15 +514,17 @@
xma = ma.array(X, copy=True)
xa = xma.filled(0)
mask_bad = ma.getmask(xma)
+ del xma
if xa.dtype.char in np.typecodes['Float']:
np.putmask(xa, xa==1.0, 0.9999999) #Treat 1.0 as slightly less
than 1.
# The following clip is fast, and prevents possible
# conversion of large positive values to negative integers.
+ xa *= self.N
if NP_CLIP_OUT:
- np.clip(xa * self.N, -1, self.N, out=xa)
+ np.clip(xa, -1, self.N, out=xa)
else:
- xa = np.clip(xa * self.N, -1, self.N)
+ xa = np.clip(xa, -1, self.N)
# ensure that all 'under' values will still have negative
# value after casting to int
@@ -803,25 +805,33 @@
if cbook.iterable(value):
vtype = 'array'
- val = ma.asarray(value).astype(np.float)
+ result = ma.asarray(value)
+ if result.dtype.kind == 'f':
+ if isinstance(value, np.ndarray):
+ result = result.copy()
+ elif result.dtype.itemsize > 2:
+ result = result.astype(np.float)
+ else:
+ result = result.astype(np.float32)
else:
vtype = 'scalar'
- val = ma.array([value]).astype(np.float)
+ result = ma.array([value]).astype(np.float)
- self.autoscale_None(val)
+ self.autoscale_None(result)
vmin, vmax = self.vmin, self.vmax
if vmin > vmax:
raise ValueError("minvalue must be less than or equal to maxvalue")
- elif vmin==vmax:
- result = 0.0 * val
+ elif vmin == vmax:
+ result[:] = 0.0
else:
vmin = float(vmin)
vmax = float(vmax)
if clip:
- mask = ma.getmask(val)
- val = ma.array(np.clip(val.filled(vmax), vmin, vmax),
- mask=mask)
- result = (val-vmin) / (vmax-vmin)
+ mask = ma.getmask(result)
+ result = ma.array(np.clip(result.filled(vmax), vmin, vmax),
+ mask=mask)
+ result -= vmin
+ result /= vmax - vmin
if vtype == 'scalar':
result = result[0]
return result
------------------------------------------------------------------------------
The modern datacenter depends on network connectivity to access resources
and provide services. The best practices for maximizing a physical server's
connectivity to a physical network are well understood - see how these
rules translate into the virtual world?
http://p.sf.net/sfu/oracle-sfdevnlfb
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users
