I am trying to speed up some analysis routines on arrays that are
approximately 16 x 200,000,000 elements (stored in HDF5 arrays that
were originally created with PyTables). I was looking into whether I
could speed up the analysis using tricks such as memmap and numexpr;
however, since I need to perform row-wise operations (e.g. computing
the dot product with a 16x16 array followed by a scipy.signal.filter
operation) which requires indexing, I do not believe I can use
numexpr. This leaves "memmaping", but I understand that PyTables
offers something similar. I found a very old discussion on this
mailing list
(http://www.mail-archive.com/pytables-users@lists.sourceforge.net/msg01295.html),
but the link Francesc provided to the slides from Euro Scipy
describing how to use a blocking technique with PyTables no longer
works. Does anyone have access to the original slides?
I'm assuming that the blocking technique is as simple as determining a
chunk size to operate on and then looping through the PyTables Array,
loading the chunk into memory, running np.dot and scipy.signal.filter
and saving the result to a new PyTables array, but I was curious to
see if the slides point out any subtleties of this approach that I
should be aware of.
If I understand correctly, the blocking approach is as simple as the following:
# note that diff is a 16x16 array
source = f_in.root.data
dest = f_in.createCArray('/', 'result', atom=tables.Float32Atom(),
size=source.size)
temp = np.empty((16, chunksize))
for chunk in range(n_chunks):
block = source[:, chunk*chunksize:chunk*chunksize+chunksize]
result = np.dot(diff, block, out=temp)
result = scipy.signal.filtfilt(b, a, result)
dest[:, chunk*chunksize:chunk*chunksize+chunksize] = result
Thanks!
Brad
------------------------------------------------------------------------------
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
