On November 9, 2011 02:04:45 AM Francesc Alted wrote:
> 2011/11/8 Thibault North <tno...@fedoraproject.org>:
>> [...]
>
> For things like that, it is always recommended to get your big
> matrices out of Table objects. In your case, you may want to put
> `steps` (and probably `output` too) in external EArray objects. For
> example, if your table is going to be L rows large, then you can keep
> `steps` on a EArray with shape (L, K, N). With this, accessing your
> EArray data in typical table loops is easy:
>
> for row in mytable:
> niter = row['iteration']
> myearray[row.nrow]
> # operate with this
>
> If (K,N) arrays are too long, EArray objects lets you fill (and
> retrieve) parts of it easily via indexing. For example:
>
> myearray[my_l_idx, my_k_idx]
>
> retrieves the row `my_k_idx` from `my_l_idx` element. Similarly, you
> can update parts too:
>
> myearray[my_l_idx, my_k_idx] = np.arange(N, dtype=np.double)
Thanks for your detailled reply.
I am now using a EArray, as you suggested, with:
myearray = h5file.createEArray(h5file.root, 'steps', atom
, (L, K, 0), "desc")
(Number of iteration N is unknown but usually small, L and K are known)
If I understand well, I need to append an initially empty (L,K) matrix for
each iteration:
myearray.append(np.zeros((L,K))).reshape(L, K, 1))
and then fill it by iterating on the K dimension:
for line in mymatrix:
myearray[line, :, iteration] = mydatacol
This actually works, but:
- the append() requires to load an array of size(L,K) in memory
- I get a warning (with L=2e4+1 and K=1e3)
/usr/lib64/python2.7/site-packages/tables/leaf.py:416: PerformanceWarning: The
Leaf ``/steps`` is exceeding the maximum recommended rowsize (104857600
bytes);
be ready to see PyTables asking for *lots* of memory and possibly slow
I/O. You may want to reduce the rowsize by trimming the value of
dimensions that are orthogonal (and preferably close) to the *main*
dimension of this leave. Alternatively, in case you have specified a
very small/large chunksize, you may want to increase/decrease it.
PerformanceWarning)
Is that just a matter of choosing the good position for the extendable
dimension?
I quickly checked the docs, but couldn't find a way to "fill [...] parts of it
[the array] easily via indexing". Append() seems to be required before...
Thanks again,
Thibault
>
> Alternatively, you can also use a CArray here, with the difference
> that, with this, you should declare the final length of the container
> (while this is not necessary with EArray). Read the documentation
> about EArray/CArray more carefully to explore their rich set of
> features.
>
> Hope this helps,
------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
