2011/11/8 Thibault North <tno...@fedoraproject.org>:
> Hello,
>
> I just discovered PyTables and am trying to find a good table structure to
> describe my data.
>
> A table is defined as:
> mytable = h5file.createTable(group, 'outputs', Foobar, "Foobar description")
>
> with a class Foobar:
>
> class Foobar(tables.IsDescription):
> iteration = tables.Int32Col()
> output = tables.Float64Col(shape=(1, N))
> steps = tables.Float64Col(shape=(K, N))
>
> Now I can easily use mytable.row['iteration'] or mytable.row['output'] and
> feed them with some data.
> Unfortunately, regarding mytable.row['steps'], it is very likely that the
> array of size (K,N) won't fit in memory. Is it possible to fill it by
> iterating on K ?
> Something like mytable.row['steps'][j] = foo with 0 < j < K and foo of size
> (1,N) doesn't work.
>
> I was thinking of using a CArray instead, but AFAIK it is not possible to
> include it directly into Foobar.
For things like that, it is always recommended to get your big
matrices out of Table objects. In your case, you may want to put
`steps` (and probably `output` too) in external EArray objects. For
example, if your table is going to be L rows large, then you can keep
`steps` on a EArray with shape (L, K, N). With this, accessing your
EArray data in typical table loops is easy:
for row in mytable:
niter = row['iteration']
myearray[row.nrow]
# operate with this
If (K,N) arrays are too long, EArray objects lets you fill (and
retrieve) parts of it easily via indexing. For example:
myearray[my_l_idx, my_k_idx]
retrieves the row `my_k_idx` from `my_l_idx` element. Similarly, you
can update parts too:
myearray[my_l_idx, my_k_idx] = np.arange(N, dtype=np.double)
Alternatively, you can also use a CArray here, with the difference
that, with this, you should declare the final length of the container
(while this is not necessary with EArray). Read the documentation
about EArray/CArray more carefully to explore their rich set of
features.
Hope this helps,
--
Francesc Alted
------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
