A Friday 12 June 2009 11:26:40 Jon Olav Vik escrigué: > Francesc Alted <faltet <at> pytables.org> writes: > > > Question 2: Further on I only need to work with the last 20% or so of > > > each row. Is there an efficient way to slice from a row without having > > > to load it all from disk? > > > > > > for i in range(len(y)): > > > yj = y[i][-2000:] # not having to read y[i][:6500] > > > ... > > > > I'm afraid you can't. The thing is that the VL types cannot be divided > > and the entire data element must be transferred. See: > > > > http://www.hdfgroup.org/HDF5/doc/UG/11_Datatypes.html > > > > section 4.3.2.3 for more info on this. > > Ouch. This means I'll have to store my own "abstracts" of the data, and in > many cases it will be faster to re-compute the details I need than store > them all in HDF5 files. I already currently have something like this: a > table with a field for the VLArray row index. I'll need to somehow expand > it with more fields, but I vaguely recall that it's not possible to alter > table descriptions (add/drop fields). I guess a new table aligned with the > old one is the easiest way out, or a manual loop: > > # get description of old_table > # add new fields > # create new_table > from itertools import izip > for oldrow, newrow in izip(old_table, new_table): > for field in old_fields: > newrow[field] = oldrow[field]
Yes. Something like this should work (and quite efficiently). > I'm a little surprised that the design of HDF5 does not permit striding and > slicing of VLArray rows; I thought a VLArray mostly behaved like any other > array. Well, this is expected, because variable length datasets are made of variable length *data types*, and, contrarily to datasets, HDF5 does not provide accessors to pieces of data types. That could change in the future, I suppose. Regards, -- Francesc Alted ------------------------------------------------------------------------------ Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
