Hi Jason,
Glad we could help, and that your problem seemed to get worked out.
I agree that VLArrayCol would be nice, even though I still think that nested
tables handle this (albeit with a little more syntax). Feel free to open an
issue on github about this, including some of the comments and references
here.
Be Well
Anthony
On Thu, Jun 30, 2011 at 11:08 AM, Jason Moore moorepa...@gmail.com wrote:
Thanks for all the tips on this. It is finally clear what is going on. I
think they fact that you can specify a shape for table columns leads
beginners down a the road I'm on. When I chose to use pytables after reading
some of the documentation and playing with the code, it never occurred to me
that I'd have this issue.
Turns out that I'm only storing arrays of shape (1,) in the columns, so it
is very easy to pad and un-pad them and this doesn't really add much size to
my database. I'll stick with that for now, but in version two of my
software, I'll certainly look into the ability to add a VLArrayCol. It would
be a nice feature.
Jason
On Wed, Jun 22, 2011 at 6:10 PM, Anthony Scopatz scop...@gmail.comwrote:
Hi Jason,
Upon further inspection, I think that this comes down to a spelling issue.
Really, what you want is a VL structure inside of a table. Tables are
themselves
VL structures and have been special cased to nest themselves. This could
be
changed to allow more general VL structures to be nested, but that would
require
a good amount of effort (that I sadly don't have time for now).
But really the only thing that you are doing when you nest a table within
a table
rather than nesting a vlarray in a table is that you are providing *extra*
information.
This isn't really a bad thing, and gets to what I mean by spelling.
Nesting a table in a table you have,
desc = {'A': Float64Col, 'B': {'C': Int32Col}}
Nesting a vlarry inside of a column would presumably be
desc = {'A': Float64Col, 'B': VLArrayCol(atom=Int32Col)}
Unfortunately, the VLArrayCol object doesn't exist. The reason it
doesn't
I suspect is because of how you would spell the path to the nested data.
In nested tables you use the column names:
t['B/C']
In nested vlarrays, you'd have to use the index, which is weird.
t['B/0'], t['B/1'], t['B/2'], ...
Nested tables are everything you want, plus a tiny bit more in terms of
having to
name the dimension you want to be variable.
It should be noted that you end up with the same weird spelling issues in
numpy
structured arrays, which similarly disallow such funny business.
Of course, as Dav mentions, there are other options. Store your VLArrays
outside of
the table, then link to them in the table either through path or hard
links, etc.
At the end of the day I would still suggest the table in a table idea. It
isn't that much
extra work, and on the HDF5 level it is exactly what you want.
Be Well
Anthony
On Wed, Jun 22, 2011 at 6:26 PM, Jason Moore moorepa...@gmail.comwrote:
Anthony,
I read of the documentation and tried creating a table with a VLArray
nested inside, but can't seem to figure out how to nest anything inside a
table other than another table description.
I found this question that is trying to do exactly what I want to do:
http://stackoverflow.com/questions/5366099/in-pytables-how-to-create-nested-array-of-variable-length
It is well written but has no answers.
Any specific ideas on how to make this example work?
Jason
On Wed, Jun 22, 2011 at 2:57 PM, Anthony Scopatz scop...@gmail.comwrote:
Also note that whenever you are dealing with variable length stuctures,
chuncksize is very important.
On Wed, Jun 22, 2011 at 4:55 PM, Anthony Scopatz scop...@gmail.comwrote:
Hi Jason,
You can try nesting variable length arrays inside of your table (
http://www.pytables.org/docs/manual/ch04.html#VLArrayClassDescr). You
might need to refer to nested tables to get an idea of how to do this (
http://www.pytables.org/docs/manual/ch03.html#id332794). On the HDF5
level, tables are just a convenient wrapper around variable length arrays
of
structs. Since you can nest tables, you should be able to nest VLArrays
inside of tables.
Let us know if you run into specific problems.
Be Well
Anthony
On Wed, Jun 22, 2011 at 4:46 PM, Jason Moore moorepa...@gmail.comwrote:
Hi,
I've been attempting to use pyTables to store data from a
data acquisition system that I'm using for my experimental work. I
currently
make use of several pytables tables to store the data and really like the
querying ability that the table object provides. But I have an issue with
storing arrays in a table column. For each experiment that I do, I record
data for a different amount of time and potentially a different sampling
rate. I'd like the rows of the table to correspond with each experiment
and
the columns with the various data signals that I collect. But it seems
that
a pytables table object requires every
