Thanks for all the tips on this. It is finally clear what is going on. I
think they fact that you can specify a shape for table columns leads
beginners down a the road I'm on. When I chose to use pytables after reading
some of the documentation and playing with the code, it never occurred to me
that I'd have this issue.
Turns out that I'm only storing arrays of shape (1,) in the columns, so it
is very easy to pad and un-pad them and this doesn't really add much size to
my database. I'll stick with that for now, but in version two of my
software, I'll certainly look into the ability to add a VLArrayCol. It would
be a nice feature.
Jason
On Wed, Jun 22, 2011 at 6:10 PM, Anthony Scopatz <scop...@gmail.com> wrote:
> Hi Jason,
>
> Upon further inspection, I think that this comes down to a spelling issue.
>
>
> Really, what you want is a VL structure inside of a table. Tables are
> themselves
> VL structures and have been special cased to nest themselves. This could
> be
> changed to allow more general VL structures to be nested, but that would
> require
> a good amount of effort (that I sadly don't have time for now).
>
> But really the only thing that you are doing when you nest a table within a
> table
> rather than nesting a vlarray in a table is that you are providing *extra*
> information.
> This isn't really a bad thing, and gets to what I mean by spelling.
>
> Nesting a table in a table you have,
>
> desc = {'A': Float64Col, 'B': {'C': Int32Col}}
>
>
> Nesting a vlarry inside of a column would presumably be
>
> desc = {'A': Float64Col, 'B': VLArrayCol(atom=Int32Col)}
>
>
> Unfortunately, the VLArrayCol object doesn't exist. The reason it doesn't
> I suspect is because of how you would spell the path to the nested data.
> In nested tables you use the column names:
>
> t['B/C']
>
> In nested vlarrays, you'd have to use the index, which is weird.
>
> t['B/0'], t['B/1'], t['B/2'], ...
>
> Nested tables are everything you want, plus a tiny bit more in terms of
> having to
> name the dimension you want to be variable.
>
> It should be noted that you end up with the same weird spelling issues in
> numpy
> structured arrays, which similarly disallow such funny business.
>
> Of course, as Dav mentions, there are other options. Store your VLArrays
> outside of
> the table, then link to them in the table either through path or hard
> links, etc.
>
> At the end of the day I would still suggest the table in a table idea. It
> isn't that much
> extra work, and on the HDF5 level it is exactly what you want.
>
> Be Well
> Anthony
>
> On Wed, Jun 22, 2011 at 6:26 PM, Jason Moore <moorepa...@gmail.com> wrote:
>
>> Anthony,
>>
>> I read of the documentation and tried creating a table with a VLArray
>> nested inside, but can't seem to figure out how to nest anything inside a
>> table other than another table description.
>>
>> I found this question that is trying to do exactly what I want to do:
>>
>>
>> http://stackoverflow.com/questions/5366099/in-pytables-how-to-create-nested-array-of-variable-length
>>
>> It is well written but has no answers.
>>
>> Any specific ideas on how to make this example work?
>>
>> Jason
>>
>>
>> On Wed, Jun 22, 2011 at 2:57 PM, Anthony Scopatz <scop...@gmail.com>wrote:
>>
>>> Also note that whenever you are dealing with variable length stuctures,
>>> chuncksize is very important.
>>>
>>>
>>> On Wed, Jun 22, 2011 at 4:55 PM, Anthony Scopatz <scop...@gmail.com>wrote:
>>>
>>>> Hi Jason,
>>>>
>>>> You can try nesting variable length arrays inside of your table (
>>>> http://www.pytables.org/docs/manual/ch04.html#VLArrayClassDescr). You
>>>> might need to refer to nested tables to get an idea of how to do this (
>>>> http://www.pytables.org/docs/manual/ch03.html#id332794). On the HDF5
>>>> level, tables are just a convenient wrapper around variable length arrays
>>>> of
>>>> structs. Since you can nest tables, you should be able to nest VLArrays
>>>> inside of tables.
>>>>
>>>> Let us know if you run into specific problems.
>>>>
>>>> Be Well
>>>> Anthony
>>>>
>>>> On Wed, Jun 22, 2011 at 4:46 PM, Jason Moore <moorepa...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I've been attempting to use pyTables to store data from a
>>>>> data acquisition system that I'm using for my experimental work. I
>>>>> currently
>>>>> make use of several pytables tables to store the data and really like the
>>>>> querying ability that the table object provides. But I have an issue with
>>>>> storing arrays in a table column. For each experiment that I do, I record
>>>>> data for a different amount of time and potentially a different sampling
>>>>> rate. I'd like the rows of the table to correspond with each experiment
>>>>> and
>>>>> the columns with the various data signals that I collect. But it seems
>>>>> that
>>>>> a pytables table object requires every array in a column to be of the same
>>>>> shape. Right now I choose an arbitrarily large shape in my table
>>>>> definition
>>>>> and I'm awkwardly padding the data signals before I store them in the
>>>>> table.
>>>>> Are there other ways to do this?
>>>>>
>>>>> Jason
>>>>>
>>>>> --
>>>>> http://mae.ucdavis.edu/~biosport/jkm/
>>>>> Sports Biomechanics Lab <http://biosport.ucdavis.edu>, UC Davis
>>>>> Davis Bike Collective <http://www.davisbikecollective.org> Minister,
>>>>> Davis, CA
>>>>> BikeDavis.info
>>>>> Office: +01 530-752-2163
>>>>> Lab: +01 530-752-2235
>>>>> Home: +01 530-753-0794
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Simplify data backup and recovery for your virtual environment with
>>>>> vRanger.
>>>>> Installation's a snap, and flexible recovery options mean your data is
>>>>> safe,
>>>>> secure and there when you need it. Data protection magic?
>>>>> Nope - It's vRanger. Get your free trial download today.
>>>>> http://p.sf.net/sfu/quest-sfdev2dev
>>>>> _______________________________________________
>>>>> Pytables-users mailing list
>>>>> Pytables-users@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>>
>>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Simplify data backup and recovery for your virtual environment with
>>> vRanger.
>>> Installation's a snap, and flexible recovery options mean your data is
>>> safe,
>>> secure and there when you need it. Data protection magic?
>>> Nope - It's vRanger. Get your free trial download today.
>>> http://p.sf.net/sfu/quest-sfdev2dev
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pytables-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>>
>>
>>
>> --
>> http://mae.ucdavis.edu/~biosport/jkm/
>> Sports Biomechanics Lab <http://biosport.ucdavis.edu>, UC Davis
>> Davis Bike Collective <http://www.davisbikecollective.org> Minister,
>> Davis, CA
>> BikeDavis.info
>> Office: +01 530-752-2163
>> Lab: +01 530-752-2235
>> Home: +01 530-753-0794
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Simplify data backup and recovery for your virtual environment with
>> vRanger.
>> Installation's a snap, and flexible recovery options mean your data is
>> safe,
>> secure and there when you need it. Data protection magic?
>> Nope - It's vRanger. Get your free trial download today.
>> http://p.sf.net/sfu/quest-sfdev2dev
>> _______________________________________________
>> Pytables-users mailing list
>> Pytables-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Simplify data backup and recovery for your virtual environment with
> vRanger.
> Installation's a snap, and flexible recovery options mean your data is
> safe,
> secure and there when you need it. Data protection magic?
> Nope - It's vRanger. Get your free trial download today.
> http://p.sf.net/sfu/quest-sfdev2dev
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
--
http://mae.ucdavis.edu/~biosport/jkm/
Sports Biomechanics Lab <http://biosport.ucdavis.edu>, UC Davis
Davis Bike Collective <http://www.davisbikecollective.org> Minister, Davis,
CA
BikeDavis.info
Office: +01 530-752-2163
Lab: +01 530-752-2235
Home: +01 530-753-0794
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
