[Pytables-users] compare 2 tables from 2 different h5 files?
Are queries limited to 1 table only? Is it possible to compare rows of 2 tables in same database? Is it possible to compare rows of 2 tables located in different database/files? Regards, -- Emmanuel -- Doing More with Less: The Next Generation Virtual Desktop What are the key obstacles that have prevented many mid-market businesses from deploying virtual desktops? How do next-generation virtual desktops provide companies an easier-to-deploy, easier-to-manage and more affordable virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] compare 2 tables from 2 different h5 files?
I use a control system (50K+ parameters). i would like to take a snapshot/backup of a running configuration and save all the 50K+ parameters in one file. A feature I need is to compare snapshot with each others. For example: how does the current configuration compare to the snapshot taken 3 weeks ago. 1/ list of parameters to save are in a table of a h5 (50K+ parameters) 2/ take a snapshot (for each row extend with value of parameter) 3/ compare new snapshot to old snapshot where parameter names are the same ( with a tolerance on the parameter let's say of 1% up to 3% function of parameter) Also is there a way to easily edit h5 files (not sure vitables can do that based on its old documentation) and add a parameter in a table for future backup. I am interested in h5 because my parameters can be classified in /machine_id/systems/subsystems all of which could be groups. would be nice for browsing and selective restore. Any info on vitables and editable entries/row in tables? At this point, I am not yet convinced I should use h5/pytables yet... just exploring On Thu, Sep 15, 2011 at 11:58 AM, Anthony Scopatz ascop...@enthought.com wrote: Hello Emmanuel, Could you please be more specific? You can always take data from where ever your want (including independent tables) and run the comparison in Python (or NumPy). What are you trying to accomplish? Be Well Anthony On Thu, Sep 15, 2011 at 1:26 PM, Emmanuel Mayssat emays...@gmail.com wrote: Are queries limited to 1 table only? Is it possible to compare rows of 2 tables in same database? Is it possible to compare rows of 2 tables located in different database/files? Regards, -- Emmanuel -- Doing More with Less: The Next Generation Virtual Desktop What are the key obstacles that have prevented many mid-market businesses from deploying virtual desktops? How do next-generation virtual desktops provide companies an easier-to-deploy, easier-to-manage and more affordable virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Doing More with Less: The Next Generation Virtual Desktop What are the key obstacles that have prevented many mid-market businesses from deploying virtual desktops? How do next-generation virtual desktops provide companies an easier-to-deploy, easier-to-manage and more affordable virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Doing More with Less: The Next Generation Virtual Desktop What are the key obstacles that have prevented many mid-market businesses from deploying virtual desktops? How do next-generation virtual desktops provide companies an easier-to-deploy, easier-to-manage and more affordable virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] compare 2 tables from 2 different h5 files?
On Thu, Sep 15, 2011 at 5:16 PM, Emmanuel Mayssat emays...@gmail.comwrote: I use a control system (50K+ parameters). i would like to take a snapshot/backup of a running configuration and save all the 50K+ parameters in one file. A feature I need is to compare snapshot with each others. For example: how does the current configuration compare to the snapshot taken 3 weeks ago. 1/ list of parameters to save are in a table of a h5 (50K+ parameters) 2/ take a snapshot (for each row extend with value of parameter) 3/ compare new snapshot to old snapshot where parameter names are the same ( with a tolerance on the parameter let's say of 1% up to 3% function of parameter) I guess I am unclear on how you are storing these parameters. They can't be columns in a table due to a hard HDF5 limitation. Thus the parameters are either row entries in a table or they are stored as an array. How you are storing them is going to determine how you might what to perform the comparison. Also, any comparison where you want to build in a tolerance is going to have to (at some point) hit Python. That is to say, you'll have to read into memory the parts of the file(s) that you want to do the comparison on (also true no matter what interface you use). If you could describe your actual storage layout, we could help you more. Also is there a way to easily edit h5 files (not sure vitables can do that based on its old documentation) and add a parameter in a table for future backup. Currently, sadly no. ViTables does not let you edit. I have toyed with the idea of writing a Traits-based application to do just that. I have some parts of it written, if you or anyone else is interested It actually wouldn't be *that* much work. I don't know of an HDF5 editor, only viewers. I am interested in h5 because my parameters can be classified in /machine_id/systems/subsystems all of which could be groups. would be nice for browsing and selective restore. Groups are a common feature to all HDF5 libraries (indeed they are the 'H')! Be Well Anthony Any info on vitables and editable entries/row in tables? At this point, I am not yet convinced I should use h5/pytables yet... just exploring On Thu, Sep 15, 2011 at 11:58 AM, Anthony Scopatz ascop...@enthought.com wrote: Hello Emmanuel, Could you please be more specific? You can always take data from where ever your want (including independent tables) and run the comparison in Python (or NumPy). What are you trying to accomplish? Be Well Anthony On Thu, Sep 15, 2011 at 1:26 PM, Emmanuel Mayssat emays...@gmail.com wrote: Are queries limited to 1 table only? Is it possible to compare rows of 2 tables in same database? Is it possible to compare rows of 2 tables located in different database/files? Regards, -- Emmanuel -- Doing More with Less: The Next Generation Virtual Desktop What are the key obstacles that have prevented many mid-market businesses from deploying virtual desktops? How do next-generation virtual desktops provide companies an easier-to-deploy, easier-to-manage and more affordable virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Doing More with Less: The Next Generation Virtual Desktop What are the key obstacles that have prevented many mid-market businesses from deploying virtual desktops? How do next-generation virtual desktops provide companies an easier-to-deploy, easier-to-manage and more affordable virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Doing More with Less: The Next Generation Virtual Desktop What are the key obstacles that have prevented many mid-market businesses from deploying virtual desktops? How do next-generation virtual desktops provide companies an easier-to-deploy, easier-to-manage and more affordable virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- BlackBerryreg; DevCon Americas, Oct. 18-20, San Francisco, CA http://p.sf.net/sfu/rim-devcon-copy2___ Pytables-users mailing
Re: [Pytables-users] compare 2 tables from 2 different h5 files?
I guess I am unclear on how you are storing these parameters. They can't be columns in a table due to a hard HDF5 limitation. Thus the parameters are either row entries in a table or they are stored as an array. How you are storing them is going to determine how you might what to perform the comparison. 1/ I need a hierarchical model cls1/injector/magnets/correctors cls1/injector/magnets/quads cls1/injecotr/magnets/solenoids In my table, 1 row is parameter name, value, other parameter meta data 2/ Assuming the hierarchical layout should not interfere with the comparison (the 2 snapshots to be compared can have different 'hierarchical model' due to evolution/ future development / future optimization, but should still be comparable) This forces me to have * 1 table at the root with all backed up parameters with their respective values = the backup table * 1 static table with parameters to be backed up + metadata on parameter (i.e subsystems it belongs to, etc) * 1 hierarchy model to show how subsystems are related to systems and machines A backup, backups everything At restore, if I select the machine, the software (which is not written yet!) will restore everything. If i select a systems, only its subsystems will be restored. Several systems or subsystem can be selected and restored in the same operation. Now for the most common restore operations, instead of forcing the operator to dig through the tree, i want to have 'preselected' configuration. I think I could do this with 'links' So for groups I will have /machine./system/subsystem/subsubsystem/... but also /shortcut/common backup name/system1/subset of its subsystems/... ./system2/subset of its subsystems/ So that if I select the common backup name in GUI, I restore a common list of systems/subsystems Once the groups/directories have been selected, I query the 'backup table' and find which parameters belongs to the selected (sub)systems and restore only those. That's the idea I would like to be able to compare snapshot of several subsystems (without restoring). I need a tolerance on the parameter which can be stored in the parameter metadata table. If abs(paramX in snapshot 1 - paramX in snapshot 2) tolerance : then print paramX val1, val2, tolerance Then the operator can know what is different from a known working configuration. Well, that's the idea. Also, any comparison where you want to build in a tolerance is going to have to (at some point) hit Python. That is to say, you'll have to read into memory the parts of the file(s) that you want to do the comparison on (also true no matter what interface you use). Is that true eevnt if the tolerance is stored in the h5 file (in the parameter metadata table in my case) ? If you could describe your actual storage layout, we could help you more. If you have a better storage layout, let me know ;-) Currently, sadly no. ViTables does not let you edit. I have toyed with the idea of writing a Traits-based application to do just that. I have some parts of it written, if you or anyone else is interested It actually wouldn't be *that* much work. I don't know of an HDF5 editor, only viewers. I have already so much programming to do, but i could have someone looking at this. Send me whatever you have so I can look at it. ;-) I am interested in h5 because my parameters can be classified in /machine_id/systems/subsystems all of which could be groups. would be nice for browsing and selective restore. Groups are a common feature to all HDF5 libraries (indeed they are the 'H')! I actually found out about hdf5/pytables by looking at what is packaged in python(x,y) I am on linux though... I tested vitables on windows. But couldn't install it on linux though... But that's another thread... on a different mailing list! Cheers, -- E -- BlackBerryreg; DevCon Americas, Oct. 18-20, San Francisco, CA http://p.sf.net/sfu/rim-devcon-copy2 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
