Roel, I am a newbie at HDF but have been reading this thread and wanted to suggest an idea. We are working on some new tools that make connections between HDF and data in XML and JSON. As I have read this thread I am wondering if a strategy like this might work:
1) Make a single input file that has all of the configuration arrays you need and make all of these arrays larger than you ever expect to use. Fill them with some fill value. The example you sent had very few items. I'm not sure that is representative, but would empty space in these arrays be a problem? 2) create a json/xml representation of the configuration information which would be the target of the editing interface. Could be edited as text or a lightweight web interface that writes json/xml could be used. Once the data are edited, the xml/json would be converted to HDF, using some tools we are testing here, and the model would run. The json/xml could also be displayed through the web interface in a way that helps users understand what they have and ensure it is correct. These formats could also be tested to make sure errors were not introduced in the editing process. This approach is a bit different, but it is one of the sorts of things we are thinking about as possible future directions. Sounds like this could be an interesting use case. Am I right? Ted ==== Ted Habermann =========================== Director of Earth Science, The HDF Group New phone#: (217) 531-4202 New email: thaberm...@hdfgroup.org ==== thaberm...@hdfgroup.org ================== On May 16, 2013, at 6:21 AM, Roel Vanhout <rvanh...@riks.nl> wrote: > Hi, > > > For me personally it wouldn't be much of a problem - I would write my own > command line tools that drive the baseline tools anyway, or amend my existing > C++ tools to resize arrays. However the whole point of me switching to hdf5 > would be that other users (internally and externally) would have easy-to-use, > GUI tools for working with this data. Some users struggle with the concept of > an array with more than 2 dimensions - suggesting they work with command line > tools, especially ones with large amounts of options and manuals of dozens of > pages, would be a regression in terms of support burden for me. > > While HDF Explorer seems very capable in most respects and is very reasonably > priced, it doesn't let you resize data sets, which is a showstopper for me. > > Thanks for taking the time to think about my issues! > > > regards, > > Roel > > > > > -----Original Message----- > From: Hdf-forum [mailto:hdf-forum-boun...@hdfgroup.org] On Behalf Of H. Joe > Lee > Sent: Wednesday, May 15, 2013 20:33 > To: HDF Users Discussion List > Subject: Re: [Hdf-forum] Tools to edit hdf5 files? > > Hi, Roel! > > Thanks for your reply. I hope someone can write an easy-to-use GUI tool like > MS Excel for HDF data array manipulation you described in near future. > > If you're not limited to GUI tool, I think script-language tool like NCL > (free) IDL/MATLAB (commercial) can do the array manipulation you described. > > For example, you can easily read & subset HDF5 array [1], reshape the array > [2], and save it [3] in NetCDF-4 (which is HDF5 under the > hood) in NCL. The only drawback of NCL compared to IDL/MATLAB is that it > doesn't run on Windows but it supports Cygwin so you may want to try it. > > > [1] http://hdfeos.org/zoo/NSIDC/GLAH13_633_2103_001_1317_0_01_0001.h5.ncl > [2] > http://hdfeos.org/zoo/LaRC/MISR_AM1_GRP_ELLIPSOID_GM_P117_O058421_BA_F03_0024_Blue_Radiance_RDQI.ncl > [3] http://www.ncl.ucar.edu/Applications/netcdf4.shtml > > -- > HDF: Software that Powers Science > > > On Wed, May 15, 2013 at 5:10 AM, Roel Vanhout <rvanh...@riks.nl> wrote: >> Hello, >> >> Thanks for the replies. In response to H. Joe Lee, my application is >> in earth sciences, more specifically in land use modeling, and my data >> is configuration data for the model. It is in that sense that I may be >> using >> hdf5 in an untypical way, it is not data that is acquired by sensors >> or such, but the data in my hdf5 files represents calibration >> parameters for the model. Hence the need to be able to easily edit >> them, and to resize arrays - for example when you want to change the >> number of land use classes in a specific simulation, you need to be >> able to resize a 10x500x600 array to 8x500x600 (for example). We now >> use a multitude of solutions - xml files, 2d matrices in single files >> where the number of files is the 3rd dimension, etc. >> >> As for the question below, it is mostly (2), and sometimes (1). >> Example of >> (2) I gave earlier, an example of (1) is time series of 2d data - in >> some cases you enter values for e.g. 2000 and 2030, and the software >> interpolates the values for the years between that, but a user may >> enter data for 2000, 2010, 2020 and 2030. In that case, the length of >> that dimension of the array would be 4, whereas in the first case it would >> be 2. >> >> regards >> >> Roel >> >> >> >> ________________________________ >> From: Hdf-forum [mailto:hdf-forum-boun...@hdfgroup.org] On Behalf Of >> Peter Cao >> Sent: Friday, May 10, 2013 17:17 >> To: hdf-forum@hdfgroup.org >> Subject: Re: [Hdf-forum] Tools to edit hdf5 files? >> >> Could you be more specific on resizing data? Is it any of the following? >> >> 1) you defined your data as variable size and add/delete data from it, >> 2) you define fixed length of dimensions and extend/shrink the >> dimensions, or >> 3) you define as 2D and change it to 1D, 3D, etc. >> >> Thanks >> --pc >> >> On 5/8/2013 4:00 AM, Roel Vanhout wrote: >> >> Hello, >> >> Apologies if this is a question with an obvious answer, I've searched >> the web and archives long and hard and couldn't find an answer. >> >> I'm looking to replace some of the data files of our software with >> hdf5 to make it easier to edit them with external tools. Specifically, >> I need a way to edit multi-dimensional arrays of numeric data, for >> which I figures hdf5 would be a natural fit. However what I need are arrays >> with variable sizes. >> It seems to be complicated to do that through the API, none of the >> popular tools (HDFView, HDF Explorer) seem to support resizing data, >> and ViTables I haven't been able to get to run on Windows. >> >> So my question is - is hdf5 suitable as a data format for data with >> varying dimensions? What tools will let me work with that? Thanks. >> >> regards, >> >> Roel >> >> >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> Hdf-forum@hdfgroup.org >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> Hdf-forum@hdfgroup.org >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > Hdf-forum@hdfgroup.org > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > Hdf-forum@hdfgroup.org > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@hdfgroup.org http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
