Hi Jason,
I was facing the same issues as pretty much all use case I know and
have in my visualization software and context use and require "fortran"
order of indexing, including OpenGL graphics. It's not really an issue
with HDF5 as the only thing required is to permute the indices when
accessing the HDF5 API. And the HDF5 tools of course will display data
transposed then. This index permutation is supported in the F5 library
via a generic permutation vector that is stored with a group of dataset
sharing the same properties (the F5 library is a C library on top of
HDF5 guiding towards a specific data model for various classes of data
types occurring particularly in scientific visualization):
http://www.fiberbundle.net/doc/structChartDomain__IDs.html
So via the F5 API one would see the fortran-like indexing convention,
whereas whenever accessing data with the lower-level HDF5 API, it's
C-like convention (whereby the permutation vector gives the option of
arbitrary permutations).
I remember there had been plans by the HDF5 group to introduce "named
dataspaces", similarly to "named datatypes", that could then be stored
in the file as its own entity. Such would be a good place to store
properties of a dataspace as attributes on a dataspace, and to have such
shared among datasets. It would be a natural place to store a
permutation vector, which could be reduced to a simple flag as well to
just distinguish between C and fortran indexing conventions. Of course,
all the related tools would also need to honor such an attribute then.
Until then, one could use an attribute on each dataset and implement
index permutation similar to how the F5 library does it. It may be safer
to use new API functions anyway to not break old code that always
expects C order indexing.
Werner
On 12.05.2015 06:48, Jason Newton wrote:
Hi -
I've been a evangelist for HDF5 for a few of years now, it is a noble
and amazing library that solves data storage issues occurring with
scientific and beyond applications - e.g. it can save many developers
from wasting time and money so they can spend that on solving more
original problems. But you guys knew that already. I think there's
been a mistake though - that is the lack of first class column-vs-row
major storage. In a world where we are split down the middle on what
format we used based on what application, library and language we use
we work in one or the other it is an ongoing reality that there will
never be one true standard to follow. But HDF5 sought to only support
row-major - and I can back that up - standardizing is a good thing.
But then as time has shown, that really didn't work for alot of folks
- such as those in Matlab and fortran - when they read our data - it
looks transposed to them! When HDF5 utils/our code sees their data -
it looks transposed to us! These are arguably the users you do not
want to face these difficulties as it makes it down right
embarrassing at times and hard to work around in within that language
(ahem, Matlab again is painful to work with). Not only that but it
doesn't really scale - it will always take some manual fixing and
there's no standardized mark for whether a dataset is one of these
column major masquerading datasets. So let me assure you this is
quite ugly to deal with in Matlab/etc and doesn't seem to be the path
many people take - and it can require skills many people don't have or
understanding that they can't give.
But then, why did we allow saving column major data in a row based
standard in the first place? Well, the answer seems to be
performance. Surely it can't take that long to convert the datasets -
most of the time at least - although there would for sure be some
memory based limitations to allow transposing just as HDF IOs. But
alas - the current state of the library indicates otherwise and thus
is the users job to handle correctly transforming the data back and
forth between application and party. But wait - wasn't this kind of
activity what HDF5 was built to alleviate in the first place?
So then how do we rectify the situation? Well speaking as a developer
using HDF5 extensively and writing libraries for it - it looks to me
it should be in the core library as it is exceedingly messy to handle
on the user side each time. I think the interpretation of the dataset
and it's dimensions should be based on dataset creation properties.
This would allow an official marking of what kind of interpretation
the raw storage of the data (and dimensions?) are. However, this is
only half of the battle. We'd need something like the type conversion
system to permute order in all the right places if the user needs to
IO an opposing storage layout. And it should be fast and light on
memory. Perhaps it would merely operate inplace as a new utility
subroutine taking in the mem_type and user memory. However I can still
think of one problem this does not address: compound types using a
mixture of philosophies with fields being the opposite to the dataset
layout - and this case has me completely stumped as this indicates it
should be type level as well. The compound part of this is a sticky
situation but I'd still motion that the dataset creation property
works for most things that occur in practice.
So... has the HDF5 group tried to deal with this wart yet? Let me know
if anything is on the drawing board.
-Jason
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
