I am currently trying to understand a problem I'm having where no data
is being written to a data set. I think the problem is due to how I
have my data spaces and selections set up, but I'm not sure. Here's the
situation:
The application is being fed data in near real-time. There really isn't
any a priori knowledge of the contents of the data. As the data
arrives, its location within the data set is determined and added to the
hyperslab selection. The end result could be a single hyperslab block,
or it could be many. As an example, the file selection might be
something like:
0: <1024797, 0, 0>, <1024799, 0, 1>
1: <1024797, 0, 3>, <1024799, 0, 5>
2: <1024797, 0, 7>, <1024799, 0, 7>
...and so on. The file is chunked in blocks of 10 in the 1st dimesion,
and the entirety of the 2nd and 3rd dimensions. The application is
designed to write "whole chunks", so every 10 are written together (most
of the time that 1st dimension will be monotonically increasing, but not
always).
The data is buffered before writing. The position in the buffer is
chosen as if the entire block is selected (in this case, it'd be
something like <1024790,0,0>,<1024799,0,119>). Originally, we were
reading the block off disk before writing, but this seemed to result in
a lot of overhead.
It appears to me that the buffer for the selection example above should
consist of a total of 18 values in sequence as if each hyperslab block
were the entirety of the selection... that is, element 0 in the buffer
array would correspond to 1024797,0,0 and element 17 would be
1024799,0,7. Is that correct? That's not how the program is currently
storing data in its buffer, and I think that's why I'm not seeing any
real data on disk.
Is there a sensible way to write data in this fashion (select,
semi-random elements within the data set)? If my statement above is
true, it will be pretty difficult to manage the buffer as subsequent
selections could radically change the required organization of the
buffer, and therefore require significant shuffling of data in memory.
I'm not keen on that.
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
