Hi John,
On 8/22/2012 8:56 AM, Biddiscombe, John A. wrote:
Mohamad
**
*herr_t H5Dtransfer (hid_t src_space_id, const void *src_buf, H5T_t
type, hid_t dst_space_id, /*out*/void *dst_buf);*
I'm not certain that this will be any use on its own. If a selection
is gigabytes and the VOL layer only has a small buffer available, then
it needs to be able to make these transfers in pieces, performing
puts/sends or whatever as appropriate. The function would be more
useful if it was re-entrant and had a buffer size_type so that the
selection could be copied by the first N, then next N, then next N
until exhausted.
This would be more of what the second routine would do (as you mention
later). This routine was originally gather/scatter routines that we have
mostly implemented internally and would like to expose for users. We
then decided to make it more general.
Granted that if the data is very large compared to memory available, the
second routine would be better to break up the requests in case of a
client-server plugin.
*herr_t H5Sselect_iterate (hid_t dataspace_id, H5S_select_iterator_t
op, void *op_data);*
This function would be great and should be higher priority than the
first because you can do the first using this second one - and don't
have the problem of the limited buffer size. The user can iterate over
the dataspace and copy as much or as little on each entry to the
callback function as desired and maintain their own book keeping of
where they left off. If the selection has huge contiguous chunks, the
user callback can break these into pieces and perform the appropriate
copies as substeps. If the selection is very sparse, then an internal
buffer can be filled and acted upon as the iterations progress.
The reason that the first is a priority I would say is because that it
is very easy to implement (if not already implemented internally) and
would fit our needs directly. We are still scoping user needs for the
iterate routine and see if it needs to be adjusted.
the callback function
*herr_t (*H5S_select_iterate_t)(hsize_t *offset_coords, hsize_t
length, void * op_data);*
could be improved by adding a user callback void *pointer so that when
you call iterate -- you pass a pointer to the function - and also a
user pointer to a data structure of the user's choice, which is passed
to the callback as a user parameter. This way we can track
intermediate transfer objects (like if we only partially transferred
data or are filling an internal buffer) and in the case of multiple
threads acting on these iterations, we can make sure each thread has
its own data pointer an avoid static/global objects which will not be
safe.
I'm not sure I quite understand what you are suggesting here. the
callback (H5S_select_iterator_t op) IS a user callback. The user
specifies also his own udata (void *op_data) in the iterate routine,
that will get passed to his specified callback too. The select_iterate
routine here is similar to, for example, the dataset iterate
(H5Diterate) operation that gets called on elements of a dataset, but
here it calls on selection coordinates.
If you want another callback, you can add it the the user data (op_data)
or just call it from the user callback (op).
Thanks,
Mohamad
I just wrote this off the top of my head, so criticism welcome.
JB
-----Original Message-----
From: [email protected] [mailto:[email protected]]
On Behalf Of Mohamad Chaarawi
Sent: 21 August 2012 23:45
To: HDF5 Virtual Object Layer (VOL) Discussions;
[email protected]; [email protected]
Subject: [H5vol] RFC for new Dataspace routines
Hi All,
Please find attached an RFC that describes a couple of dataspace
routines that we plan to add to the HDF5 API in the near future. If
you have the time, please give it a read and feel free to send us
comments.
We would like to hear from you if you see that you could benefit from
those routines but would like to change something or would like us to
consider adding other routines. It would be great, in either case, if
you could include your use case.
Thank you,
Mohamad
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
