Hi Andrew,
Thanks for the clarification. Thank you also for the introduction to
H5Diterate. I had never noticed that function before.
In the most general case, your solution will not work for me, at least
without some level of recursion, I think. This goes back to needing
different allocators depending on the type of data being converted. So,
for instance, if I have a vlen of vlens of doubles, the memory allocator
I need for the inner vlen is different than the allocator I need for the
outer. You can only set one allocator for any given convert (or read)
call, so the implicit recursive conversion of the inner vlen would end
up calling the wrong allocator. If the allocators knew which datatype
they were allocating for, this would not be an issue. One might also be
able to do something really whacked out like predict the order in which
the different allocators would be called for a complete conversion (e.g.
outer allocator, inner, inner, inner, out, 6*in, out, 3*in, or whatever)
and pass that information into a higher level allocator, but that seems
really nasty.
If I understand correctly, it also has the draw-back of requiring and
extra buffer, namely the one that you allocate before calling H5Dread.
On the plus side, your solution, unlike my solution, is not dependent on
a bug in the library!
Thanks for the input.
Jason
On 9/28/2012 11:25 AM, Andrew Collette wrote:
Hi,
Andrew, thanks for your input.
Where do you do these operations (read to a conversion buffer, etc.)? Do you
do this as a separate call, e.g.
a) H5Dread returns buffer followed by n calls to H5Tconvert
or do you do it inside a custom conversion function
b) H5Dread calls custom conversion which calls H5Tconvert
It's close to (a), but even simpler than that. First I do an H5Dread
into a contiguous buffer with the same datatype as the dataset; this
seems to get rid of the odd internal structures. Then, I convert the
entire buffer in-place from the dataset type to the (opaque) memory
type with a single call to H5Tconvert (you can specify the number of
elements to be converted). As part of the normal course of
conversion, H5Tconvert calls my custom conversion function. Then, I
scatter the points from the conversion buffer to the destination in
memory. I use H5Diterate to do the scattering and gathering.
This has the advantage that my custom conversion callback can be
written as normal. All the workarounds are in the read/write layer,
not the conversion layer.
With regard to your followup post... I do have a hard conversion path
registered from vlen string to Python string, and with this system
it's called as normal. I have not tried generic vlen data myself but
one of my users just contributed a patch which does more or less the
same thing with hvl_t's; I don't know whether this has to be
registered as hard or soft to work.
Andrew
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
