Hi Kent,
On May 2, 2011, at 9:19 AM, Williams, Norman K wrote:
> I'm implementing an HDF image reader for the ITK library. ITK has a set
> of classes for Image I/O to hide the implementation details of reading and
> writing images. One of the itk::ImageIO feature is streamable reading and
> writing.
>
> Streaming is meant to allow for reading subsets of large images to reduce
> the memory footprint. With a properly implement ImageIO class, you can set
> up a processing pipeline that reads parts of the images, processes them,
> and writes the parts out, only requiring the memory for the current part
> of the image.
>
> This is a big win when processing, for example large time-series.
>
> It looks as though to read or write subsets of an image, you specify the
> desired hyperslab, and the same on writing. But according to the
> documentation, the hyperslab-based partitioning of datasets is a
> 'scatter/gather' process that occurs in memory. This leads me to believe
> when you create a new DataSet or open a new Dataset, the memory of the
> entire dataset is allocated.
>
> So my question is this: What is the 'HDF5 Way' to implement streaming of
> smaller chunks of a dataset?
Hyperslabs are the correct way to select a smaller region of a dataset
in the file. You can read that smaller region into an appropriately sized
memory buffer, without allocating a memory buffer that is the size of the
dataset in the file. Search for "H5Sselect_hyperslab" in the examples
subdirectory of the HDF5 distribution and you will find many use cases to draw
from.
Quincey
> --
> Kent Williams [email protected]
>
>
>
>
> ________________________________
> Notice: This UI Health Care e-mail (including attachments) is covered by the
> Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential
> and may be legally privileged. If you are not the intended recipient, you
> are hereby notified that any retention, dissemination, distribution, or
> copying of this communication is strictly prohibited. Please reply to the
> sender that you have received the message in error, then delete it. Thank
> you.
> ________________________________
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
