Hi Kent,
On May 3, 2011, at 10:16 AM, Williams, Norman K wrote:
> I am using the H5Sselect_hyperslab method for both reading and writing
> already. Additionally when I write out an image, I 'chunk' the output and
> compress it.
>
>
> The question I have is what I can do to minimize the memory footprint.
> Based on my reading in the documentation, the Hyperslab interface actually
> scatters/gathers to/from an in-memory dataset, leading me to believe the
> entire dataset will be allocated in system memory.
>
> So the question is how would I use HDF5 in such a way as to minimize the
> memory footprint in this context?
You are allowed to create a memory dataspace that is different from the
dataset's dataspace in the file. That will allow you to tune the memory
footprint.
Quincey
> On 5/3/11 7:30 AM, "Quincey Koziol" <[email protected]> wrote:
>
>> Hi Kent,
>>
>> On May 2, 2011, at 9:19 AM, Williams, Norman K wrote:
>>
>>> I'm implementing an HDF image reader for the ITK library. ITK has a set
>>> of classes for Image I/O to hide the implementation details of reading
>>> and
>>> writing images. One of the itk::ImageIO feature is streamable reading
>>> and
>>> writing.
>>>
>>> Streaming is meant to allow for reading subsets of large images to
>>> reduce
>>> the memory footprint. With a properly implement ImageIO class, you can
>>> set
>>> up a processing pipeline that reads parts of the images, processes them,
>>> and writes the parts out, only requiring the memory for the current part
>>> of the image.
>>>
>>> This is a big win when processing, for example large time-series.
>>>
>>> It looks as though to read or write subsets of an image, you specify the
>>> desired hyperslab, and the same on writing. But according to the
>>> documentation, the hyperslab-based partitioning of datasets is a
>>> 'scatter/gather' process that occurs in memory. This leads me to
>>> believe
>>> when you create a new DataSet or open a new Dataset, the memory of the
>>> entire dataset is allocated.
>>>
>>> So my question is this: What is the 'HDF5 Way' to implement streaming of
>>> smaller chunks of a dataset?
>>
>> Hyperslabs are the correct way to select a smaller region of a
>> dataset in the file. You can read that smaller region into an
>> appropriately sized memory buffer, without allocating a memory buffer
>> that is the size of the dataset in the file. Search for
>> "H5Sselect_hyperslab" in the examples subdirectory of the HDF5
>> distribution and you will find many use cases to draw from.
>>
>> Quincey
>>
>>> --
>>> Kent Williams [email protected]
>>>
>>>
>>>
>>>
>>> ________________________________
>>> Notice: This UI Health Care e-mail (including attachments) is covered
>>> by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is
>>> confidential and may be legally privileged. If you are not the intended
>>> recipient, you are hereby notified that any retention, dissemination,
>>> distribution, or copying of this communication is strictly prohibited.
>>> Please reply to the sender that you have received the message in error,
>>> then delete it. Thank you.
>>> ________________________________
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> [email protected]
>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
>
> ________________________________
> Notice: This UI Health Care e-mail (including attachments) is covered by the
> Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential
> and may be legally privileged. If you are not the intended recipient, you
> are hereby notified that any retention, dissemination, distribution, or
> copying of this communication is strictly prohibited. Please reply to the
> sender that you have received the message in error, then delete it. Thank
> you.
> ________________________________
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
