Hi Rob,
On May 28, 2014, at 4:22 PM, Rob Latham <[email protected]> wrote:
>
>
> On 05/27/2014 02:45 PM, Quincey Koziol wrote:
>> On May 27, 2014, at 2:43 PM, Dana Robinson <[email protected]
>> <mailto:[email protected]>> wrote:
>>>
>>> Hi,
>>>
>>> The HDF5 library does not support asynchronous I/O at this time. We
>>> are looking into including async I/O support in a future release, however.
>>>
>>
>> I've attached a document that describes our current ideas in this space.
>
> Good read. Just how compute bound is HDF5, anyway? I'm always living in a
> land of large datasets, where library overhead is dwarfed by the I/O workload
> overhead.
Generally speaking, HDF5 is not compute bound. It's only when an
application asks for a compute-oriented task that something could be expensive
(datatype conversion, compression, etc).
> you did not mention the multi-dataset I/O approach: it's a half-step towards
> asynchronism -- or maybe a half-step backwards -- in that instead of
> decoupling the description of the data with the execution of the data, HDF5's
> multi-dataset routines will describe more data in a single call.
I think multi-dataset reads/writes are neutral on the asynchrony axis -
a multi-dataset I/O operation could be made asynchronous in the same way as any
other operation that touches the file.
> I don't think the global HDF5 lock precludes an async approach. Probably this
> async facility should exist on top of HDF5, though, and can provide the
> caching, read-ahead, coalescing, and other benefits while leaving the bulk of
> the 300k lines of C code untouched. In my head it's MPI_THREAD_FUNNELED for
> HDF5.
I think there's actually a good case for pushing a portion of the
asynchrony inside they HDF5 library, since it allows existing applications
(which aren't using async I/O variants of the API routines) to get the benefit
of asynchronous metadata operations. (ie. flushing dirty metadata to the file
in the background)
> The various ways one can manage MPI progress are instructive here.
Indeed. :-)
Thanks for the feedback,
Quincey
> ==rob
>
>>
>> Quincey
>>
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>> Twitter: https://twitter.com/hdf5
>>
>
> --
> Rob Latham
> Mathematics and Computer Science Division
> Argonne National Lab, IL USA
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
