Hi,

I look for SWMR and h5serv:

H5serv thanks good way but:

I read the h5serv release notes and notify:

High Performance usage ‐ the current release of h5serv serializes all requests, 
so would not be suitable for a demanding environment with a large number of 
clients and/or high throughput rates.

and

Variable length datatypes - Variable length datatypes are now supported.

This are remarkable issues for me.

SWMR sounds good but:

What I see, is in  early stage.
I need also parallel write for big data loads.


Thanks and regards


-----Ursprüngliche Nachricht-----
Von: Hdf-forum [mailto:hdf-forum-boun...@lists.hdfgroup.org] Im Auftrag von 
hdf-forum-requ...@lists.hdfgroup.org
Gesendet: Montag, 7. August 2017 17:13
An: hdf-forum@lists.hdfgroup.org
Betreff: Hdf-forum Digest, Vol 98, Issue 7

Send Hdf-forum mailing list submissions to
        hdf-forum@lists.hdfgroup.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

or, via email, send a message with subject or body 'help' to
        hdf-forum-requ...@lists.hdfgroup.org

You can reach the person managing the list at
        hdf-forum-ow...@lists.hdfgroup.org

When replying, please edit your Subject line so it is more specific than "Re: 
Contents of Hdf-forum digest..."


Today's Topics:

   1. Re: hdf5 parallel h5py (Jialin Liu)


----------------------------------------------------------------------

Message: 1
Date: Mon, 7 Aug 2017 11:12:23 -0400
From: Jialin Liu <jaln...@lbl.gov>
To: hdf-forum@lists.hdfgroup.org
Subject: Re: [Hdf-forum] hdf5 parallel h5py
Message-ID:
        <caowawpgnlzz38c0cnpv00ryctgadw9bgwlc+gnqvx_0z7kc...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,
The way to parallel h5py is first to compile h5py with parallel HDF5. If you 
are looking for examples of parallel H5py, take a look at the nersc
webpage:
http://www.nersc.gov/users/data-analytics/data-management/i-o-libraries/hdf5-2/h5py/

Best,
Jialin


On Mon, Aug 7, 2017 at 11:06 AM, <hdf-forum-requ...@lists.hdfgroup.org>
wrote:

> Send Hdf-forum mailing list submissions to
>         hdf-forum@lists.hdfgroup.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_
> lists.hdfgroup.org
>
> or, via email, send a message with subject or body 'help' to
>         hdf-forum-requ...@lists.hdfgroup.org
>
> You can reach the person managing the list at
>         hdf-forum-ow...@lists.hdfgroup.org
>
> When replying, please edit your Subject line so it is more specific 
> than "Re: Contents of Hdf-forum digest..."
>
>
> Today's Topics:
>
>    1. Re: hdf5 parallel h5py (Thomas Caswell)
>    2. Re: "File too large" error, seemingly related to MPI
>       (Quincey Koziol)
>    3. Re: VCS URI (Quincey Koziol)
>    4. Re: hdf5 parallel h5py (John Readey)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 05 Aug 2017 16:48:23 +0000
> From: Thomas Caswell <tcasw...@gmail.com>
> To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>,
>         "friedhelm.mat...@iscad-it.de" <friedhelm.mat...@iscad-it.de>
> Subject: Re: [Hdf-forum] hdf5 parallel h5py
> Message-ID:
>         <CAA48SF_LEzc5tRgMWgTwv03SBOoCvBcFeycP6
> pdrh_mvfp1...@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> I would also look at h5serv (https://github.com/HDFGroup/h5serv) which 
> puts a server in front of your hdf5 file.  It serves as a single 
> process owns it and serves as the serialization point which side-steps 
> almost all of the multiple-client issues.  h5pyd 
> (https://github.com/HDFGroup/h5pyd) is a client of h5serv which has an 
> identical high-level API to h5py.
>
> Tom
>
> On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <nelso...@llnl.gov> wrote:
>
> > It doesn?t sound like parallel HDF5 is what you are wanting to do here.
> > Parallel HDF5 is for an application where all applications are 
> > writing
> in a
> > very coordinated manner. All processes need to write the same 
> > metadata to the file in ?collective? calls to the library, i.e. each 
> > application
> makes
> > the same calls using the same arguments in the same order when 
> > making
> calls
> > that modify the file metadata (creating files, datasets or groups,
> writing
> > attributes, etc.
> > <https://support.hdfgroup.org/HDF5/doc/RM/CollectiveCalls.html>).
> >
> > It sounds like you have separate applications that are executing in 
> > a somewhat independent manner. This will not work with parallel HDF5.
> >
> >
> >
> > Using the serial library, I can think of at least one approach that 
> > might work well for you. HDF5 1.10 introduced a Single-writer 
> > multi-reader
> (SWMR)
> > mode = to open a file. Using a SWMR file for each process, each 
> > process would open one file as the writer in SWMR mode, and open the 
> > files from
> all
> > the other processes as read-only in SWMR mode.
> >
> >
> >
> > http://docs.h5py.org/en/latest/swmr.html
> >
> >
> >
> > Jarom
> >
> >
> >
> > *From:* Hdf-forum [mailto:hdf-forum-boun...@lists.hdfgroup.org] *On 
> > Behalf Of *ISCaD GmbH
> > *Sent:* Thursday, August 3, 2017 1:19 AM
> > *To:* hdf-forum@lists.hdfgroup.org
> > *Subject:* [Hdf-forum] hdf5 parallel h5py
> >
> >
> >
> > Dear all,
> >
> >
> >
> > I work on an web application which should store and receive the data 
> > from an hdf5 file.
> >
> >
> >
> > Because several people work with this file and long running 
> > processes I would like
> >
> > to use the mpi4py, h5py and HDF5.
> >
> >
> >
> > I work on debian linux stretch 64 Bit.
> >
> >
> >
> > What?s the way to parallel h5py.
> >
> >
> >
> > Thanks and regards
> >
> >
> >
> > Friedhelm Matten
> >
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > Hdf-forum@lists.hdfgroup.org
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.
> > org
> > Twitter: https://twitter.com/hdf5
> -------------- next part -------------- An HTML attachment was 
> scrubbed...
> URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.
> hdfgroup.org/attachments/20170805/4d84f060/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 7 Aug 2017 09:14:28 -0500
> From: Quincey Koziol <koz...@lbl.gov>
> To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
> Subject: Re: [Hdf-forum] "File too large" error, seemingly related to
>         MPI
> Message-ID: <11a08003-02c4-494d-a746-d86f1c948...@lbl.gov>
> Content-Type: text/plain; charset=utf-8
>
> Hi Frederic,
>         Could you give us some more details about your file and the
> call(s) you are making to HDF5?   I can?t think of any reason that it would
> crash when creating a file like this, but something interesting could be
> going on?   :-)
>
>         Quincey
>
>
> > On Aug 7, 2017, at 5:28 AM, Frederic Perez 
> > <fredericper...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > While writing significant amount of data in parallel, I obtain the 
> > following error stack:
> >
> > HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 66:
> >  #000: H5D.c line 194 in H5Dcreate2(): unable to create dataset
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #001: H5Dint.c line 453 in H5D__create_named(): unable to create 
> > and link to dataset
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #002: H5L.c line 1638 in H5L_link_object(): unable to create new 
> > link to object
> >    major: Links
> >    minor: Unable to initialize object
> >  #003: H5L.c line 1882 in H5L_create_real(): can't insert link
> >    major: Symbol table
> >    minor: Unable to insert object
> >  #004: H5Gtraverse.c line 861 in H5G_traverse(): internal path 
> > traversal
> failed
> >    major: Symbol table
> >    minor: Object not found
> >  #005: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal 
> > operator
> failed
> >    major: Symbol table
> >    minor: Callback failed
> >  #006: H5L.c line 1685 in H5L_link_cb(): unable to create object
> >    major: Object header
> >    minor: Unable to initialize object
> >  #007: H5O.c line 3016 in H5O_obj_create(): unable to open object
> >    major: Object header
> >    minor: Can't open object
> >  #008: H5Doh.c line 293 in H5O__dset_create(): unable to create dataset
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #009: H5Dint.c line 1060 in H5D__create(): can't update the 
> > metadata
> cache
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #010: H5Dint.c line 852 in H5D__update_oh_info(): unable to update 
> > layout/pline/efl header message
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #011: H5Dlayout.c line 238 in H5D__layout_oh_create(): unable to 
> > initialize storage
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #012: H5Dint.c line 1713 in H5D__alloc_storage(): unable to 
> > initialize dataset with fill value
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #013: H5Dint.c line 1805 in H5D__init_storage(): unable to allocate 
> > all chunks of dataset
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #014: H5Dchunk.c line 3575 in H5D__chunk_allocate(): unable to 
> > write raw data to file
> >    major: Low-level I/O
> >    minor: Write failed
> >  #015: H5Dchunk.c line 3745 in H5D__chunk_collective_fill(): unable 
> > to write raw data to file
> >    major: Low-level I/O
> >    minor: Write failed
> >  #016: H5Fio.c line 171 in H5F_block_write(): write through metadata 
> > accumulator failed
> >    major: Low-level I/O
> >    minor: Write failed
> >  #017: H5Faccum.c line 825 in H5F__accum_write(): file write failed
> >    major: Low-level I/O
> >    minor: Write failed
> >  #018: H5FDint.c line 260 in H5FD_write(): driver write request failed
> >    major: Virtual File Layer
> >    minor: Write failed
> >  #019: H5FDmpio.c line 1846 in H5FD_mpio_write(): 
> > MPI_File_write_at_all
> failed
> >    major: Internal error (too specific to document in detail)
> >    minor: Some MPI function failed
> >  #020: H5FDmpio.c line 1846 in H5FD_mpio_write(): Other I/O error , 
> > error stack:
> > ADIOI_NFS_WRITESTRIDED(672): Other I/O error File too large
> >    major: Internal error (too specific to document in detail)
> >    minor: MPI Error String
> >
> >
> > It basically claims that I am creating a file too large. But I 
> > verified that the filesystem is capable of handling such a size. In 
> > my case, the file is around 4 TB when it crashes. Where could this 
> > problem come from? I thought HDF5 does not have a problem with very 
> > large files. Plus, I am dividing the file in several datasets, and 
> > the write operations work perfectly until, at some point, it crashes 
> > with the errors above.
> >
> > Could it be an issue with HDF5? Or could it be an MPI limitation? I 
> > am skeptic about the latter option: at the beginning, the program 
> > writes several datasets inside the file succesfully (all the 
> > datasets being the same size). If MPI was to blame, why wouldn't it 
> > crash at the first write?
> >
> > Thank you for your help.
> > Fred
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > Hdf-forum@lists.hdfgroup.org
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.
> > org
> > Twitter: https://twitter.com/hdf5
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 7 Aug 2017 09:15:41 -0500
> From: Quincey Koziol <koz...@lbl.gov>
> To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
> Subject: Re: [Hdf-forum] VCS URI
> Message-ID: <adb59151-50fc-4927-8165-9d2b0c975...@lbl.gov>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi David,
>         Sure, the git repo is here:  https://bitbucket.hdfgroup.
> org/projects/HDFFV/repos/hdf5/browse <https://bitbucket.hdfgroup.
> org/projects/HDFFV/repos/hdf5/browse>
>
>                 Quincey
>
>
> > On Aug 7, 2017, at 5:27 AM, David Seifert <s...@gentoo.org> wrote:
> >
> > Hi HDF5 team and users,
> > is there any possibility for me to develop again the current VCS 
> > sources for adding pkgconfig + Meson support? This makes development 
> > and feature addition a lot easier, as there won't be any conflicts.
> >
> > Regards
> > David
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > Hdf-forum@lists.hdfgroup.org
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.
> > org
> > Twitter: https://twitter.com/hdf5
>
> -------------- next part -------------- An HTML attachment was 
> scrubbed...
> URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.
> hdfgroup.org/attachments/20170807/bebdf029/attachment-0001.html>
>
> ------------------------------
>
> Message: 4
> Date: Mon, 7 Aug 2017 15:05:28 +0000
> From: John Readey <jrea...@hdfgroup.org>
> To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>,
>         "friedhelm.mat...@iscad-it.de" <friedhelm.mat...@iscad-it.de>
> Subject: Re: [Hdf-forum] hdf5 parallel h5py
> Message-ID: <82d9ce59-6bbe-4e47-9d05-18de03326...@hdfgroup.org>
> Content-Type: text/plain; charset="utf-8"
>
> Friedhelm,
>
>    I?m not familiar with the specifics of your web app, but another 
> possibility is to just have the app call h5serv directly.
>
>    Anika @ NASA Goddard wrote a nice blog article on this approach:
> https://www.hdfgroup.org/2017/04/the-gfed-analysis-tool-an-
> hdf-server-implementation/.
>
> John
>
> From: Hdf-forum <hdf-forum-boun...@lists.hdfgroup.org> on behalf of 
> Thomas Caswell <tcasw...@gmail.com>
> Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
> Date: Saturday, August 5, 2017 at 9:48 AM
> To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>, "
> friedhelm.mat...@iscad-it.de" <friedhelm.mat...@iscad-it.de>
> Subject: Re: [Hdf-forum] hdf5 parallel h5py
>
> I would also look at h5serv (https://github.com/HDFGroup/h5serv) which 
> puts a server in front of your hdf5 file.  It serves as a single 
> process owns it and serves as the serialization point which side-steps 
> almost all of the multiple-client issues.  h5pyd 
> (https://github.com/HDFGroup/h5pyd)
> is a client of h5serv which has an identical high-level API to h5py.
>
> Tom
>
> On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <nelso...@llnl.gov<mailto:
> nelso...@llnl.gov>> wrote:
> It doesn?t sound like parallel HDF5 is what you are wanting to do here.
> Parallel HDF5 is for an application where all applications are writing 
> in a very coordinated manner. All processes need to write the same 
> metadata to the file in ?collective? calls to the library, i.e. each 
> application makes the same calls using the same arguments in the same 
> order when making calls that modify the file metadata (creating files, 
> datasets or groups, writing attributes, 
> etc.<https://support.hdfgroup.org/HDF5/doc/RM/
> CollectiveCalls.html>).
> It sounds like you have separate applications that are executing in a 
> somewhat independent manner. This will not work with parallel HDF5.
>
> Using the serial library, I can think of at least one approach that 
> might work well for you. HDF5 1.10 introduced a Single-writer 
> multi-reader (SWMR) mode = to open a file. Using a SWMR file for each 
> process, each process would open one file as the writer in SWMR mode, 
> and open the files from all the other processes as read-only in SWMR mode.
>
> http://docs.h5py.org/en/latest/swmr.html
>
> Jarom
>
> From: Hdf-forum 
> [mailto:hdf-forum-boun...@lists.hdfgroup.org<mailto:hdf-
> forum-boun...@lists.hdfgroup.org>] On Behalf Of ISCaD GmbH
> Sent: Thursday, August 3, 2017 1:19 AM
> To: hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>
> Subject: [Hdf-forum] hdf5 parallel h5py
>
> Dear all,
>
> I work on an web application which should store and receive the data 
> from an hdf5 file.
>
> Because several people work with this file and long running processes 
> I would like to use the mpi4py, h5py and HDF5.
>
> I work on debian linux stretch 64 Bit.
>
> What?s the way to parallel h5py.
>
> Thanks and regards
>
> Friedhelm Matten
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
> Twitter: https://twitter.com/hdf5
> -------------- next part -------------- An HTML attachment was 
> scrubbed...
> URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.
> hdfgroup.org/attachments/20170807/d8afdcac/attachment.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
>
>
> ------------------------------
>
> End of Hdf-forum Digest, Vol 98, Issue 6
> ****************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
<http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20170807/9e689fe5/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


------------------------------

End of Hdf-forum Digest, Vol 98, Issue 7
****************************************


_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Reply via email to