Hi, I look for SWMR and h5serv:
H5serv thanks good way but: I read the h5serv release notes and notify: High Performance usage ‐ the current release of h5serv serializes all requests, so would not be suitable for a demanding environment with a large number of clients and/or high throughput rates. and Variable length datatypes - Variable length datatypes are now supported. This are remarkable issues for me. SWMR sounds good but: What I see, is in early stage. I need also parallel write for big data loads. Thanks and regards -----Ursprüngliche Nachricht----- Von: Hdf-forum [mailto:hdf-forum-boun...@lists.hdfgroup.org] Im Auftrag von hdf-forum-requ...@lists.hdfgroup.org Gesendet: Montag, 7. August 2017 17:13 An: hdf-forum@lists.hdfgroup.org Betreff: Hdf-forum Digest, Vol 98, Issue 7 Send Hdf-forum mailing list submissions to hdf-forum@lists.hdfgroup.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org or, via email, send a message with subject or body 'help' to hdf-forum-requ...@lists.hdfgroup.org You can reach the person managing the list at hdf-forum-ow...@lists.hdfgroup.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Hdf-forum digest..." Today's Topics: 1. Re: hdf5 parallel h5py (Jialin Liu) ---------------------------------------------------------------------- Message: 1 Date: Mon, 7 Aug 2017 11:12:23 -0400 From: Jialin Liu <jaln...@lbl.gov> To: hdf-forum@lists.hdfgroup.org Subject: Re: [Hdf-forum] hdf5 parallel h5py Message-ID: <caowawpgnlzz38c0cnpv00ryctgadw9bgwlc+gnqvx_0z7kc...@mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hi, The way to parallel h5py is first to compile h5py with parallel HDF5. If you are looking for examples of parallel H5py, take a look at the nersc webpage: http://www.nersc.gov/users/data-analytics/data-management/i-o-libraries/hdf5-2/h5py/ Best, Jialin On Mon, Aug 7, 2017 at 11:06 AM, <hdf-forum-requ...@lists.hdfgroup.org> wrote: > Send Hdf-forum mailing list submissions to > hdf-forum@lists.hdfgroup.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_ > lists.hdfgroup.org > > or, via email, send a message with subject or body 'help' to > hdf-forum-requ...@lists.hdfgroup.org > > You can reach the person managing the list at > hdf-forum-ow...@lists.hdfgroup.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Hdf-forum digest..." > > > Today's Topics: > > 1. Re: hdf5 parallel h5py (Thomas Caswell) > 2. Re: "File too large" error, seemingly related to MPI > (Quincey Koziol) > 3. Re: VCS URI (Quincey Koziol) > 4. Re: hdf5 parallel h5py (John Readey) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 05 Aug 2017 16:48:23 +0000 > From: Thomas Caswell <tcasw...@gmail.com> > To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>, > "friedhelm.mat...@iscad-it.de" <friedhelm.mat...@iscad-it.de> > Subject: Re: [Hdf-forum] hdf5 parallel h5py > Message-ID: > <CAA48SF_LEzc5tRgMWgTwv03SBOoCvBcFeycP6 > pdrh_mvfp1...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > I would also look at h5serv (https://github.com/HDFGroup/h5serv) which > puts a server in front of your hdf5 file. It serves as a single > process owns it and serves as the serialization point which side-steps > almost all of the multiple-client issues. h5pyd > (https://github.com/HDFGroup/h5pyd) is a client of h5serv which has an > identical high-level API to h5py. > > Tom > > On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <nelso...@llnl.gov> wrote: > > > It doesn?t sound like parallel HDF5 is what you are wanting to do here. > > Parallel HDF5 is for an application where all applications are > > writing > in a > > very coordinated manner. All processes need to write the same > > metadata to the file in ?collective? calls to the library, i.e. each > > application > makes > > the same calls using the same arguments in the same order when > > making > calls > > that modify the file metadata (creating files, datasets or groups, > writing > > attributes, etc. > > <https://support.hdfgroup.org/HDF5/doc/RM/CollectiveCalls.html>). > > > > It sounds like you have separate applications that are executing in > > a somewhat independent manner. This will not work with parallel HDF5. > > > > > > > > Using the serial library, I can think of at least one approach that > > might work well for you. HDF5 1.10 introduced a Single-writer > > multi-reader > (SWMR) > > mode = to open a file. Using a SWMR file for each process, each > > process would open one file as the writer in SWMR mode, and open the > > files from > all > > the other processes as read-only in SWMR mode. > > > > > > > > http://docs.h5py.org/en/latest/swmr.html > > > > > > > > Jarom > > > > > > > > *From:* Hdf-forum [mailto:hdf-forum-boun...@lists.hdfgroup.org] *On > > Behalf Of *ISCaD GmbH > > *Sent:* Thursday, August 3, 2017 1:19 AM > > *To:* hdf-forum@lists.hdfgroup.org > > *Subject:* [Hdf-forum] hdf5 parallel h5py > > > > > > > > Dear all, > > > > > > > > I work on an web application which should store and receive the data > > from an hdf5 file. > > > > > > > > Because several people work with this file and long running > > processes I would like > > > > to use the mpi4py, h5py and HDF5. > > > > > > > > I work on debian linux stretch 64 Bit. > > > > > > > > What?s the way to parallel h5py. > > > > > > > > Thanks and regards > > > > > > > > Friedhelm Matten > > > > > > _______________________________________________ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@lists.hdfgroup.org > > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup. > > org > > Twitter: https://twitter.com/hdf5 > -------------- next part -------------- An HTML attachment was > scrubbed... > URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists. > hdfgroup.org/attachments/20170805/4d84f060/attachment-0001.html> > > ------------------------------ > > Message: 2 > Date: Mon, 7 Aug 2017 09:14:28 -0500 > From: Quincey Koziol <koz...@lbl.gov> > To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org> > Subject: Re: [Hdf-forum] "File too large" error, seemingly related to > MPI > Message-ID: <11a08003-02c4-494d-a746-d86f1c948...@lbl.gov> > Content-Type: text/plain; charset=utf-8 > > Hi Frederic, > Could you give us some more details about your file and the > call(s) you are making to HDF5? I can?t think of any reason that it would > crash when creating a file like this, but something interesting could be > going on? :-) > > Quincey > > > > On Aug 7, 2017, at 5:28 AM, Frederic Perez > > <fredericper...@gmail.com> > wrote: > > > > Hi, > > > > While writing significant amount of data in parallel, I obtain the > > following error stack: > > > > HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 66: > > #000: H5D.c line 194 in H5Dcreate2(): unable to create dataset > > major: Dataset > > minor: Unable to initialize object > > #001: H5Dint.c line 453 in H5D__create_named(): unable to create > > and link to dataset > > major: Dataset > > minor: Unable to initialize object > > #002: H5L.c line 1638 in H5L_link_object(): unable to create new > > link to object > > major: Links > > minor: Unable to initialize object > > #003: H5L.c line 1882 in H5L_create_real(): can't insert link > > major: Symbol table > > minor: Unable to insert object > > #004: H5Gtraverse.c line 861 in H5G_traverse(): internal path > > traversal > failed > > major: Symbol table > > minor: Object not found > > #005: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal > > operator > failed > > major: Symbol table > > minor: Callback failed > > #006: H5L.c line 1685 in H5L_link_cb(): unable to create object > > major: Object header > > minor: Unable to initialize object > > #007: H5O.c line 3016 in H5O_obj_create(): unable to open object > > major: Object header > > minor: Can't open object > > #008: H5Doh.c line 293 in H5O__dset_create(): unable to create dataset > > major: Dataset > > minor: Unable to initialize object > > #009: H5Dint.c line 1060 in H5D__create(): can't update the > > metadata > cache > > major: Dataset > > minor: Unable to initialize object > > #010: H5Dint.c line 852 in H5D__update_oh_info(): unable to update > > layout/pline/efl header message > > major: Dataset > > minor: Unable to initialize object > > #011: H5Dlayout.c line 238 in H5D__layout_oh_create(): unable to > > initialize storage > > major: Dataset > > minor: Unable to initialize object > > #012: H5Dint.c line 1713 in H5D__alloc_storage(): unable to > > initialize dataset with fill value > > major: Dataset > > minor: Unable to initialize object > > #013: H5Dint.c line 1805 in H5D__init_storage(): unable to allocate > > all chunks of dataset > > major: Dataset > > minor: Unable to initialize object > > #014: H5Dchunk.c line 3575 in H5D__chunk_allocate(): unable to > > write raw data to file > > major: Low-level I/O > > minor: Write failed > > #015: H5Dchunk.c line 3745 in H5D__chunk_collective_fill(): unable > > to write raw data to file > > major: Low-level I/O > > minor: Write failed > > #016: H5Fio.c line 171 in H5F_block_write(): write through metadata > > accumulator failed > > major: Low-level I/O > > minor: Write failed > > #017: H5Faccum.c line 825 in H5F__accum_write(): file write failed > > major: Low-level I/O > > minor: Write failed > > #018: H5FDint.c line 260 in H5FD_write(): driver write request failed > > major: Virtual File Layer > > minor: Write failed > > #019: H5FDmpio.c line 1846 in H5FD_mpio_write(): > > MPI_File_write_at_all > failed > > major: Internal error (too specific to document in detail) > > minor: Some MPI function failed > > #020: H5FDmpio.c line 1846 in H5FD_mpio_write(): Other I/O error , > > error stack: > > ADIOI_NFS_WRITESTRIDED(672): Other I/O error File too large > > major: Internal error (too specific to document in detail) > > minor: MPI Error String > > > > > > It basically claims that I am creating a file too large. But I > > verified that the filesystem is capable of handling such a size. In > > my case, the file is around 4 TB when it crashes. Where could this > > problem come from? I thought HDF5 does not have a problem with very > > large files. Plus, I am dividing the file in several datasets, and > > the write operations work perfectly until, at some point, it crashes > > with the errors above. > > > > Could it be an issue with HDF5? Or could it be an MPI limitation? I > > am skeptic about the latter option: at the beginning, the program > > writes several datasets inside the file succesfully (all the > > datasets being the same size). If MPI was to blame, why wouldn't it > > crash at the first write? > > > > Thank you for your help. > > Fred > > > > _______________________________________________ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@lists.hdfgroup.org > > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup. > > org > > Twitter: https://twitter.com/hdf5 > > > > > ------------------------------ > > Message: 3 > Date: Mon, 7 Aug 2017 09:15:41 -0500 > From: Quincey Koziol <koz...@lbl.gov> > To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org> > Subject: Re: [Hdf-forum] VCS URI > Message-ID: <adb59151-50fc-4927-8165-9d2b0c975...@lbl.gov> > Content-Type: text/plain; charset="us-ascii" > > Hi David, > Sure, the git repo is here: https://bitbucket.hdfgroup. > org/projects/HDFFV/repos/hdf5/browse <https://bitbucket.hdfgroup. > org/projects/HDFFV/repos/hdf5/browse> > > Quincey > > > > On Aug 7, 2017, at 5:27 AM, David Seifert <s...@gentoo.org> wrote: > > > > Hi HDF5 team and users, > > is there any possibility for me to develop again the current VCS > > sources for adding pkgconfig + Meson support? This makes development > > and feature addition a lot easier, as there won't be any conflicts. > > > > Regards > > David > > > > _______________________________________________ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@lists.hdfgroup.org > > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup. > > org > > Twitter: https://twitter.com/hdf5 > > -------------- next part -------------- An HTML attachment was > scrubbed... > URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists. > hdfgroup.org/attachments/20170807/bebdf029/attachment-0001.html> > > ------------------------------ > > Message: 4 > Date: Mon, 7 Aug 2017 15:05:28 +0000 > From: John Readey <jrea...@hdfgroup.org> > To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>, > "friedhelm.mat...@iscad-it.de" <friedhelm.mat...@iscad-it.de> > Subject: Re: [Hdf-forum] hdf5 parallel h5py > Message-ID: <82d9ce59-6bbe-4e47-9d05-18de03326...@hdfgroup.org> > Content-Type: text/plain; charset="utf-8" > > Friedhelm, > > I?m not familiar with the specifics of your web app, but another > possibility is to just have the app call h5serv directly. > > Anika @ NASA Goddard wrote a nice blog article on this approach: > https://www.hdfgroup.org/2017/04/the-gfed-analysis-tool-an- > hdf-server-implementation/. > > John > > From: Hdf-forum <hdf-forum-boun...@lists.hdfgroup.org> on behalf of > Thomas Caswell <tcasw...@gmail.com> > Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org> > Date: Saturday, August 5, 2017 at 9:48 AM > To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>, " > friedhelm.mat...@iscad-it.de" <friedhelm.mat...@iscad-it.de> > Subject: Re: [Hdf-forum] hdf5 parallel h5py > > I would also look at h5serv (https://github.com/HDFGroup/h5serv) which > puts a server in front of your hdf5 file. It serves as a single > process owns it and serves as the serialization point which side-steps > almost all of the multiple-client issues. h5pyd > (https://github.com/HDFGroup/h5pyd) > is a client of h5serv which has an identical high-level API to h5py. > > Tom > > On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <nelso...@llnl.gov<mailto: > nelso...@llnl.gov>> wrote: > It doesn?t sound like parallel HDF5 is what you are wanting to do here. > Parallel HDF5 is for an application where all applications are writing > in a very coordinated manner. All processes need to write the same > metadata to the file in ?collective? calls to the library, i.e. each > application makes the same calls using the same arguments in the same > order when making calls that modify the file metadata (creating files, > datasets or groups, writing attributes, > etc.<https://support.hdfgroup.org/HDF5/doc/RM/ > CollectiveCalls.html>). > It sounds like you have separate applications that are executing in a > somewhat independent manner. This will not work with parallel HDF5. > > Using the serial library, I can think of at least one approach that > might work well for you. HDF5 1.10 introduced a Single-writer > multi-reader (SWMR) mode = to open a file. Using a SWMR file for each > process, each process would open one file as the writer in SWMR mode, > and open the files from all the other processes as read-only in SWMR mode. > > http://docs.h5py.org/en/latest/swmr.html > > Jarom > > From: Hdf-forum > [mailto:hdf-forum-boun...@lists.hdfgroup.org<mailto:hdf- > forum-boun...@lists.hdfgroup.org>] On Behalf Of ISCaD GmbH > Sent: Thursday, August 3, 2017 1:19 AM > To: hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org> > Subject: [Hdf-forum] hdf5 parallel h5py > > Dear all, > > I work on an web application which should store and receive the data > from an hdf5 file. > > Because several people work with this file and long running processes > I would like to use the mpi4py, h5py and HDF5. > > I work on debian linux stretch 64 Bit. > > What?s the way to parallel h5py. > > Thanks and regards > > Friedhelm Matten > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or > g > Twitter: https://twitter.com/hdf5 > -------------- next part -------------- An HTML attachment was > scrubbed... > URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists. > hdfgroup.org/attachments/20170807/d8afdcac/attachment.html> > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > Hdf-forum@lists.hdfgroup.org > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or > g > > > ------------------------------ > > End of Hdf-forum Digest, Vol 98, Issue 6 > **************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20170807/9e689fe5/attachment.html> ------------------------------ Subject: Digest Footer _______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org ------------------------------ End of Hdf-forum Digest, Vol 98, Issue 7 **************************************** _______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
