Friedhelm, > H5serv thanks good way but: I read the h5serv release notes and notify: > High Performance usage �\ the current release of h5serv > serializes all requests, so would not be suitable for a demanding > environment with a large number of clients and/or high throughput rates....
We've been actually working on a successor product to h5serv -- called HDF Cloud -- which we will be announcing in a couple weeks. HDF Cloud adds significant functionality beyond h5serv, including scalability for large numbers of clients, support for AWS S3 object storage, etc. We have several Beta clients in testing already. > SWMR sounds good but:... What I see, is in early stage. I need also parallel > write for big data loads. SWMR is fairly mature... there's always room for improvement but plenty of HDF users rely on this capability and can share their experience here. Feel free to reach out to me or Dax if you would like to learn more. Cheers, -- Dave Pearah ________________________________ From: Hdf-forum <hdf-forum-boun...@lists.hdfgroup.org> on behalf of ISCaD GmbH <friedhelm.mat...@iscad-it.de> Sent: Tuesday, August 8, 2017 2:24:46 AM To: hdf-forum@lists.hdfgroup.org Subject: Re: [Hdf-forum] hdf5 parallel h5py Hi, I look for SWMR and h5serv: H5serv thanks good way but: I read the h5serv release notes and notify: High Performance usage �\ the current release of h5serv serializes all requests, so would not be suitable for a demanding environment with a large number of clients and/or high throughput rates. and Variable length datatypes - Variable length datatypes are now supported. This are remarkable issues for me. SWMR sounds good but: What I see, is in early stage. I need also parallel write for big data loads. Thanks and regards -----Ursprüngliche Nachricht----- Von: Hdf-forum [mailto:hdf-forum-boun...@lists.hdfgroup.org] Im Auftrag von hdf-forum-requ...@lists.hdfgroup.org Gesendet: Montag, 7. August 2017 17:13 An: hdf-forum@lists.hdfgroup.org Betreff: Hdf-forum Digest, Vol 98, Issue 7 Send Hdf-forum mailing list submissions to hdf-forum@lists.hdfgroup.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org or, via email, send a message with subject or body 'help' to hdf-forum-requ...@lists.hdfgroup.org You can reach the person managing the list at hdf-forum-ow...@lists.hdfgroup.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Hdf-forum digest..." Today's Topics: 1. Re: hdf5 parallel h5py (Jialin Liu) ---------------------------------------------------------------------- Message: 1 Date: Mon, 7 Aug 2017 11:12:23 -0400 From: Jialin Liu <jaln...@lbl.gov> To: hdf-forum@lists.hdfgroup.org Subject: Re: [Hdf-forum] hdf5 parallel h5py Message-ID: <caowawpgnlzz38c0cnpv00ryctgadw9bgwlc+gnqvx_0z7kc...@mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hi, The way to parallel h5py is first to compile h5py with parallel HDF5. If you are looking for examples of parallel H5py, take a look at the nersc webpage: http://www.nersc.gov/users/data-analytics/data-management/i-o-libraries/hdf5-2/h5py/ Best, Jialin On Mon, Aug 7, 2017 at 11:06 AM, <hdf-forum-requ...@lists.hdfgroup.org> wrote: > Send Hdf-forum mailing list submissions to > hdf-forum@lists.hdfgroup.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_ > lists.hdfgroup.org > > or, via email, send a message with subject or body 'help' to > hdf-forum-requ...@lists.hdfgroup.org > > You can reach the person managing the list at > hdf-forum-ow...@lists.hdfgroup.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Hdf-forum digest..." > > > Today's Topics: > > 1. Re: hdf5 parallel h5py (Thomas Caswell) > 2. Re: "File too large" error, seemingly related to MPI > (Quincey Koziol) > 3. Re: VCS URI (Quincey Koziol) > 4. Re: hdf5 parallel h5py (John Readey) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 05 Aug 2017 16:48:23 +0000 > From: Thomas Caswell <tcasw...@gmail.com> > To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>, > "friedhelm.mat...@iscad-it.de" <friedhelm.mat...@iscad-it.de> > Subject: Re: [Hdf-forum] hdf5 parallel h5py > Message-ID: > <CAA48SF_LEzc5tRgMWgTwv03SBOoCvBcFeycP6 > pdrh_mvfp1...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > I would also look at h5serv (https://github.com/HDFGroup/h5serv) which > puts a server in front of your hdf5 file. It serves as a single > process owns it and serves as the serialization point which side-steps > almost all of the multiple-client issues. h5pyd > (https://github.com/HDFGroup/h5pyd) is a client of h5serv which has an > identical high-level API to h5py. > > Tom > > On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <nelso...@llnl.gov> wrote: > > > It doesn?t sound like parallel HDF5 is what you are wanting to do here. > > Parallel HDF5 is for an application where all applications are > > writing > in a > > very coordinated manner. All processes need to write the same > > metadata to the file in ?collective? calls to the library, i.e. each > > application > makes > > the same calls using the same arguments in the same order when > > making > calls > > that modify the file metadata (creating files, datasets or groups, > writing > > attributes, etc. > > <https://support.hdfgroup.org/HDF5/doc/RM/CollectiveCalls.html>). > > > > It sounds like you have separate applications that are executing in > > a somewhat independent manner. This will not work with parallel HDF5. > > > > > > > > Using the serial library, I can think of at least one approach that > > might work well for you. HDF5 1.10 introduced a Single-writer > > multi-reader > (SWMR) > > mode = to open a file. Using a SWMR file for each process, each > > process would open one file as the writer in SWMR mode, and open the > > files from > all > > the other processes as read-only in SWMR mode. > > > > > > > > http://docs.h5py.org/en/latest/swmr.html > > > > > > > > Jarom > > > > > > > > *From:* Hdf-forum [mailto:hdf-forum-boun...@lists.hdfgroup.org] *On > > Behalf Of *ISCaD GmbH > > *Sent:* Thursday, August 3, 2017 1:19 AM > > *To:* hdf-forum@lists.hdfgroup.org > > *Subject:* [Hdf-forum] hdf5 parallel h5py > > > > > > > > Dear all, > > > > > > > > I work on an web application which should store and receive the data > > from an hdf5 file. > > > > > > > > Because several people work with this file and long running > > processes I would like > > > > to use the mpi4py, h5py and HDF5. > > > > > > > > I work on debian linux stretch 64 Bit. > > > > > > > > What?s the way to parallel h5py. > > > > > > > > Thanks and regards > > > > > > > > Friedhelm Matten > > > > > > _______________________________________________ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@lists.hdfgroup.org > > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup. > > org > > Twitter: https://twitter.com/hdf5 > -------------- next part -------------- An HTML attachment was > scrubbed... > URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists. > hdfgroup.org/attachments/20170805/4d84f060/attachment-0001.html> > > ------------------------------ > > Message: 2 > Date: Mon, 7 Aug 2017 09:14:28 -0500 > From: Quincey Koziol <koz...@lbl.gov> > To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org> > Subject: Re: [Hdf-forum] "File too large" error, seemingly related to > MPI > Message-ID: <11a08003-02c4-494d-a746-d86f1c948...@lbl.gov> > Content-Type: text/plain; charset=utf-8 > > Hi Frederic, > Could you give us some more details about your file and the > call(s) you are making to HDF5? I can?t think of any reason that it would > crash when creating a file like this, but something interesting could be > going on? :-) > > Quincey > > > > On Aug 7, 2017, at 5:28 AM, Frederic Perez > > <fredericper...@gmail.com> > wrote: > > > > Hi, > > > > While writing significant amount of data in parallel, I obtain the > > following error stack: > > > > HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 66: > > #000: H5D.c line 194 in H5Dcreate2(): unable to create dataset > > major: Dataset > > minor: Unable to initialize object > > #001: H5Dint.c line 453 in H5D__create_named(): unable to create > > and link to dataset > > major: Dataset > > minor: Unable to initialize object > > #002: H5L.c line 1638 in H5L_link_object(): unable to create new > > link to object > > major: Links > > minor: Unable to initialize object > > #003: H5L.c line 1882 in H5L_create_real(): can't insert link > > major: Symbol table > > minor: Unable to insert object > > #004: H5Gtraverse.c line 861 in H5G_traverse(): internal path > > traversal > failed > > major: Symbol table > > minor: Object not found > > #005: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal > > operator > failed > > major: Symbol table > > minor: Callback failed > > #006: H5L.c line 1685 in H5L_link_cb(): unable to create object > > major: Object header > > minor: Unable to initialize object > > #007: H5O.c line 3016 in H5O_obj_create(): unable to open object > > major: Object header > > minor: Can't open object > > #008: H5Doh.c line 293 in H5O__dset_create(): unable to create dataset > > major: Dataset > > minor: Unable to initialize object > > #009: H5Dint.c line 1060 in H5D__create(): can't update the > > metadata > cache > > major: Dataset > > minor: Unable to initialize object > > #010: H5Dint.c line 852 in H5D__update_oh_info(): unable to update > > layout/pline/efl header message > > major: Dataset > > minor: Unable to initialize object > > #011: H5Dlayout.c line 238 in H5D__layout_oh_create(): unable to > > initialize storage > > major: Dataset > > minor: Unable to initialize object > > #012: H5Dint.c line 1713 in H5D__alloc_storage(): unable to > > initialize dataset with fill value > > major: Dataset > > minor: Unable to initialize object > > #013: H5Dint.c line 1805 in H5D__init_storage(): unable to allocate > > all chunks of dataset > > major: Dataset > > minor: Unable to initialize object > > #014: H5Dchunk.c line 3575 in H5D__chunk_allocate(): unable to > > write raw data to file > > major: Low-level I/O > > minor: Write failed > > #015: H5Dchunk.c line 3745 in H5D__chunk_collective_fill(): unable > > to write raw data to file > > major: Low-level I/O > > minor: Write failed > > #016: H5Fio.c line 171 in H5F_block_write(): write through metadata > > accumulator failed > > major: Low-level I/O > > minor: Write failed > > #017: H5Faccum.c line 825 in H5F__accum_write(): file write failed > > major: Low-level I/O > > minor: Write failed > > #018: H5FDint.c line 260 in H5FD_write(): driver write request failed > > major: Virtual File Layer > > minor: Write failed > > #019: H5FDmpio.c line 1846 in H5FD_mpio_write(): > > MPI_File_write_at_all > failed > > major: Internal error (too specific to document in detail) > > minor: Some MPI function failed > > #020: H5FDmpio.c line 1846 in H5FD_mpio_write(): Other I/O error , > > error stack: > > ADIOI_NFS_WRITESTRIDED(672): Other I/O error File too large > > major: Internal error (too specific to document in detail) > > minor: MPI Error String > > > > > > It basically claims that I am creating a file too large. But I > > verified that the filesystem is capable of handling such a size. In > > my case, the file is around 4 TB when it crashes. Where could this > > problem come from? I thought HDF5 does not have a problem with very > > large files. Plus, I am dividing the file in several datasets, and > > the write operations work perfectly until, at some point, it crashes > > with the errors above. > > > > Could it be an issue with HDF5? Or could it be an MPI limitation? I > > am skeptic about the latter option: at the beginning, the program > > writes several datasets inside the file succesfully (all the > > datasets being the same size). If MPI was to blame, why wouldn't it > > crash at the first write? > > > > Thank you for your help. > > Fred > > > > _______________________________________________ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@lists.hdfgroup.org > > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup. > > org > > Twitter: https://twitter.com/hdf5 > > > > > ------------------------------ > > Message: 3 > Date: Mon, 7 Aug 2017 09:15:41 -0500 > From: Quincey Koziol <koz...@lbl.gov> > To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org> > Subject: Re: [Hdf-forum] VCS URI > Message-ID: <adb59151-50fc-4927-8165-9d2b0c975...@lbl.gov> > Content-Type: text/plain; charset="us-ascii" > > Hi David, > Sure, the git repo is here: https://bitbucket.hdfgroup. > org/projects/HDFFV/repos/hdf5/browse <https://bitbucket.hdfgroup. > org/projects/HDFFV/repos/hdf5/browse> > > Quincey > > > > On Aug 7, 2017, at 5:27 AM, David Seifert <s...@gentoo.org> wrote: > > > > Hi HDF5 team and users, > > is there any possibility for me to develop again the current VCS > > sources for adding pkgconfig + Meson support? This makes development > > and feature addition a lot easier, as there won't be any conflicts. > > > > Regards > > David > > > > _______________________________________________ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@lists.hdfgroup.org > > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup. > > org > > Twitter: https://twitter.com/hdf5 > > -------------- next part -------------- An HTML attachment was > scrubbed... > URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists. > hdfgroup.org/attachments/20170807/bebdf029/attachment-0001.html> > > ------------------------------ > > Message: 4 > Date: Mon, 7 Aug 2017 15:05:28 +0000 > From: John Readey <jrea...@hdfgroup.org> > To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>, > "friedhelm.mat...@iscad-it.de" <friedhelm.mat...@iscad-it.de> > Subject: Re: [Hdf-forum] hdf5 parallel h5py > Message-ID: <82d9ce59-6bbe-4e47-9d05-18de03326...@hdfgroup.org> > Content-Type: text/plain; charset="utf-8" > > Friedhelm, > > I?m not familiar with the specifics of your web app, but another > possibility is to just have the app call h5serv directly. > > Anika @ NASA Goddard wrote a nice blog article on this approach: > https://www.hdfgroup.org/2017/04/the-gfed-analysis-tool-an- > hdf-server-implementation/. > > John > > From: Hdf-forum <hdf-forum-boun...@lists.hdfgroup.org> on behalf of > Thomas Caswell <tcasw...@gmail.com> > Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org> > Date: Saturday, August 5, 2017 at 9:48 AM > To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>, " > friedhelm.mat...@iscad-it.de" <friedhelm.mat...@iscad-it.de> > Subject: Re: [Hdf-forum] hdf5 parallel h5py > > I would also look at h5serv (https://github.com/HDFGroup/h5serv) which > puts a server in front of your hdf5 file. It serves as a single > process owns it and serves as the serialization point which side-steps > almost all of the multiple-client issues. h5pyd > (https://github.com/HDFGroup/h5pyd) > is a client of h5serv which has an identical high-level API to h5py. > > Tom > > On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <nelso...@llnl.gov<mailto: > nelso...@llnl.gov>> wrote: > It doesn?t sound like parallel HDF5 is what you are wanting to do here. > Parallel HDF5 is for an application where all applications are writing > in a very coordinated manner. All processes need to write the same > metadata to the file in ?collective? calls to the library, i.e. each > application makes the same calls using the same arguments in the same > order when making calls that modify the file metadata (creating files, > datasets or groups, writing attributes, > etc.<https://support.hdfgroup.org/HDF5/doc/RM/ > CollectiveCalls.html>). > It sounds like you have separate applications that are executing in a > somewhat independent manner. This will not work with parallel HDF5. > > Using the serial library, I can think of at least one approach that > might work well for you. HDF5 1.10 introduced a Single-writer > multi-reader (SWMR) mode = to open a file. Using a SWMR file for each > process, each process would open one file as the writer in SWMR mode, > and open the files from all the other processes as read-only in SWMR mode. > > http://docs.h5py.org/en/latest/swmr.html > > Jarom > > From: Hdf-forum > [mailto:hdf-forum-boun...@lists.hdfgroup.org<mailto:hdf- > forum-boun...@lists.hdfgroup.org>] On Behalf Of ISCaD GmbH > Sent: Thursday, August 3, 2017 1:19 AM > To: hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org> > Subject: [Hdf-forum] hdf5 parallel h5py > > Dear all, > > I work on an web application which should store and receive the data > from an hdf5 file. > > Because several people work with this file and long running processes > I would like to use the mpi4py, h5py and HDF5. > > I work on debian linux stretch 64 Bit. > > What?s the way to parallel h5py. > > Thanks and regards > > Friedhelm Matten > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or > g > Twitter: https://twitter.com/hdf5 > -------------- next part -------------- An HTML attachment was > scrubbed... > URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists. > hdfgroup.org/attachments/20170807/d8afdcac/attachment.html> > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > Hdf-forum@lists.hdfgroup.org > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or > g > > > ------------------------------ > > End of Hdf-forum Digest, Vol 98, Issue 6 > **************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20170807/9e689fe5/attachment.html> ------------------------------ Subject: Digest Footer _______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org ------------------------------ End of Hdf-forum Digest, Vol 98, Issue 7 **************************************** _______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
