Hi,

the database for the web application is a lot of measured data stream and
data which come in on events.
in a staging area the data will collected and then transferred into the web
app data area.
Some data must be collected via forms.

That’s the orchestra with a lot of hierarchical data which - you all know -
not the best thing for traditional sql.

Other people use the system for reporting, summary and statistics.

We have a database version with high availability and a lot of work for
stability and ... and  ...

We search for new ways ...???!!

Thanks and regards

Friedhelm

-----Urspr√ľngliche Nachricht-----
Von: Hdf-forum [mailto:hdf-forum-boun...@lists.hdfgroup.org] Im Auftrag von
hdf-forum-requ...@lists.hdfgroup.org
Gesendet: Montag, 7. August 2017 17:06
An: hdf-forum@lists.hdfgroup.org
Betreff: Hdf-forum Digest, Vol 98, Issue 6

Send Hdf-forum mailing list submissions to
        hdf-forum@lists.hdfgroup.org

To subscribe or unsubscribe via the World Wide Web, visit
        
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

or, via email, send a message with subject or body 'help' to
        hdf-forum-requ...@lists.hdfgroup.org

You can reach the person managing the list at
        hdf-forum-ow...@lists.hdfgroup.org

When replying, please edit your Subject line so it is more specific than
"Re: Contents of Hdf-forum digest..."


Today's Topics:

   1. Re: hdf5 parallel h5py (Thomas Caswell)
   2. Re: "File too large" error, seemingly related to MPI
      (Quincey Koziol)
   3. Re: VCS URI (Quincey Koziol)
   4. Re: hdf5 parallel h5py (John Readey)


----------------------------------------------------------------------

Message: 1
Date: Sat, 05 Aug 2017 16:48:23 +0000
From: Thomas Caswell <tcasw...@gmail.com>
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>,
        "friedhelm.mat...@iscad-it.de" <friedhelm.mat...@iscad-it.de>
Subject: Re: [Hdf-forum] hdf5 parallel h5py
Message-ID:
        <caa48sf_lezc5trgmwgtwv03sboocvbcfeycp6pdrh_mvfp1...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I would also look at h5serv (https://github.com/HDFGroup/h5serv) which puts
a server in front of your hdf5 file.  It serves as a single process owns it
and serves as the serialization point which side-steps almost all of the
multiple-client issues.  h5pyd (https://github.com/HDFGroup/h5pyd) is a
client of h5serv which has an identical high-level API to h5py.

Tom

On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <nelso...@llnl.gov> wrote:

> It doesn?t sound like parallel HDF5 is what you are wanting to do here.
> Parallel HDF5 is for an application where all applications are writing 
> in a very coordinated manner. All processes need to write the same 
> metadata to the file in ?collective? calls to the library, i.e. each 
> application makes the same calls using the same arguments in the same 
> order when making calls that modify the file metadata (creating files, 
> datasets or groups, writing attributes, etc.
> <https://support.hdfgroup.org/HDF5/doc/RM/CollectiveCalls.html>).
>
> It sounds like you have separate applications that are executing in a 
> somewhat independent manner. This will not work with parallel HDF5.
>
>
>
> Using the serial library, I can think of at least one approach that 
> might work well for you. HDF5 1.10 introduced a Single-writer 
> multi-reader (SWMR) mode = to open a file. Using a SWMR file for each 
> process, each process would open one file as the writer in SWMR mode, 
> and open the files from all the other processes as read-only in SWMR mode.
>
>
>
> http://docs.h5py.org/en/latest/swmr.html
>
>
>
> Jarom
>
>
>
> *From:* Hdf-forum [mailto:hdf-forum-boun...@lists.hdfgroup.org] *On 
> Behalf Of *ISCaD GmbH
> *Sent:* Thursday, August 3, 2017 1:19 AM
> *To:* hdf-forum@lists.hdfgroup.org
> *Subject:* [Hdf-forum] hdf5 parallel h5py
>
>
>
> Dear all,
>
>
>
> I work on an web application which should store and receive the data 
> from an hdf5 file.
>
>
>
> Because several people work with this file and long running processes 
> I would like
>
> to use the mpi4py, h5py and HDF5.
>
>
>
> I work on debian linux stretch 64 Bit.
>
>
>
> What?s the way to parallel h5py.
>
>
>
> Thanks and regards
>
>
>
> Friedhelm Matten
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
> Twitter: https://twitter.com/hdf5
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachment
s/20170805/4d84f060/attachment-0001.html>

------------------------------

Message: 2
Date: Mon, 7 Aug 2017 09:14:28 -0500
From: Quincey Koziol <koz...@lbl.gov>
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] "File too large" error, seemingly related to
        MPI
Message-ID: <11a08003-02c4-494d-a746-d86f1c948...@lbl.gov>
Content-Type: text/plain; charset=utf-8

Hi Frederic,
        Could you give us some more details about your file and the call(s)
you are making to HDF5?   I can?t think of any reason that it would crash
when creating a file like this, but something interesting could be going on?
:-)

        Quincey


> On Aug 7, 2017, at 5:28 AM, Frederic Perez <fredericper...@gmail.com>
wrote:
> 
> Hi,
> 
> While writing significant amount of data in parallel, I obtain the 
> following error stack:
> 
> HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 66:
>  #000: H5D.c line 194 in H5Dcreate2(): unable to create dataset
>    major: Dataset
>    minor: Unable to initialize object
>  #001: H5Dint.c line 453 in H5D__create_named(): unable to create and 
> link to dataset
>    major: Dataset
>    minor: Unable to initialize object
>  #002: H5L.c line 1638 in H5L_link_object(): unable to create new link 
> to object
>    major: Links
>    minor: Unable to initialize object
>  #003: H5L.c line 1882 in H5L_create_real(): can't insert link
>    major: Symbol table
>    minor: Unable to insert object
>  #004: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal
failed
>    major: Symbol table
>    minor: Object not found
>  #005: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator
failed
>    major: Symbol table
>    minor: Callback failed
>  #006: H5L.c line 1685 in H5L_link_cb(): unable to create object
>    major: Object header
>    minor: Unable to initialize object
>  #007: H5O.c line 3016 in H5O_obj_create(): unable to open object
>    major: Object header
>    minor: Can't open object
>  #008: H5Doh.c line 293 in H5O__dset_create(): unable to create dataset
>    major: Dataset
>    minor: Unable to initialize object
>  #009: H5Dint.c line 1060 in H5D__create(): can't update the metadata
cache
>    major: Dataset
>    minor: Unable to initialize object
>  #010: H5Dint.c line 852 in H5D__update_oh_info(): unable to update 
> layout/pline/efl header message
>    major: Dataset
>    minor: Unable to initialize object
>  #011: H5Dlayout.c line 238 in H5D__layout_oh_create(): unable to 
> initialize storage
>    major: Dataset
>    minor: Unable to initialize object
>  #012: H5Dint.c line 1713 in H5D__alloc_storage(): unable to 
> initialize dataset with fill value
>    major: Dataset
>    minor: Unable to initialize object
>  #013: H5Dint.c line 1805 in H5D__init_storage(): unable to allocate 
> all chunks of dataset
>    major: Dataset
>    minor: Unable to initialize object
>  #014: H5Dchunk.c line 3575 in H5D__chunk_allocate(): unable to write 
> raw data to file
>    major: Low-level I/O
>    minor: Write failed
>  #015: H5Dchunk.c line 3745 in H5D__chunk_collective_fill(): unable to 
> write raw data to file
>    major: Low-level I/O
>    minor: Write failed
>  #016: H5Fio.c line 171 in H5F_block_write(): write through metadata 
> accumulator failed
>    major: Low-level I/O
>    minor: Write failed
>  #017: H5Faccum.c line 825 in H5F__accum_write(): file write failed
>    major: Low-level I/O
>    minor: Write failed
>  #018: H5FDint.c line 260 in H5FD_write(): driver write request failed
>    major: Virtual File Layer
>    minor: Write failed
>  #019: H5FDmpio.c line 1846 in H5FD_mpio_write(): MPI_File_write_at_all
failed
>    major: Internal error (too specific to document in detail)
>    minor: Some MPI function failed
>  #020: H5FDmpio.c line 1846 in H5FD_mpio_write(): Other I/O error , 
> error stack:
> ADIOI_NFS_WRITESTRIDED(672): Other I/O error File too large
>    major: Internal error (too specific to document in detail)
>    minor: MPI Error String
> 
> 
> It basically claims that I am creating a file too large. But I 
> verified that the filesystem is capable of handling such a size. In my 
> case, the file is around 4 TB when it crashes. Where could this 
> problem come from? I thought HDF5 does not have a problem with very 
> large files. Plus, I am dividing the file in several datasets, and the 
> write operations work perfectly until, at some point, it crashes with 
> the errors above.
> 
> Could it be an issue with HDF5? Or could it be an MPI limitation? I am 
> skeptic about the latter option: at the beginning, the program writes 
> several datasets inside the file succesfully (all the datasets being 
> the same size). If MPI was to blame, why wouldn't it crash at the 
> first write?
> 
> Thank you for your help.
> Fred
> 
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
> Twitter: https://twitter.com/hdf5




------------------------------

Message: 3
Date: Mon, 7 Aug 2017 09:15:41 -0500
From: Quincey Koziol <koz...@lbl.gov>
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] VCS URI
Message-ID: <adb59151-50fc-4927-8165-9d2b0c975...@lbl.gov>
Content-Type: text/plain; charset="us-ascii"

Hi David,
        Sure, the git repo is here:
https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse
<https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse>

                Quincey


> On Aug 7, 2017, at 5:27 AM, David Seifert <s...@gentoo.org> wrote:
> 
> Hi HDF5 team and users,
> is there any possibility for me to develop again the current VCS 
> sources for adding pkgconfig + Meson support? This makes development 
> and feature addition a lot easier, as there won't be any conflicts.
> 
> Regards
> David
> 
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
> Twitter: https://twitter.com/hdf5

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachment
s/20170807/bebdf029/attachment-0001.html>

------------------------------

Message: 4
Date: Mon, 7 Aug 2017 15:05:28 +0000
From: John Readey <jrea...@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>,
        "friedhelm.mat...@iscad-it.de" <friedhelm.mat...@iscad-it.de>
Subject: Re: [Hdf-forum] hdf5 parallel h5py
Message-ID: <82d9ce59-6bbe-4e47-9d05-18de03326...@hdfgroup.org>
Content-Type: text/plain; charset="utf-8"

Friedhelm,

   I?m not familiar with the specifics of your web app, but another
possibility is to just have the app call h5serv directly.

   Anika @ NASA Goddard wrote a nice blog article on this approach:
https://www.hdfgroup.org/2017/04/the-gfed-analysis-tool-an-hdf-server-implem
entation/.

John

From: Hdf-forum <hdf-forum-boun...@lists.hdfgroup.org> on behalf of Thomas
Caswell <tcasw...@gmail.com>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Date: Saturday, August 5, 2017 at 9:48 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>,
"friedhelm.mat...@iscad-it.de" <friedhelm.mat...@iscad-it.de>
Subject: Re: [Hdf-forum] hdf5 parallel h5py

I would also look at h5serv (https://github.com/HDFGroup/h5serv) which puts
a server in front of your hdf5 file.  It serves as a single process owns it
and serves as the serialization point which side-steps almost all of the
multiple-client issues.  h5pyd (https://github.com/HDFGroup/h5pyd) is a
client of h5serv which has an identical high-level API to h5py.

Tom

On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom
<nelso...@llnl.gov<mailto:nelso...@llnl.gov>> wrote:
It doesn?t sound like parallel HDF5 is what you are wanting to do here.
Parallel HDF5 is for an application where all applications are writing in a
very coordinated manner. All processes need to write the same metadata to
the file in ?collective? calls to the library, i.e. each application makes
the same calls using the same arguments in the same order when making calls
that modify the file metadata (creating files, datasets or groups, writing
attributes,
etc.<https://support.hdfgroup.org/HDF5/doc/RM/CollectiveCalls.html>).
It sounds like you have separate applications that are executing in a
somewhat independent manner. This will not work with parallel HDF5.

Using the serial library, I can think of at least one approach that might
work well for you. HDF5 1.10 introduced a Single-writer multi-reader (SWMR)
mode = to open a file. Using a SWMR file for each process, each process
would open one file as the writer in SWMR mode, and open the files from all
the other processes as read-only in SWMR mode.

http://docs.h5py.org/en/latest/swmr.html

Jarom

From: Hdf-forum
[mailto:hdf-forum-boun...@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.
hdfgroup.org>] On Behalf Of ISCaD GmbH
Sent: Thursday, August 3, 2017 1:19 AM
To: hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] hdf5 parallel h5py

Dear all,

I work on an web application which should store and receive the data from an
hdf5 file.

Because several people work with this file and long running processes I
would like to use the mpi4py, h5py and HDF5.

I work on debian linux stretch 64 Bit.

What?s the way to parallel h5py.

Thanks and regards

Friedhelm Matten

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachment
s/20170807/d8afdcac/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


------------------------------

End of Hdf-forum Digest, Vol 98, Issue 6
****************************************


_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Reply via email to