On Thu, Sep 14, 2017 at 11:26 AM, Matthew Knepley <knep...@gmail.com> wrote:
> On Thu, Sep 14, 2017 at 1:07 PM, Kong, Fande <fande.k...@inl.gov> wrote: > >> >> >> On Thu, Sep 14, 2017 at 10:35 AM, Barry Smith <bsm...@mcs.anl.gov> wrote: >> >>> >>> > On Sep 14, 2017, at 11:10 AM, Kong, Fande <fande.k...@inl.gov> wrote: >>> > >>> > >>> > >>> > On Thu, Sep 14, 2017 at 9:47 AM, Matthew Knepley <knep...@gmail.com> >>> wrote: >>> > On Thu, Sep 14, 2017 at 11:43 AM, Adriano Côrtes < >>> adrimacor...@gmail.com> wrote: >>> > Dear Matthew, >>> > >>> > Thank you for your return. It worked, but this prompts another >>> question. So why PetscViewer does not write both files (.h5 and .xmf) >>> directly, instead of having to post-proc the .h5 file (in serial)? >>> > >>> > 1) Maintenance: Changing the Python is much easier than changing the C >>> you would add to generate it >>> > >>> > 2) Performance: On big parallel system, writing files is expensive so >>> I wanted to minimize what I had to do. >>> > >>> > 3) Robustness: Moving 1 file around is much easier than remembering 2. >>> I just always regenerate the xdmf when needed. >>> > >>> > And what about big 3D simulations? PETSc always serialize the output >>> of the distributed dmplex? Is there a way to output one .h5 per mesh >>> partition? >>> > >>> > Given the way I/O is structured on big machines, we believe the >>> multiple file route is a huge mistake. Also, all our measurements >>> > say that sending some data on the network is not noticeable given the >>> disk access costs. >>> > >>> > I have slightly different things here. We tried the serial output, it >>> looks really slow for large-scale problems, and the first processor often >>> runs out of memory because of gathering all data from other processor cores. >>> >>> Where in PETSc is this? What type of viewer? Is there an example that >>> reproduces the problem? Even when we do not use MPI IO in PETSc we attempt >>> to not "put the entire object on the first process" so memory should not be >>> an issue. For example VecVew() should memory scale both with or without MPI >>> IO >>> >> >> We manually gather all data to the first processor core, and write it as >> a single vtk file. >> > > Of course I am not doing that. I reduce everything to an ISView or a > VecView call. That way it uses MPI I/O if its turned on. > I meant Fande manually gathers all data to the first processor core in his in-house code. > > Matt > > >> >>> >>> > The parallel IO runs smoothly and much faster than I excepted. We have >>> done experiments with ten thousands of cores for a problem with 1 billion >>> of unknowns. >>> >>> Is this your own canned IO or something in PETSc? >>> >> >> We implement the writer based on the ISView and VecView with HDF5 viewer >> in PETSc to output all data as a single HDF. ISView and VecView do the >> magic job for me. >> >> >> >>> >>> > I did not see any concern so far. >>> >>> Ten thousand files is possibly manageable but I question 2 million. >>> >> >> Just one single HDF5 file. >> >> Fande, >> >> >>> >>> > >>> > >>> > Fande, >>> > >>> > >>> > Thanks, >>> > >>> > Matt >>> > >>> > Best regards, >>> > Adriano. >>> > >>> > >>> > 2017-09-14 12:00 GMT-03:00 Matthew Knepley <knep...@gmail.com>: >>> > On Thu, Sep 14, 2017 at 10:30 AM, Adriano Côrtes < >>> adrimacor...@gmail.com> wrote: >>> > Dear all, >>> > >>> > I am running the SNES ex12 and I'm passing the options -dm_view >>> hdf5:sol.h5 -vec_view hdf5:sol.h5::append to generate an output file. The >>> .h5 file is generated, but I'm not being able to load it in Paraview >>> (5.4.0-64bits). Paraview recognizes the file and offers severel options to >>> read it, here is the complete list >>> > >>> > Chombo Files >>> > GTC Files >>> > M3DC1 Files >>> > Multilevel 3D Plasma Files >>> > PFLOTRAN Files >>> > Pixie Files >>> > Tetrad Files >>> > UNIC Files >>> > VizSchema Files >>> > >>> > The problem is none of the options above work :( >>> > I'm using the configure option '-download-hdf5' and it installs hdf5 >>> version 1.8.18 >>> > Any hint of how to fix it and have the visualization working? >>> > >>> > Yes, Paraview does not directly read HDF5. It needs you to tell it >>> what the data in the HDF5 file means. You do >>> > this by creating a *.xdmf file, which is XML. We provide a tool >>> > >>> > $PETSC_DIR/bin/petsc_gen_xdmf.py <HDF5 file> >>> > >>> > which should automatically produce this file for you. Let us know if >>> it does not work. >>> > >>> > Thanks, >>> > >>> > Matt >>> > >>> > >>> > Best regards, >>> > Adriano. >>> > >>> > -- >>> > Adriano Côrtes >>> > ================================================= >>> > Campus Duque de Caxias and >>> > High-performance Computing Center (NACAD/COPPE) >>> > Federal University of Rio de Janeiro (UFRJ) >>> > >>> > >>> > >>> > -- >>> > What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> > -- Norbert Wiener >>> > >>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.caam >>> .rice.edu_-7Emk51_&d=DwIFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ >>> _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY& >>> m=YTLjMkjfS0tYLZ3RxmJFoe8BT56h48axFCzaadZwBXA&s=iLsaHQugaY4g >>> j4DKE9gq8XdBt7q3ejdpDRfJ8RFerE0&e= >>> > >>> > >>> > >>> > -- >>> > Adriano Côrtes >>> > ================================================= >>> > Campus Duque de Caxias and >>> > High-performance Computing Center (NACAD/COPPE) >>> > Federal University of Rio de Janeiro (UFRJ) >>> > >>> > >>> > >>> > -- >>> > What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> > -- Norbert Wiener >>> > >>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.caam >>> .rice.edu_-7Emk51_&d=DwIFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ >>> _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY& >>> m=YTLjMkjfS0tYLZ3RxmJFoe8BT56h48axFCzaadZwBXA&s=iLsaHQugaY4g >>> j4DKE9gq8XdBt7q3ejdpDRfJ8RFerE0&e= >>> > >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.caam.rice.edu_-7Emk51_&d=DwMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=2o_-FxtYVq5ZmM0k6MovstGXLWyZ_015zsBHbW6JgaE&s=s6ewDEwkUoF8dxaHMvNEGWAlCxbhXyarJBoLEKcSwaI&e=> >