On Fri, Apr 11, 2014 at 12:35 PM, Matthew Knepley <[email protected]> wrote:
> On Fri, Apr 11, 2014 at 12:11 PM, Andrés Alessandro León Baldelli < > [email protected]> wrote: > >> Hi, >> I am trying to use the new Plex to HDF5 interface. From what I >> understand, it only works in sequential mode. >> I would like to write a sequential DM, created e.g. from an exodusII mesh >> via DMPlexCreateExodusFromFile. >> What puzzles me is that I only can do it if I run on a single processor, >> since it crashes as soon its communicator contains more than one proc. >> >> My understanding is that, if numproc>1, the crash is due to the call to >> ISView in src/dm/impls/plex/plex.c:785 which fails because its two >> arguments do not share the same communicator. >> >> Could you help me understand a little more what is going on, what I am >> doing wrong and what could help solving this issue? >> > > Use 'next', not 'master' > I just pushed an update to next that should allow parallel reading and writing to HDF5. I test using -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append and then DMLoad()/VecLoad(). It of course needs a lot more testing, but it works on my small case right now. Matt > Matt > > >> I attach the error message, a minimal code to reproduce the crash and a >> test mesh. >> >> Best, >> >> ---------- begin output---------- >> geothermal:dmcomplex kumiori$ mpirun -n 2 ./testHDF5Plex -i >> data/TwoTri.gen >> cubit(/Users/blaise/Development/DMComplex/data/junkTri.gen): 08/31/2013: >> 19:01:0 in 2 dimensions: >> 0-cells: 6 0 >> 2-cells: 4 0 >> Labels: >> depth: 2 strata of sizes (6, 4) >> Cell Sets: 1 strata of sizes (4) >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Arguments must have same communicators >> [0]PETSC ERROR: Different communicators in the two objects: Argument # 1 >> and 2 flag 3 >> [0]PETSC ERROR: See http:// >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.4.4-3957-g24a0f54 GIT >> Date: 2014-04-02 15:35:24 -0500 >> [0]PETSC ERROR: ./testHDF5Plex on a Darwin-llvm50-g named >> geothermal.lsu.edu by kumiori Fri Apr 11 12:06:44 2014 >> [0]PETSC ERROR: Configure options --CFLAGS=-Wno-unused --download-chaco=1 >> --download-exodusii=1 --download-hdf5=1 --download-metis=1 >> --download-mpich=1 --download-netcdf=1 --download-parmetis=1 >> --download-sowing=1 --download-triangle=1 --download-yaml=1 >> --with-cmake=cmake --with-debugging=1 --with-pic --with-shared-libraries=1 >> --with-x11=1 PETSC_ARCH=Darwin-llvm50-g >> [0]PETSC ERROR: #1 ISView() line 756 in >> /opt/HPC/petsc/src/vec/is/is/interface/index.c >> [0]PETSC ERROR: #2 DMPlexView_HDF5() line 785 in >> /opt/HPC/petsc/src/dm/impls/plex/plex.c >> [0]PETSC ERROR: #3 DMView_Plex() line 812 in >> /opt/HPC/petsc/src/dm/impls/plex/plex.c >> [0]PETSC ERROR: #4 main() line 61 in >> /Users/kumiori/Code/dmcomplex/testHDF5Plex.c >> [0]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to [email protected] >> application called MPI_Abort(MPI_COMM_WORLD, 80) - process 0 >> [cli_0]: aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 80) - process 0 >> HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) MPI-process 1: >> #000: H5F.c line 2058 in H5Fclose(): decrementing file ID failed >> major: Object atom >> minor: Unable to close file >> #001: H5I.c line 1479 in H5I_dec_app_ref(): can't decrement ID ref count >> major: Object atom >> minor: Unable to decrement reference count >> #002: H5F.c line 1835 in H5F_close(): can't close file >> major: File accessability >> minor: Unable to close file >> #003: H5F.c line 1997 in H5F_try_close(): problems closing file >> major: File accessability >> minor: Unable to close file >> #004: H5F.c line 1142 in H5F_dest(): low level truncate failed >> major: File accessability >> minor: Write failed >> #005: H5FD.c line 1897 in H5FD_truncate(): driver truncate request >> failed >> major: Virtual File Layer >> minor: Can't update object >> #006: H5FDmpio.c line 1984 in H5FD_mpio_truncate(): MPI_File_set_size >> failed >> major: Internal error (too specific to document in detail) >> minor: Some MPI function failed >> #007: H5FDmpio.c line 1984 in H5FD_mpio_truncate(): Invalid argument, >> error stack: >> MPI_FILE_SET_SIZE(74): Inconsistent arguments to collective routine >> major: Internal error (too specific to document in detail) >> minor: MPI Error String >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: >> or try >> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >> corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] PetscError line 363 /opt/HPC/petsc/src/sys/error/err.c >> [0]PETSC ERROR: [0] ISView line 750 >> /opt/HPC/petsc/src/vec/is/is/interface/index.c >> [0]PETSC ERROR: [0] DMPlexView_HDF5 line 656 >> /opt/HPC/petsc/src/dm/impls/plex/plex.c >> [0]PETSC ERROR: [0] DMView_Plex line 803 >> /opt/HPC/petsc/src/dm/impls/plex/plex.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Signal received >> [0]PETSC ERROR: See http:// >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.4.4-3957-g24a0f54 GIT >> Date: 2014-04-02 15:35:24 -0500 >> [0]PETSC ERROR: ./testHDF5Plex on a Darwin-llvm50-g named >> geothermal.lsu.edu by kumiori Fri Apr 11 12:06:44 2014 >> [0]PETSC ERROR: Configure options --CFLAGS=-Wno-unused --download-chaco=1 >> --download-exodusii=1 --download-hdf5=1 --download-metis=1 >> --download-mpich=1 --download-netcdf=1 --download-parmetis=1 >> --download-sowing=1 --download-triangle=1 --download-yaml=1 >> --with-cmake=cmake --with-debugging=1 --with-pic --with-shared-libraries=1 >> --with-x11=1 PETSC_ARCH=Darwin-llvm50-g >> [0]PETSC ERROR: #5 User provided function() line 0 in unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> [cli_0]: aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = EXIT CODE: 59 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> >> =================================================================================== >> geothermal:dmcomplex kumiori$ >> >> ---------- end output---------- >> >> ----------begin C source code testHDF5plex.c -------------- >> /* usage: testHDF5plex -i meshfilename >> */ >> #include <petscdmplex.h> >> #include <petscsf.h> >> #include <exodusII.h> >> #include <petsc-private/dmimpl.h> /*I "petscdm.h" I*/ >> #include <petscsf.h> >> #include <petscviewerhdf5.h> >> #include <petscdmplex.h> >> >> #undef __FUNCT__ >> #define __FUNCT__ "main" >> int main(int argc,char **argv) >> { >> PetscErrorCode ierr; >> DM dm; >> char ifilename[PETSC_MAX_PATH_LEN]; >> PetscBool flg; >> int CPU_word_size = 0,IO_word_size = 0,exoid; >> float version; >> PetscMPIInt numproc,rank; >> PetscViewer hdf5Viewer; >> PetscInt dim; >> PetscInt numFields = 2; >> PetscInt numComp[2] = {1,1}; >> PetscInt numDof[6] = {1, 0, 0, >> 0, 0, 1}; /*{Vertex, Edge, Cell} >> */ >> PetscInt bcFields[1] = {0}, numBC=0; >> IS bcPoints[1] = {NULL}; >> PetscSection section; >> MPI_Comm comm; >> >> ierr = PetscInitialize(&argc, &argv, (char*)0, >> help);CHKERRQ(ierr); >> comm = PETSC_COMM_WORLD; >> >> ierr = PetscOptionsBegin(PETSC_COMM_WORLD,"","testHDF5Plex Test >> Options","none");CHKERRQ(ierr); >> ierr = PetscOptionsEnd(); >> ierr = PetscOptionsGetString(PETSC_NULL,"-i",ifilename,sizeof >> ifilename,&flg);CHKERRQ(ierr); >> >> ierr = MPI_Comm_size(PETSC_COMM_WORLD,&numproc); >> ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); >> >> ierr = DMPlexCreateExodusFromFile(PETSC_COMM_WORLD, ifilename, >> PETSC_FALSE, &dm);CHKERRQ(ierr); >> ierr = DMPlexGetDimension(dm, &dim);CHKERRQ(ierr); >> ierr = DMPlexCreateSection(dm, dim, numFields, numComp, numDof, >> numBC, bcFields, bcPoints, NULL, §ion);CHKERRQ(ierr); >> ierr = DMSetDefaultSection(dm, section);CHKERRQ(ierr); >> ierr = PetscViewerHDF5Open(PetscObjectComm((PetscObject) dm), >> "out/seqDM.h5", FILE_MODE_WRITE, &hdf5Viewer); CHKERRQ(ierr); >> // Crashes if numproc > 1, >> ierr = DMView_Plex(dm, hdf5Viewer); CHKERRQ(ierr); >> PetscViewerDestroy(&hdf5Viewer); >> >> ierr = DMDestroy(&dm);CHKERRQ(ierr); >> ierr = PetscFinalize(); >> return 0; >> } >> >> ----------end C source code testHDF5plex.c -------------- >> >> ---------- test mesh >> >> Andrés A. León Baldelli >> -- >> LSU Center for Computation & Technology >> 2099 Louisiana Digital Media Center >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
