On Wed 7. Jul 2021 at 20:41, Thibault Bridel-Bertomeu < thibault.bridelberto...@gmail.com> wrote:
> Dear all, > > I have been having issues with large Vec (based on DMPLex) and massive MPI > I/O ... it looks like the data that is written by the Petsc Binary Viewer > is gibberish for large meshes split on a high number of processes. For > instance, I am using a mesh that has around 50 million cells, split on 1024 > processors. > The computation seems to run fine, the timestep computed from the data > makes sense so I think internally everything is fine. But when I look at > the solution (one example attached) it's noise - at this point it should > show a bow shock developing on the left near the step. > The piece of code I use is below for the output : > > call DMGetOutputSequenceNumber(dm, save_seqnum, > save_seqval, ierr); CHKERRA(ierr) > call DMSetOutputSequenceNumber(dm, -1, 0.d0, ierr); > CHKERRA(ierr) > write(filename,'(A,I8.8,A)') "restart_", stepnum, ".bin" > call PetscViewerCreate(PETSC_COMM_WORLD, binViewer, ierr); > CHKERRA(ierr) > call PetscViewerSetType(binViewer, PETSCVIEWERBINARY, > ierr); CHKERRA(ierr) > call PetscViewerFileSetMode(binViewer, FILE_MODE_WRITE, > ierr); CHKERRA(ierr); > call PetscViewerBinarySetUseMPIIO(binViewer, PETSC_TRUE, > ierr); CHKERRA(ierr); > > Do you get the correct output if you don’t call the function above (or equivalently use PETSC_FALSE) call PetscViewerFileSetName(binViewer, trim(filename), ierr); CHKERRA(ierr) > call VecView(X, binViewer, ierr); CHKERRA(ierr) > call PetscViewerDestroy(binViewer, ierr); CHKERRA(ierr) > call DMSetOutputSequenceNumber(dm, save_seqnum, > save_seqval, ierr); CHKERRA(ierr) > > I do not think there is anything wrong with it but of course I would be > happy to hear your feedback. > Nonetheless my question was : how far have you tested the binary mpi i/o > of a Vec ? Does it make some sense that for a 50 million cell mesh split on > 1024 processes, it could somehow fail ? > Or is it my python drawing method that is completely incapable of handling > this dataset ? (paraview displays the same thing though so I'm not sure ...) > Are you using the python provided tools within petsc to load the Vec from file? Thanks, Dave > Thank you very much for your advice and help !!! > > Thibault >