On Fri, Aug 21, 2020 at 3:32 PM Barry Smith <bsm...@petsc.dev> wrote:
> > Yes, absolutely a test suite will not solve all problems. In the PETSc > model, which is not uncommon, each bug/problem found is suppose to result > in another test to detect that problem, thus the test suite can find > repeats of the problem without again all the hard work from scratch. > > So this OpenMPI suite, if it gets off the ground, will be valuable ONLY > if they accept community additions efficiently and happily. For example > would the test suite detect the problem reported by the PETSc user? It > should be trivial to have the user run the suite on their system (which is > why it needs be very easy to run) and determine. If it does not detect the > problem then working with the appropriate "test suite" community we could > submit a MR to the test suite that looks for the problem and finds it. Now > the test suite is better and we have one less hassle that comes up multiple > times for us. In addition the OpenMPI, MPICH developers etc should do the > same thing, each time they fix a bug that was not detected by testing they > should donate to the universal test suite the code to reproduce the bug. > > The question is would our effort in helping the MPI test suite community > be more than our "wasted" effort dealing with buggy MPIs? > > Barry > > It is a bit curious that after 25 years no friendly extensible universal > MPI test suite community has emerged. Perhaps it is because each MPI > implementation has its own test processes and suites and cannot form the > wider community to have a single friendly extensible universal MPI test > suite. Looking back one could say this was a mistake of the MPI forum, they > should have started that in motion in 1995, would have saved a lot of > duplication of effort and would be very very good now. > I think they do not do it because people do not hold implementors accountable, only the packages using MPI. Matt > On Aug 21, 2020, at 2:17 PM, Junchao Zhang <junchao.zh...@gmail.com> > wrote: > > Barry, > I mentioned a test suite from MPICH at > https://lists.mcs.anl.gov/pipermail/petsc-users/2020-July/041738.html. > Since it is not easy to use, I did not put it on PETSc FAQ. > I also asked in the OpenMPI mailing list. An OpenMPI developer said he > could make their tests public, and is in the process of checking with all > authors to have a license :). If it is done, it will be at > https://github.com/open-mpi/ompi-tests-public > > A test suite will be helpful but I doubt it will solve the problem. > User's particular case (number of ranks, message size, > communication pattern etc) might not be covered by a test suite. > --Junchao Zhang > > > On Fri, Aug 21, 2020 at 12:33 PM Barry Smith <bsm...@petsc.dev> wrote: > >> >> There really needs to be a usable extensive MPI test suite that can >> find these performance issues, we spend time helping users with these >> problems when it is really the MPI communities job. >> >> >> >> On Aug 21, 2020, at 11:55 AM, Manav Bhatia <bhatiama...@gmail.com> wrote: >> >> I built petsc with mpich-3.3.2 on my MacBook Pro with Apple clang 11.0.3 >> and the test is finishing at my end. >> >> So, it appears that there is some issue with openmpi-4.0.1 on this >> machine. >> >> I will now build all my dependency toolchain with mpich and hopefully >> things will work for my application code. >> >> Thank you again for your help. >> >> Regards, >> Manav >> >> >> On Aug 20, 2020, at 10:45 PM, Junchao Zhang <junchao.zh...@gmail.com> >> wrote: >> >> Manav, >> I downloaded your petsc_mat.tgz but could not reproduce the problem, on >> both Linux and Mac. I used the petsc commit id df0e4300 you mentioned. >> On Linux, I have openmpi-4.0.2 + gcc-8.3.0, and petsc is configured >> --with-debugging --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpifort >> --COPTFLAGS="-g -O0" --FOPTFLAGS="-g -O0" --CXXOPTFLAGS="-g -O0" >> --PETSC_ARCH=linux-host-dbg >> On Mac, I have mpich-3.3.1 + clang-11.0.0-apple, and petsc is >> configured --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx >> --with-fc=mpifort --with-ctable=0 COPTFLAGS="-O0 -g" CXXOPTFLAGS="-O0 -g" >> PETSC_ARCH=mac-clang-dbg >> >> mpirun -n 8 ./test >> rank: 1 : stdout.processor.1 >> rank: 4 : stdout.processor.4 >> rank: 0 : stdout.processor.0 >> rank: 5 : stdout.processor.5 >> rank: 6 : stdout.processor.6 >> rank: 7 : stdout.processor.7 >> rank: 3 : stdout.processor.3 >> rank: 2 : stdout.processor.2 >> rank: 1 : Beginning reading nnz... >> rank: 4 : Beginning reading nnz... >> rank: 0 : Beginning reading nnz... >> rank: 5 : Beginning reading nnz... >> rank: 7 : Beginning reading nnz... >> rank: 2 : Beginning reading nnz... >> rank: 3 : Beginning reading nnz... >> rank: 6 : Beginning reading nnz... >> rank: 5 : Finished reading nnz >> rank: 5 : Beginning mat preallocation... >> rank: 3 : Finished reading nnz >> rank: 3 : Beginning mat preallocation... >> rank: 4 : Finished reading nnz >> rank: 4 : Beginning mat preallocation... >> rank: 7 : Finished reading nnz >> rank: 7 : Beginning mat preallocation... >> rank: 1 : Finished reading nnz >> rank: 1 : Beginning mat preallocation... >> rank: 0 : Finished reading nnz >> rank: 0 : Beginning mat preallocation... >> rank: 2 : Finished reading nnz >> rank: 2 : Beginning mat preallocation... >> rank: 6 : Finished reading nnz >> rank: 6 : Beginning mat preallocation... >> rank: 5 : Finished preallocation >> rank: 5 : Beginning reading and setting matrix values... >> rank: 1 : Finished preallocation >> rank: 1 : Beginning reading and setting matrix values... >> rank: 7 : Finished preallocation >> rank: 7 : Beginning reading and setting matrix values... >> rank: 2 : Finished preallocation >> rank: 2 : Beginning reading and setting matrix values... >> rank: 4 : Finished preallocation >> rank: 4 : Beginning reading and setting matrix values... >> rank: 0 : Finished preallocation >> rank: 0 : Beginning reading and setting matrix values... >> rank: 3 : Finished preallocation >> rank: 3 : Beginning reading and setting matrix values... >> rank: 6 : Finished preallocation >> rank: 6 : Beginning reading and setting matrix values... >> rank: 1 : Finished reading and setting matrix values >> rank: 1 : Beginning mat assembly... >> rank: 5 : Finished reading and setting matrix values >> rank: 5 : Beginning mat assembly... >> rank: 4 : Finished reading and setting matrix values >> rank: 4 : Beginning mat assembly... >> rank: 2 : Finished reading and setting matrix values >> rank: 2 : Beginning mat assembly... >> rank: 3 : Finished reading and setting matrix values >> rank: 3 : Beginning mat assembly... >> rank: 7 : Finished reading and setting matrix values >> rank: 7 : Beginning mat assembly... >> rank: 6 : Finished reading and setting matrix values >> rank: 6 : Beginning mat assembly... >> rank: 0 : Finished reading and setting matrix values >> rank: 0 : Beginning mat assembly... >> rank: 1 : Finished mat assembly >> rank: 3 : Finished mat assembly >> rank: 7 : Finished mat assembly >> rank: 0 : Finished mat assembly >> rank: 5 : Finished mat assembly >> rank: 2 : Finished mat assembly >> rank: 4 : Finished mat assembly >> rank: 6 : Finished mat assembly >> >> --Junchao Zhang >> >> >> On Thu, Aug 20, 2020 at 5:29 PM Junchao Zhang <junchao.zh...@gmail.com> >> wrote: >> >>> I will have a look and report back to you. Thanks. >>> --Junchao Zhang >>> >>> >>> On Thu, Aug 20, 2020 at 5:23 PM Manav Bhatia <bhatiama...@gmail.com> >>> wrote: >>> >>>> I have created a standalone test that demonstrates the problem at my >>>> end. I have stored the indices, etc. from my problem in a text file >>>> for each rank, which I use to initialize the matrix. >>>> Please note that the test is specifically for 8 ranks. >>>> >>>> The .tgz file is on my google drive: >>>> https://drive.google.com/file/d/1R-WjS36av3maXX3pUyiR3ndGAxteTVj-/view?usp=sharing >>>> >>>> >>>> This contains a README file with instructions on running. Please note >>>> that the work directory needs the index files. >>>> >>>> Please let me know if I can provide any further information. >>>> >>>> Thank you all for your help. >>>> >>>> Regards, >>>> Manav >>>> >>>> On Aug 20, 2020, at 12:54 PM, Jed Brown <j...@jedbrown.org> wrote: >>>> >>>> Matthew Knepley <knep...@gmail.com> writes: >>>> >>>> On Thu, Aug 20, 2020 at 11:09 AM Manav Bhatia <bhatiama...@gmail.com> >>>> wrote: >>>> >>>> >>>> >>>> On Aug 20, 2020, at 8:31 AM, Stefano Zampini <stefano.zamp...@gmail.com >>>> > >>>> wrote: >>>> >>>> Can you add a MPI_Barrier before >>>> >>>> ierr = MatAssemblyBegin(aij->A,mode);CHKERRQ(ierr); >>>> >>>> >>>> With a MPI_Barrier before this function call: >>>> — three of the processes have already hit this barrier, >>>> — the other 5 are inside MatStashScatterGetMesg_Private -> >>>> MatStashScatterGetMesg_BTS -> MPI_Waitsome(2 processes)/MPI_Waitall(3 >>>> processes) >>>> >>>> >>>> This is not itself evidence of inconsistent state. You can use >>>> >>>> -build_twosided allreduce >>>> >>>> to avoid the nonblocking sparse algorithm. >>>> >>>> >>>> Okay, you should run this with -matstash_legacy just to make sure it is >>>> not >>>> a bug in your MPI implementation. But it looks like >>>> there is inconsistency in the parallel state. This can happen because we >>>> have a bug, or it could be that you called a collective >>>> operation on a subset of the processes. Is there any way you could cut >>>> down >>>> the example (say put all 1s in the matrix, etc) so >>>> that you could give it to us to run? >>>> >>>> >>>> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
