This concerns IBM PE-MPI on iDataPlex, which is likely based upon the cluster implementation of PAMI, which is a completely different code base from the PAMI Blue Gene implementation. If you can reproduce it on Blue Gene/Q, I will care.
As an IBM customer, NCAR is endowed with the ability to file bug reports directly with IBM related to the products they possess. There is a link to their support system on http://www2.cisl.ucar.edu/resources/yellowstone, which is the appropriate channel for users of Yellowstone that have issues with the system software installed there. Jeff ----- Original Message ----- From: "Jed Brown" <[email protected]> To: "Fande Kong" <[email protected]>, "petsc-users" <[email protected]> Cc: "Jeff Hammond" <[email protected]> Sent: Wednesday, June 26, 2013 9:21:48 AM Subject: Re: [petsc-users] How to understand these error messages Fande Kong <[email protected]> writes: > Hi Barry, > > If I use the intel mpi, my code can correctly run and can produce some > correct results. Yes, you are right. The IBM MPI has some bugs. Fande, please report this issue to the IBM. Jeff, Fande has a reproducible case where when running on 10k cores and problem sizes over 100M, this MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status); returns [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892! It runs correctly for smaller problem sizes, smaller core counts, or for all sizes when using Intel MPI. This is on Yellowstone (iDataPlex, 4500 dx360 nodes). Do you know someone at IBM that should be notified? -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute [email protected] / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides
