Mark,
Ok, in this situation VecScatter cannot detect that it is an all to all so
will generate a message from each process to each other process. Given my past
experience with Cray MPI (why do they even have their own MPI when Intel
provides one; in fact why does Cray even exist when they just take other
people's products and put their name on them) I am not totally surprised if the
Cray MPI chocks on this flood of messages.
1) Test with Intel MPI, perhaps they handle this case in a scalable way
2) If Intel MPI also produces poor performance then (interesting, how come
on other systems in the past this wasn't a bottleneck for the code?) the
easiest solution is to separate the operation into two parts. Use a
VecScatterCreateToAll() to get all the data to all the processes and then use
another (purely sequential) VecScatter to get the data from this intermediate
buffer into the final vector that has the "extra" locations for the boundary
conditions in the final destination vector.
BTW: You know this already, but any implementation that requires storing the
"entire" vector on each process is, by definition, not scalable and hence
should not even be considered for funding by ECP or SciDAC.
Barry
> On Mar 8, 2017, at 8:43 PM, Mark Adams <[email protected]> wrote:
>
>>
>> Is the scatter created with VecScatterCreateToAll()? If so, internally
>> the VecScatterBegin/End will use VecScatterBegin_MPI_ToAll() which then uses
>> a MPI_Allgatherv() to do the communication. You can check in the debugger
>> for this (on 2 processes) by just putting a break point in
>> VecScatterBegin_MPI_ToAll() to confirm if it is called.
>
> Alas, not I did not use VecScatterCreateToAll and
> VecScatterCreateToAll will take some code changes.
>
> There are boundary conditions in the destination vector, and so we
> scatter into a larger vector the the global size of the PETSc vector,
> using a general IS. With code that looks like this:
>
> call
> ISCreateGeneral(PETSC_COMM_SELF,nreal,petsc_xgc,PETSC_COPY_VALUES,is,ierr)
> call
> VecScatterCreate(this%xVec,PETSC_NULL_OBJECT,vec,is,this%from_petsc,ierr)
> ! reverse scatter object
>
> If we want to make this change then I could help a developer or you
> can get me set up with a (small) test problem and a branch and I can
> do it at NERSC.
>
> Thanks,
