Stefano,
Thanks this is very helpful.
---------------------
Why not? here is my naive implementation with AlltoAll, which perform better in
my case
PetscErrorCode PetscGatherMessageLengths(MPI_Comm comm,PetscMPIInt
nsends,PetscMPIInt nrecvs,const PetscMPIInt ilengths[],PetscMPIInt
**onodes,PetscMPIInt **olengths)
{
PetscErrorCode ierr;
PetscMPIInt size,i,j;
PetscMPIInt *all_lengths;
PetscFunctionBegin;
ierr = MPI_Comm_size(comm,&size);CHKERRQ(ierr);
ierr = PetscMalloc(size*sizeof(PetscMPIInt),&all_lengths);CHKERRQ(ierr);
ierr =
MPI_Alltoall((void*)ilengths,1,MPI_INT,all_lengths,1,MPI_INT,comm);CHKERRQ(ierr);
ierr = PetscMalloc(nrecvs*sizeof(PetscMPIInt),olengths);CHKERRQ(ierr);
ierr = PetscMalloc(nrecvs*sizeof(PetscMPIInt),onodes);CHKERRQ(ierr);
for (i=0,j=0; i<size; i++) {
if (all_lengths[i]) {
(*olengths)[j] = all_lengths[i];
(*onodes)[j] = i;
j++;
}
}
if (j != nrecvs) SETERRQ2(comm,PETSC_ERR_PLIB,"Unexpected number of senders
%d != %d",j,nrecvs);
ierr = PetscFree(all_lengths);CHKERRQ(ierr);
PetscFunctionReturn(0);
}
-----------------------
However I think this is only half the answer. If I look at
VecScatterCreate_PtoS() for example it has
ierr = PetscGatherNumberOfMessages(comm,NULL,nprocs,&nrecvs);CHKERRQ(ierr);
ierr =
PetscGatherMessageLengths(comm,nsends,nrecvs,nprocs,&onodes1,&olengths1);CHKERRQ(ierr);
ierr = PetscSortMPIIntWithArray(nrecvs,onodes1,olengths1);CHKERRQ(ierr);
recvtotal = 0; for (i=0; i<nrecvs; i++) recvtotal += olengths1[i];
/* post receives: */
ierr =
PetscMalloc3(recvtotal,&rvalues,nrecvs,&source,nrecvs,&recv_waits);CHKERRQ(ierr);
count = 0;
for (i=0; i<nrecvs; i++) {
ierr =
MPI_Irecv((rvalues+count),olengths1[i],MPIU_INT,onodes1[i],tag,comm,recv_waits+i);CHKERRQ(ierr);
count += olengths1[i];
}
/* do sends:
1) starts[i] gives the starting index in svalues for stuff going to
the ith processor
*/
nxr = 0;
for (i=0; i<nx; i++) {
if (owner[i] != rank) nxr++;
}
ierr =
PetscMalloc3(nxr,&svalues,nsends,&send_waits,size+1,&starts);CHKERRQ(ierr);
starts[0] = 0;
for (i=1; i<size; i++) starts[i] = starts[i-1] + nprocs[i-1];
for (i=0; i<nx; i++) {
if (owner[i] != rank) svalues[starts[owner[i]]++] = bs*inidx[i];
}
starts[0] = 0;
for (i=1; i<size+1; i++) starts[i] = starts[i-1] + nprocs[i-1];
count = 0;
for (i=0; i<size; i++) {
if (nprocs[i]) {
ierr =
MPI_Isend(svalues+starts[i],nprocs[i],MPIU_INT,i,tag,comm,send_waits+count++);CHKERRQ(ierr);
}
}
So I need to (1) use your alltoall PetscGatherMessageLengths() but also (2)
replace the sends and receives above with alltoallv();
Is that correct? Did you also fix (2) or did fixing (1) help so much you didn't
need to fix (2)?
Don't go to sleep yet, I may have more questions :-)
Barry
> On Mar 17, 2017, at 2:32 PM, Stefano Zampini <[email protected]>
> wrote:
>
> Pierre,
>
> I remember I had a similar problem some years ago when working with matrices
> with "process-dense" rows (i.e., when the off-diagonal part is shared by many
> processes). I fixed the issue by changing the implementation of
> PetscGatherMessageLenghts, from rendez-vous to all-to-all.
>
> Barry, if you had access to petsc-maint, the title of the thread is "Problem
> with PetscGatherMessageLengths".
>
> Hope this helps,
> Stefano
>
>
> 2017-03-17 22:21 GMT+03:00 Barry Smith <[email protected]>:
>
> > On Mar 17, 2017, at 4:04 AM, Pierre Jolivet <[email protected]>
> > wrote:
> >
> > On Thu, 16 Mar 2017 15:37:17 -0500, Barry Smith wrote:
> >>> On Mar 16, 2017, at 10:57 AM, Pierre Jolivet <[email protected]>
> >>> wrote:
> >>>
> >>> Thanks Barry.
> >>> I actually tried the application myself with my optimized build + your
> >>> option. I'm attaching two logs for a strong scaling analysis, if someone
> >>> could spend a minute or two looking at the numbers I'd be really grateful:
> >>> 1) MatAssembly still takes a rather long time IMHO. This is actually the
> >>> bottleneck of my application. Especially on 1600 cores, the problem here
> >>> is that I don't know if the huge time (almost a 5x slow-down w.r.t. the
> >>> run on 320 cores) is due to MatMPIAIJSetPreallocationCSR (which I assumed
> >>> beforehand was a no-op, but which is clearly not the case looking at the
> >>> run on 320 cores) or the the option -pc_bjacobi_blocks 320 which also
> >>> does one MatAssembly.
> >>
> >> There is one additional synchronization point in the
> >> MatAssemblyEnd that has not/cannot be removed. This is the
> >> construction of the VecScatter; I think that likely explains the huge
> >> amount of time there.
>
> This concerns me
>
> MatAssemblyEnd 2 1.0 7.5767e+01 1.0 0.00e+00 0.0 5.1e+06 9.4e+03
> 1.6e+01 64 0100 8 14 64 0100 8 14 0
>
> I am thinking this is all the communication needed to set up the scatter.
> Do you have access to any performance profilers like Intel speedshop to see
> what is going on during all this time?
>
>
> -vecscatter_alltoall uses alltoall in communication in the scatters but it
> does not use all to all in setting up the scatter (that is determining
> exactly what needs to be scattered at each time). I think this is the
> problem. We need to add more scatter set up code to optimize this case.
>
>
>
> >>
> >>> 2) The other bottleneck is MatMult, which itself calls VecScatter. Since
> >>> the structure of the matrix is rather dense, I'm guessing the
> >>> communication pattern should be similar to an all-to-all. After having a
> >>> look at the thread "VecScatter scaling problem on KNL", would you also
> >>> suggest me to use -vecscatter_alltoall, or do you think this would not be
> >>> appropriate for the MatMult?
> >>
> >> Please run with
> >>
> >> -vecscatter_view ::ascii_info
> >>
> >> this will give information about the number of messages and sizes
> >> needed in the VecScatter. To help decide what to do next.
> >
> > Here are two more logs. One with -vecscatter_view ::ascii_info which I
> > don't really know how to analyze (I've spotted though that there are a
> > couple of negative integers for the data counters, maybe you are using long
> > instead of long long?), the other with -vecscatter_alltoall. The latter
> > option gives a 2x speed-up for the MatMult, and for the PCApply too (which
> > is weird to me because there should be no global communication with bjacobi
> > and the diagonal blocks are only of size "5 processes" so the speed-up
> > seems rather huge for just doing VecScatter for gathering and scattering
> > the RHS/solution for all 320 MUMPS instances).
>
> ok, this is good, it confirms that the large amount of communication needed
> in the scatters were a major problem and using the all to all helps. This is
> about all you can do about the scatter time.
>
>
>
> Barry
>
> >
> > Thanks for your help,
> > Pierre
> >
> >> Barry
> >>
> >>
> >>
> >>
> >>>
> >>> Thank you very much,
> >>> Pierre
> >>>
> >>> On Mon, 6 Mar 2017 09:34:53 -0600, Barry Smith wrote:
> >>>> I don't think the lack of the --with-debugging=no is important here.
> >>>> Though he/she should use --with-debugging=no for production runs.
> >>>>
> >>>> I think the reason for the "funny" numbers is that
> >>>> MatAssemblyBegin and End in this case have explicit synchronization
> >>>> points so some processes are waiting for other processes to get to the
> >>>> synchronization point thus it looks like some processes are spending a
> >>>> lot of time in the assembly routines when they are not really, they
> >>>> are just waiting.
> >>>>
> >>>> You can remove the synchronization point by calling
> >>>>
> >>>> MatSetOption(mat, MAT_NO_OFF_PROC_ENTRIES, PETSC_TRUE); before
> >>>> calling MatMPIAIJSetPreallocationCSR()
> >>>>
> >>>> Barry
> >>>>
> >>>>> On Mar 6, 2017, at 8:59 AM, Pierre Jolivet <[email protected]>
> >>>>> wrote:
> >>>>>
> >>>>> Hello,
> >>>>> I have an application with a matrix with lots of nonzero entries (that
> >>>>> are perfectly load balanced between processes and rows).
> >>>>> A end user is currently using a PETSc library compiled with the
> >>>>> following flags (among others):
> >>>>> --CFLAGS=-O2 --COPTFLAGS=-O3 --CXXFLAGS="-O2 -std=c++11"
> >>>>> --CXXOPTFLAGS=-O3 --FFLAGS=-O2 --FOPTFLAGS=-O3
> >>>>> Notice the lack of --with-debugging=no
> >>>>> The matrix is assembled using MatMPIAIJSetPreallocationCSR and we end
> >>>>> up with something like that in the -log_view:
> >>>>> MatAssemblyBegin 2 1.0 1.2520e+002602.1 0.00e+00 0.0 0.0e+00
> >>>>> 0.0e+00 8.0e+00 0 0 0 0 2 0 0 0 0 2 0
> >>>>> MatAssemblyEnd 2 1.0 4.5104e+01 1.0 0.00e+00 0.0 8.2e+05
> >>>>> 3.2e+04 4.6e+01 40 0 14 4 9 40 0 14 4 9 0
> >>>>>
> >>>>> For reference, here is what the matrix looks like (keep in mind it is
> >>>>> well balanced)
> >>>>> Mat Object: 640 MPI processes
> >>>>> type: mpiaij
> >>>>> rows=10682560, cols=10682560
> >>>>> total: nonzeros=51691212800, allocated nonzeros=51691212800
> >>>>> total number of mallocs used during MatSetValues calls =0
> >>>>> not using I-node (on process 0) routines
> >>>>>
> >>>>> Are MatAssemblyBegin/MatAssemblyEnd highly sensitive to the
> >>>>> --with-debugging option on x86 even though the corresponding code is
> >>>>> compiled with -O2, i.e., should I tell the user to have its PETSc lib
> >>>>> recompiled, or would you recommend me to use another routine for
> >>>>> assembling such a matrix?
> >>>>>
> >>>>> Thanks,
> >>>>> Pierre
> >>> <AD-3D-320_7531028.o><AD-3D-1600_7513074.o>
> > <AD-3D-1600_7533982_info.o><AD-3D-1600_7533637_alltoall.o>
>
>
>
>
> --
> Stefano
