Thanks for your comments. Pierre
On Sat, 24 Jun 2017 16:50:38 -0500, Barry Smith wrote:
On Jun 24, 2017, at 4:44 PM, Pierre Jolivet <[email protected]> wrote:Hello Barry,Sorry to bump up this old thread, but I’m still struggling with MatAssembly.Just as a reminder, on 1280 processors, reference timings:MatAssemblyBegin 2 1.0 1.4302e+006436.3 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 1 0 0 0 1 1 0 0 0 1 0 MatAssemblyEnd 2 1.0 5.0301e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 31 0 0 3 7 31 0 0 3 9 0I’ve turned VecScatterCreate_PtoS into a no-op (almost), so I’m rather satisfied with this part. The new timings were: MatAssemblyBegin 2 1.0 1.6892e+007015.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 2 0 0 0 0 2 0 MatAssemblyEnd 2 1.0 2.9842e+01 1.1 0.00e+00 0.0 9.0e+03 7.9e+04 1.7e+01 15 0 0 0 9 15 0 0 0 11 0I dug around in the code, and it turned out that a high percentage of the ~30 seconds were spent in MatSetUpMultiply_MPIAIJ.I tried to set PETSC_USE_CTABLE to 0 and now I get:MatAssemblyBegin 3 1.0 1.4183e+005960.8 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 2 0 0 0 0 2 0 MatAssemblyEnd 3 1.0 7.4979e+00 1.3 0.00e+00 0.0 9.0e+03 7.9e+04 1.7e+01 4 0 70 0 14 4 0 70 0 14 0My follow-up questions are thus:1) any ideas why using ctable is a terrible idea here? Looking at ctable.c it looks like ctable is an integer-specific hash table, so I’m guessing the number of collisions is rather low and it should be efficient.We don't understand this. We occasionally see really bad performance with stable. It is usually just on a subset of processes.Have you tried with the master branch? We tried mucking with it to getbetter performance.2) why is PETSC_USE_CTABLE not a runtime option, and a preprocessor variable instead? Is it wrong to set PETSC_USE_CTABLE to 0 for MatSetUpMultiply_MPIAIJ, and 1 elsewhere?Good question. It should be runtime option attached to each object so it can be turned on and off as needed. We'd be happy to receive a pull request that fixed this or post an issue at bitbucket.com/petsc/petscBTW, just as a reminder, my matrices have really weird sparsity patterns with extremely large bandwidths. Here is an example for the dimensions of the off-diagonal block B (of the MPIAIJ format): 26871 x 21416009.Thanks in advance for your help, PierreOn 20 Mar 2017, at 3:22 PM, Pierre Jolivet <[email protected]> wrote:Hello Barry,It looks like my vendor mpirun does not support OpenSpeedShop, and I have been too lazy recompiling everything with IntelMPI. However, I did some really basic profiling and it looks like you were right, a lot of time is spent in VecScatterCreate_PtoS. I switched to a MPI_Alltoallv implementation and here is the new summary.MatAssemblyEnd 2 1.0 4.3129e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 51 0 0 4 15 51 0 0 4 15 0That's roughly 30 seconds faster, but I still find that rather slow. I'll now try an MPI_Alltoall implementation with padding because I know for a fact that BullxMPI performances for variable-sized collectives are much worse than for uniform collectives (+ all my local dimensions are almost the same so the memory cost of padding will be negligible).Thanks, Pierre On Fri, 17 Mar 2017 22:02:26 +0100, Pierre Jolivet wrote:The number of messages during the MatAssembly is effectively halvedMatAssemblyEnd 2 1.0 7.2139e+01 1.0 0.00e+00 0.0 2.6e+06 1.9e+04 1.8e+01 62 0 99 8 15 62 0 99 8 15 0 But that was only a few second faster (and this may even only be system noise). I’ll see what I can infer from the openspeedshop profiling, andmight give another MPI implementation a try during the weekend (I’m using BullxMPI, based on an ancient OpenMPI, but maybe IntelMPI givesbetter results). Thanks anyway! PierreOn Mar 17, 2017, at 9:23 PM, Pierre Jolivet wrote:Thank you for all your input. openspeedshop/2.1 is installed on mycluster but it appears something is wrong with the MPI wrapper so I’ll have to wait for the answer from the support on Monday. In the meantime I’ll try the patch from Stefano which looks very promising since it will replace 1599 sends and 1599 receives by a single all-to-all. Thanks again! PierreOn Mar 17, 2017, at 8:59 PM, Stefano Zampini wrote: 2017-03-17 22:52 GMT+03:00 Barry Smith :Stefano, Thanks this is very helpful. --------------------- Why not? here is my naive implementation with AlltoAll, which perform better in my case PetscErrorCode PetscGatherMessageLengths(MPI_Comm comm,PetscMPIInt nsends,PetscMPIInt nrecvs,const PetscMPIInt ilengths[],PetscMPIInt **onodes,PetscMPIInt **olengths) { PetscErrorCode ierr; PetscMPIInt size,i,j; PetscMPIInt *all_lengths; PetscFunctionBegin; ierr = MPI_Comm_size(comm,&size);CHKERRQ(ierr); ierr =PetscMalloc(size*sizeof(PetscMPIInt),&all_lengths);CHKERRQ(ierr);ierr =MPI_Alltoall((void*)ilengths,1,MPI_INT,all_lengths,1,MPI_INT,comm);CHKERRQ(ierr);ierr = PetscMalloc(nrecvs*sizeof(PetscMPIInt),olengths);CHKERRQ(ierr); ierr = PetscMalloc(nrecvs*sizeof(PetscMPIInt),onodes);CHKERRQ(ierr); for (i=0,j=0; iAt that time I just fixed (1), not (2). My specific problem was not with timings per se, but with MPI (IntelMPI if I remember correctly) crashing when doing the rendez-vous with thousands of processes.Don't go to sleep yet, I may have more questions :-) BarryOn Mar 17, 2017, at 2:32 PM, Stefano Zampini wrote: Pierre, I remember I had a similar problem some years ago whenworking with matrices with "process-dense" rows (i.e., when the off-diagonal part is shared by many processes). I fixed the issue by changing the implementation of PetscGatherMessageLenghts, from rendez-vous to all-to-all.Barry, if you had access to petsc-maint, the title of thethread is "Problem with PetscGatherMessageLengths".Hope this helps, Stefano 2017-03-17 22:21 GMT+03:00 Barry Smith :On Mar 17, 2017, at 4:04 AM, Pierre Jolivet wrote: On Thu, 16 Mar 2017 15:37:17 -0500, Barry Smith wrote:On Mar 16, 2017, at 10:57 AM, Pierre Jolivet wrote: Thanks Barry. I actually tried the application myself with my optimizedbuild + your option. I'm attaching two logs for a strongscalinganalysis, if someone could spend a minute or two looking at the numbers I'd be really grateful:1) MatAssembly still takes a rather long time IMHO. Thisis actually the bottleneck of my application. Especially on1600cores, the problem here is that I don't know if the huge time (almost a 5x slow-down w.r.t. the run on 320 cores) is due to MatMPIAIJSetPreallocationCSR (which I assumed beforehand was a no-op, but which is clearly not the case looking at the run on 320 cores) or the the option -pc_bjacobi_blocks 320 which also does one MatAssembly.There is one additional synchronization point in the MatAssemblyEnd that has not/cannot be removed. This is the construction of the VecScatter; I think that likelyexplains the hugeamount of time there.This concerns me MatAssemblyEnd 2 1.0 7.5767e+01 1.0 0.00e+00 0.0 5.1e+069.4e+03 1.6e+01 64 0100 8 14 64 0100 8 14 0I am thinking this is all the communication needed to set upthe scatter. Do you have access to any performance profilers like Intel speedshop to see what is going on during all this time?-vecscatter_alltoall uses alltoall in communication in thescatters but it does not use all to all in setting up the scatter (that is determining exactly what needs to be scattered at each time). I think this is the problem. We need to add more scatter set up code to optimize this case.2) The other bottleneck is MatMult, which itself callsVecScatter. Since the structure of the matrix is rather dense, I'm guessing the communication pattern should be similar to an all-to-all. After having a look at the thread "VecScatter scaling problem on KNL", would you also suggest me to use -vecscatter_alltoall, or do you think this would not be appropriate for the MatMult?Please run with -vecscatter_view ::ascii_info this will give information about the number of messagesand sizesneeded in the VecScatter. To help decide what to do next.Here are two more logs. One with -vecscatter_view::ascii_info which I don't really know how to analyze (I've spotted though that there are a couple of negative integers for the data counters, maybe you are using long instead of long long?), the other with -vecscatter_alltoall. The latter option gives a 2x speed-up for the MatMult, and for the PCApply too (which is weird to me because there should be no global communication with bjacobi and the diagonal blocks are only of size "5 processes" so the speed-up seems rather huge for just doing VecScatter for gathering and scattering the RHS/solution for all 320 MUMPS instances).ok, this is good, it confirms that the large amount ofcommunication needed in the scatters were a major problem and using the all to all helps. This is about all you can do about the scatter time.BarryThanks for your help, PierreBarryThank you very much, Pierre On Mon, 6 Mar 2017 09:34:53 -0600, Barry Smith wrote:I don't think the lack of the --with-debugging=no isimportant here.Though he/she should use --with-debugging=no forproduction runs.I think the reason for the "funny" numbers is that MatAssemblyBegin and End in this case have explicitsynchronizationpoints so some processes are waiting for other processesto get to thesynchronization point thus it looks like some processesare spending alot of time in the assembly routines when they are notreally, theyare just waiting. You can remove the synchronization point by calling MatSetOption(mat, MAT_NO_OFF_PROC_ENTRIES, PETSC_TRUE);beforecalling MatMPIAIJSetPreallocationCSR() BarryOn Mar 6, 2017, at 8:59 AM, Pierre Jolivet wrote: Hello, I have an application with a matrix with lots ofnonzero entries (that are perfectly load balanced between processes and rows).A end user is currently using a PETSc library compiledwith the following flags (among others):--CFLAGS=-O2 --COPTFLAGS=-O3 --CXXFLAGS="-O2-std=c++11" --CXXOPTFLAGS=-O3 --FFLAGS=-O2 --FOPTFLAGS=-O3Notice the lack of --with-debugging=no The matrix is assembled usingMatMPIAIJSetPreallocationCSR and we end up with something like that in the -log_view:MatAssemblyBegin 2 1.0 1.2520e+002602.1 0.00e+00 0.00.0e+00 0.0e+00 8.0e+00 0 0 0 0 2 0 0 0 0 2 0MatAssemblyEnd 2 1.0 4.5104e+01 1.0 0.00e+00 0.08.2e+05 3.2e+04 4.6e+01 40 0 14 4 9 40 0 14 4 9 0For reference, here is what the matrix looks like (keepin mind it is well balanced)Mat Object: 640 MPI processes type: mpiaij rows=10682560, cols=10682560 total: nonzeros=51691212800, allocatednonzeros=51691212800total number of mallocs used during MatSetValues calls=0not using I-node (on process 0) routines Are MatAssemblyBegin/MatAssemblyEnd highly sensitive tothe --with-debugging option on x86 even though thecorrespondingcode is compiled with -O2, i.e., should I tell the user to have its PETSc lib recompiled, or would you recommend me to use another routine for assembling such a matrix?Thanks, Pierre-- Stefano-- StefanoLinks: ------ [1] mailto:[email protected] [2] mailto:[email protected] [3] mailto:[email protected] [4] mailto:[email protected] [5] mailto:[email protected] [6] mailto:[email protected] [7] mailto:[email protected] [8] mailto:[email protected]
