a couple of comments: Looks like most of the time is spent in MatSolve(). [90% for np=1]
However on np=8 run, you have MatSolve() taking 42% time, whereas VecAssemblyBegin() taking 32% time. Depending upon whats beeing done with VecSetValues()/VecAssembly() - you might be able to reduce this time considerably. [ If you can generate values locally - then no communication is required. If you need to communicate values - then you can explore VecScatters() for more efficient communication] Wrt MatSolve() on 8 procs, the max/min time between any 2 procs is 2.6. [i.e slowest proc is taking 16 sec, so the fastest proc would probably be taking 6 sec.]. The max/min ratio of flops across procs is 1.8. So there is indeed a load balance issue that is contributing to different times on different processors [I guess the slowest proc is doing almost twice the amount of work as the fastest proc]. Satish On Tue, 20 Nov 2007, Tim Stitt wrote: > Satish, > > Logs attached...hope they help. > > Thanks, > > Tim. > > Satish Balay wrote: > > Can you send the -log_summary for your runs [say p=1, p=8] > > > > Satish > > > > On Tue, 20 Nov 2007, Tim Stitt wrote: > > > > > > > Hi all (again), > > > > > > I finally got some data back from the KSP PETSc code that I put together > > > to > > > solve this sparse inverse matrix problem I was looking into. Ideally I am > > > aiming for a O(N) (time complexity) approach to getting the first 'k' > > > columns > > > of the inverse of a sparse matrix. > > > > > > To recap the method: I have my solver which uses KSPSolve in a loop that > > > iterates over the first k columns of an identity matrix B and computes the > > > corresponding x vector. > > > > > > I am just a bit curious about some of the timings I am obtaining...which I > > > hope someone can explain. Here are the timings I obtained for a global > > > sparse > > > matrix (4704 x 4704) and solving for the first 1176 columns in the > > > identity > > > using P processes (processors) on our cluster. > > > > > > (Timings are given in seconds for each process performing work in the loop > > > and > > > were obtained by encapsulating the loop with the cpu_time() Fortran > > > intrinsic. > > > The MUMPS package was requested for factorisation/solving, although > > > similar > > > timings were obtained for both the native solver and SUPERLU) > > > > > > P=1 [30.92] > > > P=2 [15.47, 15.54] > > > P=4 [4.68, 5.49, 4.67, 5.07] > > > P=8 [2.36, 4,23, 2.81, 2.54, 3.42, 2.22, 1.41, 3.15] > > > P=16 [1.04, 0.45, 1.08, 0.27, 0.87, 0.93, 1.1, 1.06, 0.29, 0.34, 0.73, > > > 0.25, > > > 0.43, 1.09, 1.08, 1.1] > > > > > > Firstly, I notice very good scalability up to 16 processes...is this > > > expected > > > (by those people who use these solvers regularly)? > > > > > > Also I notice that the timings per process vary as we scale up. Is this a > > > load-balancing problem related to more non-zero values being on a given > > > processor than others? Once again is this expected? > > > > > > Please excuse my ignorance of matters relating to these solvers and their > > > operation...as it really isn't my field of expertise. > > > > > > Regards, > > > > > > Tim. > > > > > > > > > > > > > > > >
