> El 20 nov 2015, a las 12:06, Denis Davydov <[email protected]> escribió: > >> >> On 19 Nov 2015, at 11:19, Jose E. Roman <[email protected]> wrote: >> >>> >>> El 19 nov 2015, a las 10:49, Denis Davydov <[email protected]> escribió: >>> >>> Dear all, >>> >>> I was trying to get some scaling results for the GD eigensolver as applied >>> to the density functional theory. >>> Interestingly enough, the number of self-consistent iterations (solution of >>> couple eigenvalue problem and poisson equations) >>> depends on the number of MPI cores used. For my case the range of >>> iterations is 19-24 for MPI cores between 2 and 160. >>> That makes the whole scaling check useless as the eigenproblem is solved >>> different number of times. >>> >>> That is **not** the case when I use Krylov-Schur eigensolver with zero >>> shift, which makes me believe that I am missing some settings on GD to make >>> it fully deterministic. The only non-deterministic part I am currently >>> aware of is the initial subspace for the first SC iterations. But that’s >>> the case for both KS and GD. For subsequent iterations I provide previously >>> obtained eigenvectors as initial subspace. >>> >>> Certainly there will be some round-off error due to different partition of >>> DoFs for different number of MPI cores, >>> but i don’t expect it to have such a strong influence. Especially given the >>> fact that I don’t see this problem with KS. >>> >>> Below is the output of -eps-view for GD with -eps_type gd -eps_harmonic >>> -st_pc_type bjacobi -eps_gd_krylov_start -eps_target -10.0 >>> I would appreciate any suggestions on how to address the issue. >> >> The block Jacobi preconditioner differs when you change the number of >> processes. This will probably make GD iterate more when you use more >> processes. > > Switching to Jacobi preconditioner reduced variation in number of SC > iterations, but does not remove it. > Any other options but initial vector space which may introduce > non-deterministic behaviour? > >>> >>> As a side question, why GD uses KSP pre-only? It could as well be using a >>> proper linear solver to apply K^{-1} in the expansion state -- >> >> You can achieve that with PCKSP. But if you are going to do that, why not >> using JD instead of GD? > > It was more a general question why the inverse is implemented by pre-only for > GD and is done properly with a full control of KSP for JD.
GD uses the preconditioned residual to expand the subspace. JD uses the (approximate) solution of the correction equation. > > I will try JD as well because so far GD for my problems has a bottleneck in: > BVDot (13% time), BVOrthogonalize (10% time), DSSolve (62% time); > whereas only 11% of time is spent in MatMult. > I suppose BVDot is mostly used in BVOrthogonalize and partly in calculation > of Ritz vectors? > My best bet with DSSolve (with mpd=175 only) is a better preconditioner and > thus reduced number of iterations or double expansion with simple > preconditioner? DSSolve should always be small. Try reducing mpd. > > Regards, > Denis.
