> On 19 Nov 2015, at 11:19, Jose E. Roman <[email protected]> wrote:
>
>>
>> El 19 nov 2015, a las 10:49, Denis Davydov <[email protected]> escribió:
>>
>> Dear all,
>>
>> I was trying to get some scaling results for the GD eigensolver as applied
>> to the density functional theory.
>> Interestingly enough, the number of self-consistent iterations (solution of
>> couple eigenvalue problem and poisson equations)
>> depends on the number of MPI cores used. For my case the range of iterations
>> is 19-24 for MPI cores between 2 and 160.
>> That makes the whole scaling check useless as the eigenproblem is solved
>> different number of times.
>>
>> That is **not** the case when I use Krylov-Schur eigensolver with zero
>> shift, which makes me believe that I am missing some settings on GD to make
>> it fully deterministic. The only non-deterministic part I am currently aware
>> of is the initial subspace for the first SC iterations. But that’s the case
>> for both KS and GD. For subsequent iterations I provide previously obtained
>> eigenvectors as initial subspace.
>>
>> Certainly there will be some round-off error due to different partition of
>> DoFs for different number of MPI cores,
>> but i don’t expect it to have such a strong influence. Especially given the
>> fact that I don’t see this problem with KS.
>>
>> Below is the output of -eps-view for GD with -eps_type gd -eps_harmonic
>> -st_pc_type bjacobi -eps_gd_krylov_start -eps_target -10.0
>> I would appreciate any suggestions on how to address the issue.
>
> The block Jacobi preconditioner differs when you change the number of
> processes. This will probably make GD iterate more when you use more
> processes.
Switching to Jacobi preconditioner reduced variation in number of SC
iterations, but does not remove it.
Any other options but initial vector space which may introduce
non-deterministic behaviour?
>>
>> As a side question, why GD uses KSP pre-only? It could as well be using a
>> proper linear solver to apply K^{-1} in the expansion state --
>
> You can achieve that with PCKSP. But if you are going to do that, why not
> using JD instead of GD?
It was more a general question why the inverse is implemented by pre-only for
GD and is done properly with a full control of KSP for JD.
I will try JD as well because so far GD for my problems has a bottleneck in:
BVDot (13% time), BVOrthogonalize (10% time), DSSolve (62% time);
whereas only 11% of time is spent in MatMult.
I suppose BVDot is mostly used in BVOrthogonalize and partly in calculation of
Ritz vectors?
My best bet with DSSolve (with mpd=175 only) is a better preconditioner and
thus reduced number of iterations or double expansion with simple
preconditioner?
Regards,
Denis.
