Thank you for the explanation, first I'll try the null space set up and then I come back to your hints.
Jed Brown schrieb: > Andreas Grassl wrote: >> Barry Smith schrieb: >>> Hmm, it sounds like the difference between local "ghosted" vectors >>> and the global parallel vectors. But I do not understand why any of the >>> local vector entries would be zero. >>> Doesn't the vector X that is passed into KSP (or SNES) have the global >>> entries and uniquely define the solution? Why is viewing that not right? >>> >> I still don't understand fully the underlying processes of the whole PCNN >> solution procedure, but trying around I substituted >> >> MatCreateIS(commw, ind_length, ind_length, PETSC_DECIDE, PETSC_DECIDE, >> gridmapping, &A); > > This creates a matrix that is bigger than you want, and gives you the > dead values at the end (global dofs that are not in the range of the > LocalToGlobalMapping. > > This from the note on MatCreateIS: > > | m and n are NOT related to the size of the map, they are the size of the > part of the vector owned > | by that process. m + nghosts (or n + nghosts) is the length of map since > map maps all local points > | plus the ghost points to global indices. > >> by >> >> MatCreateIS(commw, PETSC_DECIDE, PETSC_DECIDE, actdof, actdof, gridmapping, >> &A); > > This creates a matrix of the correct size, but it looks like it could > easily end up with the "wrong" dofs owned locally. What you probably > want to do is: > > 1. Resolve ownership just like with any other DD method. This > partitions your dofs into n owned dofs and ngh ghosted dofs on each > process. The global sum of n is N, the size of the global vectors that > the solver will interact with. > > 2. Make an ISLocalToGlobalMapping where all the owned dofs come first, > mapping (0..n-1) to (rstart..rstart+n-1), followed by the ghosted dofs > (local index n..ngh-1) which map to remote processes. (rstart is the > global index of the first owned dof) > > One way to do this is to use MPI_Scan to find rstart, then number all > the owned dofs and scatter the result. The details will be dependent on > how you store your mesh. (I'm assuming it's unstructured, this step is > trivial if you use a DA.) > > 3. Call MatCreateIS(comm,n,n,PETSC_DECIDE,PETSC_DECIDE,mapping,&A); > >> Furthermore it seems, that the load balance is now better, although I still >> don't reach the expected values, e.g. >> ilu-cg 320 iterations, condition 4601 >> cg only 1662 iterations, condition 84919 >> >> nn-cg on 2 nodes 229 iterations, condition 6285 >> nn-cg on 4 nodes 331 iterations, condition 13312 >> >> or is it not to expect, that nn-cg is faster than ilu-cg? > > It depends a lot on the problem. As you probably know, for a second > order elliptic problem with exact subdomain solves, the NN > preconditioned operator (without a coarse component) has condition > number that scales as > > (1/H^2)(1 + log(H/h))^2 > > where H is the subdomain diameter and h is the element size. > In contrast, overlapping additive Schwarz is > > 1/H^2 > > and block Jacobi is > > 1/(Hh) > > (the original problem was 1/h^2) > > In particular, there is no reason to expect that NN is uniformly better > than ASM, although it may be for certain problems. When a coarse > solve is used, NN becomes > > (1 + log(H/h))^2 > > which is quasi-optimal (these methods are known as BDDC, which is > essentially equivalent to FETI-DP). The key advantage over multigrid > (or multilivel Schwarz) is improved robustness with variable > coefficients. My understanding is that PCNN is BDDC, and uses direct > subdomain solves by default, but I could have missed something. In > particular, if the coarse solve is missing or inexact solves are used, > you could easily see relatively poor scaling. AFAIK, it's not for > vector problems at this time. > > Good luck. > > > Jed > -- /"\ \ / ASCII Ribbon X against HTML email / \ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 315 bytes Desc: OpenPGP digital signature URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20090604/b76217f4/attachment.pgp>
