Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Roy Stogner
On Wed, 1 Sep 2010, Derek Gaston wrote: > On Sep 1, 2010, at 4:36 PM, Roy Stogner wrote: > >> Initialize the vector e.g. as GHOSTED with size 0 and an empty ghost >> index set? > > Really? You would really prefer that? Seems like a hack to me. We > store other info about vectors this same way

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Roy Stogner
On Wed, 1 Sep 2010, Jed Brown wrote: > On Wed, 1 Sep 2010 12:58:11 -0500 (CDT), Roy Stogner > wrote: >> I'm not sure I understood Jed's suggestion - eliminate parallel >> non-ghosted vectors entirely, then write our own ghosting support to >> handle the non-PETSc backends? That seems like a l

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Derek Gaston
On Sep 1, 2010, at 4:36 PM, Roy Stogner wrote: > Initialize the vector e.g. as GHOSTED with size 0 and an empty ghost > index set? Really? You would really prefer that? Seems like a hack to me. We store other info about vectors this same way... like whether or not to project them. Should we

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Roy Stogner
On Wed, 1 Sep 2010, Roy Stogner wrote: > On Wed, 1 Sep 2010, Derek Gaston wrote: > >> On Sep 1, 2010, at 4:22 PM, Roy Stogner wrote: >> >>> This all sounds good except for the map - what would >>> paralleltype_map["vector_name"] get you that >>> get_vector("vector_name").type() doesn't? >> >> You

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Roy Stogner
On Wed, 1 Sep 2010, Derek Gaston wrote: > On Sep 1, 2010, at 4:22 PM, Roy Stogner wrote: > >> This all sounds good except for the map - what would >> paralleltype_map["vector_name"] get you that >> get_vector("vector_name").type() doesn't? > > You're thinking too far down the path. When you cal

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Derek Gaston
On Sep 1, 2010, at 4:22 PM, Roy Stogner wrote: > This all sounds good except for the map - what would > paralleltype_map["vector_name"] get you that > get_vector("vector_name").type() doesn't? You're thinking too far down the path. When you call add_vector() the vector doesn't (normally) get cr

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Roy Stogner
On Wed, 1 Sep 2010, Derek Gaston wrote: > add a member variable to system called > "default_parallel_vector_type" that defaults to PARALLEL. > > add_vector will get an optional argument for the ParallelType > (defaulting to PARALLEL). > > add_vector will also store off the ParallelType for each v

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Derek Gaston
On Sep 1, 2010, at 1:21 PM, Roy Stogner wrote: > rhs (and adjoint_solution, and everything other than solution) > currently gets created via add_vector, so if anyone wants to enforce > non-default vector behavior they can just create the vector themselves > before ExplicitSystem::init (or Implicit

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Jed Brown
On Wed, 1 Sep 2010 12:58:11 -0500 (CDT), Roy Stogner wrote: > I'm not sure I understood Jed's suggestion - eliminate parallel > non-ghosted vectors entirely, then write our own ghosting support to > handle the non-PETSc backends? That seems like a lot of work to get > slightly more robust suppor

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Roy Stogner
On Wed, 1 Sep 2010, Derek Gaston wrote: > Essentially, you're suggesting a std::map between vector name and > ParallelType? No need - just check NumericVector::type() for each. > Then you can pass ParallelType into add_vector()? Right. > And like you mention you'll need member variables for t

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Derek Gaston
On Sep 1, 2010, at 11:58 AM, Roy Stogner wrote: > My single-bool suggestion isn't the best way to do things. Ideally > any time we're reinitializing an existing vector (lines 246, 262, 310) > we want to query it and reinitialize it with the same type it already > had. We could use a bool (or bet

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Roy Stogner
On Wed, 1 Sep 2010, Derek Gaston wrote: > On Aug 31, 2010, at 2:36 PM, Roy Stogner wrote: > >> The effect you want should be to add a get_send_list() argument and >> change PARALLEL to GHOSTED in system.C lines 230, 246, 262, 310, 593. >> Try that first and see if it works. If it turns out to b

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-09-01 Thread Derek Gaston
On Aug 31, 2010, at 2:36 PM, Roy Stogner wrote: > The effect you want should be to add a get_send_list() argument and > change PARALLEL to GHOSTED in system.C lines 230, 246, 262, 310, 593. > Try that first and see if it works. If it turns out to be what you > need, then we could add it as an opt

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Jed Brown
On Tue, 31 Aug 2010 15:24:55 -0500 (CDT), Roy Stogner wrote: > > On Tue, 31 Aug 2010, Jed Brown wrote: > > >> But it would be sufficient for your case to just create ghosted > >> instead of parallel vectors. We can't do that in general in > >> NumericVector::build() because that doesn't have a

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Derek Gaston
On Aug 31, 2010, at 2:36 PM, Roy Stogner wrote: > The effect you want should be to add a get_send_list() argument and > change PARALLEL to GHOSTED in system.C lines 230, 246, 262, 310, 593. > Try that first and see if it works. If it turns out to be what you > need, then we could add it as an opt

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Roy Stogner
On Tue, 31 Aug 2010, Derek Gaston wrote: > On Aug 31, 2010, at 2:24 PM, Roy Stogner wrote: > >> However when we're just building a parallel vector, we only call >> NumericVector::build() with the global and local sizes, not with the >> entire context that exists in the caller. The best solution

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Derek Gaston
On Aug 31, 2010, at 2:24 PM, Roy Stogner wrote: > However when we're just building a parallel vector, we only call > NumericVector::build() with the global and local sizes, not with the > entire context that exists in the caller. The best solution is to add > a special case alternative that uses

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Roy Stogner
On Tue, 31 Aug 2010, Jed Brown wrote: >> But it would be sufficient for your case to just create ghosted >> instead of parallel vectors. We can't do that in general in >> NumericVector::build() because that doesn't have access to DofMap's >> send_list. But we could do that as an option (run-tim

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Jed Brown
On Tue, 31 Aug 2010 15:13:02 -0500 (CDT), Roy Stogner wrote: > We don't want to create ghosted instead of serial vectors - sometimes > code is going to ask for a serial vector because it really needs data > that's not on the local processor or its ghost cells. (and because > it's too lazy to buil

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Roy Stogner
On Tue, 31 Aug 2010, Derek Gaston wrote: > On Aug 31, 2010, at 1:51 PM, Roy Stogner wrote: > >> Otherwise the only thing to do is add a non-default code path, where >> instead of letting PETSc work on solution while we work to keep >> current_local_solution sync'ed up we do the opposite. > > That

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Derek Gaston
On Aug 31, 2010, at 1:51 PM, Roy Stogner wrote: > Yeah; the real trouble is that ghosted vectors have to fall back to > serial when ghosting isn't available, e.g. on Trilinos. I'm going to > assume that you don't want to implement ghosted support there? We do. And I actually have someone assign

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Jed Brown
On Tue, 31 Aug 2010 13:24:44 -0600, Derek Gaston wrote: > On Aug 31, 2010, at 1:11 PM, Jed Brown wrote: > > > If a VecGhost was passed in to SNES, then all the Krylov vectors > > will also be VecGhost. If not, create a new VecGhost > > (VecCreateGhost() or VecDuplicate() an existing VecGhost) an

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Roy Stogner
On Tue, 31 Aug 2010, Derek Gaston wrote: > > On Aug 31, 2010, at 1:44 PM, Roy Stogner wrote: > >> Actually, the only vectors which currently default to ghosted are >> old_local_solution, older_local_solution in TransientSystems, >> old_local_nonlinear_solution in unsteady DiffSystems, and >> cur

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Derek Gaston
On Aug 31, 2010, at 1:44 PM, Roy Stogner wrote: > Actually, the only vectors which currently default to ghosted are > old_local_solution, older_local_solution in TransientSystems, > old_local_nonlinear_solution in unsteady DiffSystems, and > current_local_solution in any System. But we pass solu

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Roy Stogner
On Tue, 31 Aug 2010, Derek Gaston wrote: > Well, that is odd then. Because I'm sure that libMesh is passing in > ghosted vectors (hmmm... maybe the residual (rhs) isn't ghosted by > default?). Actually, the only vectors which currently default to ghosted are old_local_solution, older_local_sol

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Derek Gaston
On Aug 31, 2010, at 1:11 PM, Jed Brown wrote: > If a VecGhost was passed in to SNES, then all the Krylov vectors will > also be VecGhost. If not, create a new VecGhost (VecCreateGhost() or > VecDuplicate() an existing VecGhost) and VecCopy from the plain Vec to > the ghost Vec. Well, that is odd

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Jed Brown
On Tue, 31 Aug 2010 11:18:17 -0500 (CDT), Roy Stogner wrote: > > > On Tue, 31 Aug 2010, Derek Gaston wrote: > > > So... up until now I've only ever needed one "ghosted" vector... for > > the solution vector. And, just as we do in petsc_nonlinear_solver.C > > I've just been bastardizing System

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Derek Gaston
On Aug 31, 2010, at 10:26 AM, Derek Gaston wrote: > Based on your explanation, I'm going to see if one of the vectors is already > ghosted... and if it is then it will work fine. The _other_ one I'll use the > current_local_solution trick on. This is a bit of a hack though Unfortunately..

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Derek Gaston
On Aug 31, 2010, at 10:18 AM, Roy Stogner wrote: > In theory, assuming your Vec v is already ghosted, all you need is: > PetscVector wrapper(v); > and then use wrapper. The constructor PetscVector::PetscVector(Vec v) > checks to see whether v is ghosted, serial, or parallel (and how big v > is) a

Re: [Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Roy Stogner
On Tue, 31 Aug 2010, Derek Gaston wrote: > So... up until now I've only ever needed one "ghosted" vector... for > the solution vector. And, just as we do in petsc_nonlinear_solver.C > I've just been bastardizing System::update() and > current_local_solution to "ghostize" a vector. > > But now t

[Libmesh-devel] Need _2_ ghosted vectors

2010-08-31 Thread Derek Gaston
So... up until now I've only ever needed one "ghosted" vector... for the solution vector. And, just as we do in petsc_nonlinear_solver.C I've just been bastardizing System::update() and current_local_solution to "ghostize" a vector. But now that won't work because I need two such vectors simult