Re: [petsc-users] DMSWARM particle coordinates per rank
Okay, cool. Should I do that by explicitly setting each particle's DMSwarmField_rank when I assign its position? What role does DMSwarmMigrate play in all of this? --Matt == Matthew Young, PhD (he/him) Research Scientist II Space Science Center University of New Hampshire matthew.yo...@unh.edu == On Fri, May 12, 2023 at 10:05 AM Matthew Knepley wrote: > On Fri, May 12, 2023 at 9:40 AM Matthew Young < > myoung.space.scie...@gmail.com> wrote: > >> Got it. >> >> I'm specifically thinking about this in terms of the gather stage of my >> PIC code, where I loop over local particles to fill local density and flux >> arrays by linearly interpolating particle positions to the grid. The gather >> function currently assumes that the coordinates (i.e., the array >> representation of DMSwarmPICField_coor) of all particles on a given rank >> would correspond to only the global indices owned by that rank, via the >> relationship between indices and coordinates in the associated cell DM. >> Based on what you described, it sounds like I need to make sure that I >> initially lay down the particles so that their rank matches their >> coordinates. >> > > Right now, yes. I will fix that before August. > > Thanks, > > Matt > > >> --Matt >> == >> Matthew Young, PhD (he/him) >> Research Scientist II >> Space Science Center >> University of New Hampshire >> matthew.yo...@unh.edu >> == >> >> >> On Fri, May 12, 2023 at 5:15 AM Matthew Knepley >> wrote: >> >>> On Thu, May 11, 2023 at 9:15 PM Matthew Young < >>> myoung.space.scie...@gmail.com> wrote: >>> >>>> Does setting up a PIC-type DMSWARM with an associated cell DM guarantee >>>> that each MPI rank will own the particles with coordinates inside the >>>> bounds of the portion of the grid it owns? >>>> >>> >>> There is a caveat that we are currently fixing. Swarm communication is >>> setup to be nearest neighbor (since there is no coarse grid of >>> bounding boxes). So if your particles are initially in the right place, and >>> only move nearest neighbor, everything is fine. We are adding a hierarchy >>> of bounding boxes so that we can communicate anywhere. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> --Matt >>>> == >>>> Matthew Young, PhD (he/him) >>>> Research Scientist II >>>> Space Science Center >>>> University of New Hampshire >>>> matthew.yo...@unh.edu >>>> == >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >
Re: [petsc-users] DMSWARM particle coordinates per rank
Got it. I'm specifically thinking about this in terms of the gather stage of my PIC code, where I loop over local particles to fill local density and flux arrays by linearly interpolating particle positions to the grid. The gather function currently assumes that the coordinates (i.e., the array representation of DMSwarmPICField_coor) of all particles on a given rank would correspond to only the global indices owned by that rank, via the relationship between indices and coordinates in the associated cell DM. Based on what you described, it sounds like I need to make sure that I initially lay down the particles so that their rank matches their coordinates. --Matt == Matthew Young, PhD (he/him) Research Scientist II Space Science Center University of New Hampshire matthew.yo...@unh.edu == On Fri, May 12, 2023 at 5:15 AM Matthew Knepley wrote: > On Thu, May 11, 2023 at 9:15 PM Matthew Young < > myoung.space.scie...@gmail.com> wrote: > >> Does setting up a PIC-type DMSWARM with an associated cell DM guarantee >> that each MPI rank will own the particles with coordinates inside the >> bounds of the portion of the grid it owns? >> > > There is a caveat that we are currently fixing. Swarm communication is > setup to be nearest neighbor (since there is no coarse grid of > bounding boxes). So if your particles are initially in the right place, and > only move nearest neighbor, everything is fine. We are adding a hierarchy > of bounding boxes so that we can communicate anywhere. > > Thanks, > > Matt > > >> --Matt >> == >> Matthew Young, PhD (he/him) >> Research Scientist II >> Space Science Center >> University of New Hampshire >> matthew.yo...@unh.edu >> == >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >
[petsc-users] DMSWARM particle coordinates per rank
Does setting up a PIC-type DMSWARM with an associated cell DM guarantee that each MPI rank will own the particles with coordinates inside the bounds of the portion of the grid it owns? --Matt == Matthew Young, PhD (he/him) Research Scientist II Space Science Center University of New Hampshire matthew.yo...@unh.edu ==
Re: [petsc-users] DMSWARM with DMDA and KSP
Yup -- I realized that I had '-pc_type mg' in the script I was using to build and run as I developed. I guess that was causing the KSP to coarsen its DM, which made me think I had to force the density DM to be consistent. Still, refactoring my original grid DM into one for Vlasov components and one for the potential was useful. Thanks for the help! --Matt == Matthew Young, PhD (he/him) Research Scientist II Space Science Center University of New Hampshire matthew.yo...@unh.edu == On Mon, May 1, 2023 at 10:51 PM Dave May wrote: > > > On Mon 1. May 2023 at 18:57, Matthew Young > wrote: > >> Thanks for the suggestion to keep DMs separate, and for pointing me >> toward that example. I now have a DM for the particle quantities (i.e., >> density and flux) and another for the potential. I'm hoping to use >> KSPSetComputeOperators with PCGAMG, so I packed the density DM into the >> application context and set the potential DM on the KSP, but I'm not sure >> how to communicate changes in the KSP DM (e.g., coarsening) to the density >> DM inside my operator function. >> > > I don’t think you need to. > > GAMG only requires the fine grid operator - this will be the matrix > assembled from KSPSetComputeOperators. Hence density DM and potential DM > fields only need to be managed by you on the finest level. > > However, if you wanted to use PCMG with rediscretized operators on every > level, then you would need the density DM field defined on each level of > your geometric multigrid hierarchy. This could be done (possibly less than > ideally) by calling DMCreateInterpolation() and then using the Mat to > interpolate the density from the finest level to next coarsest level (and > so on). > > Thanks, > Dave > > >> >> --Matt >> == >> Matthew Young, PhD (he/him) >> Research Scientist II >> Space Science Center >> University of New Hampshire >> matthew.yo...@unh.edu >> == >> >> >> On Sun, Apr 30, 2023 at 1:52 PM Matthew Knepley >> wrote: >> >>> On Sun, Apr 30, 2023 at 1:12 PM Matthew Young < >>> myoung.space.scie...@gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> I am developing a particle-in-cell code that models ions as particles >>>> and electrons as an inertialess fluid. I use a PIC DMSWARM for the ions, >>>> which I gather into density and flux before solving a linear system for the >>>> electrostatic potential (phi). I currently have one DMDA with 5 degrees of >>>> freedom -- one each for density, 3 flux components, and phi. >>>> >>>> When setting up the linear system to solve for phi, I've been following >>>> examples like KSP ex34.c and ex42.c when writing the KSP operator and RHS >>>> functions but I'm not sure I have the right approach, since 4 of the DOFs >>>> are known and 1 is unknown. >>>> >>>> I saw this thread >>>> <https://lists.mcs.anl.gov/pipermail/petsc-users/2016-November/031031.html> >>>> that recommended using DMDAGetReducedDMDA, which I gather has been >>>> deprecated in favor of DMDACreateCompatibleDMDA. Is that a good approach >>>> for managing a regular grid with known and unknown quantities on each node? >>>> Could a composite DM be useful? Has anyone else worked on a problem like >>>> this? >>>> >>> >>> I recommend making a different DM for each kind of solve you want. >>> DMDACreateCompatibleDMDA() should be the implementation of DMClone(), but >>> we have yet to harmonize all things for all DMs. I would create one DM for >>> your Vlasov components and one for the Poisson. >>> We follow this strategy in our Vlasov-Poisson test for Landau damping: >>> https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/swarm/tests/ex9.c >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> --Matt >>>> == >>>> Matthew Young, PhD (he/him) >>>> Research Scientist II >>>> Space Science Center >>>> University of New Hampshire >>>> matthew.yo...@unh.edu >>>> == >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >>
Re: [petsc-users] DMSWARM with DMDA and KSP
Thanks for the suggestion to keep DMs separate, and for pointing me toward that example. I now have a DM for the particle quantities (i.e., density and flux) and another for the potential. I'm hoping to use KSPSetComputeOperators with PCGAMG, so I packed the density DM into the application context and set the potential DM on the KSP, but I'm not sure how to communicate changes in the KSP DM (e.g., coarsening) to the density DM inside my operator function. --Matt == Matthew Young, PhD (he/him) Research Scientist II Space Science Center University of New Hampshire matthew.yo...@unh.edu == On Sun, Apr 30, 2023 at 1:52 PM Matthew Knepley wrote: > On Sun, Apr 30, 2023 at 1:12 PM Matthew Young < > myoung.space.scie...@gmail.com> wrote: > >> Hi all, >> >> I am developing a particle-in-cell code that models ions as particles and >> electrons as an inertialess fluid. I use a PIC DMSWARM for the ions, which >> I gather into density and flux before solving a linear system for the >> electrostatic potential (phi). I currently have one DMDA with 5 degrees of >> freedom -- one each for density, 3 flux components, and phi. >> >> When setting up the linear system to solve for phi, I've been following >> examples like KSP ex34.c and ex42.c when writing the KSP operator and RHS >> functions but I'm not sure I have the right approach, since 4 of the DOFs >> are known and 1 is unknown. >> >> I saw this thread >> <https://lists.mcs.anl.gov/pipermail/petsc-users/2016-November/031031.html> >> that recommended using DMDAGetReducedDMDA, which I gather has been >> deprecated in favor of DMDACreateCompatibleDMDA. Is that a good approach >> for managing a regular grid with known and unknown quantities on each node? >> Could a composite DM be useful? Has anyone else worked on a problem like >> this? >> > > I recommend making a different DM for each kind of solve you want. > DMDACreateCompatibleDMDA() should be the implementation of DMClone(), but > we have yet to harmonize all things for all DMs. I would create one DM for > your Vlasov components and one for the Poisson. > We follow this strategy in our Vlasov-Poisson test for Landau damping: > https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/swarm/tests/ex9.c > > Thanks, > > Matt > > >> --Matt >> == >> Matthew Young, PhD (he/him) >> Research Scientist II >> Space Science Center >> University of New Hampshire >> matthew.yo...@unh.edu >> == >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >
[petsc-users] DMSWARM with DMDA and KSP
Hi all, I am developing a particle-in-cell code that models ions as particles and electrons as an inertialess fluid. I use a PIC DMSWARM for the ions, which I gather into density and flux before solving a linear system for the electrostatic potential (phi). I currently have one DMDA with 5 degrees of freedom -- one each for density, 3 flux components, and phi. When setting up the linear system to solve for phi, I've been following examples like KSP ex34.c and ex42.c when writing the KSP operator and RHS functions but I'm not sure I have the right approach, since 4 of the DOFs are known and 1 is unknown. I saw this thread <https://lists.mcs.anl.gov/pipermail/petsc-users/2016-November/031031.html> that recommended using DMDAGetReducedDMDA, which I gather has been deprecated in favor of DMDACreateCompatibleDMDA. Is that a good approach for managing a regular grid with known and unknown quantities on each node? Could a composite DM be useful? Has anyone else worked on a problem like this? --Matt == Matthew Young, PhD (he/him) Research Scientist II Space Science Center University of New Hampshire matthew.yo...@unh.edu ==
