Peder, Your example scatters a parallel vector to a sequential vector on one rank. It is a pattern like MPI_Gatherv. I want to see how you scatter parallel vectors to sequential vectors on every rank.
--Junchao Zhang On Fri, Jul 2, 2021 at 4:07 AM Peder Jørgensgaard Olesen <[email protected]> wrote: > Matt's method seems to work well, though instead of editing the actual > function I put the relevant parts directly into my code. I made the small > example attached here. > > > I might look into Star Forests at some point, though it's not really > touched upon in the manual (I will probably take a look at your paper, > https://arxiv.org/abs/2102.13018). > > > Med venlig hilsen / Best regards > > Peder > ------------------------------ > *Fra:* Junchao Zhang <[email protected]> > *Sendt:* 1. juli 2021 16:38:29 > *Til:* Jed Brown > *Cc:* Peder Jørgensgaard Olesen; [email protected] > *Emne:* Re: Sv: [petsc-users] Scatter parallel Vec to sequential Vec on > non-zeroth process > > Peder, > PETSCSF_PATTERN_ALLTOALL only supports MPI_Alltoall (not Alltoallv), and > is only used by petsc internally at few places. > I suggest you can go with Matt's approach. After it solves your problem, > you can distill an example to demo the communication pattern. Then we can > see how to efficiently support that in petsc. > > Thanks. > --Junchao Zhang > > > On Thu, Jul 1, 2021 at 7:42 AM Jed Brown <[email protected]> wrote: > >> Peder Jørgensgaard Olesen <[email protected]> writes: >> >> > Each process is assigned an indexed subset of the tasks (the tasks are >> of constant size), and, for each task index, the relevant data is scattered >> as a SEQVEC to the process (this is done for all processes in each step, >> using an adaption of the code in Matt's link). This way each process only >> receives just the data it needs to complete the task. While I'm currently >> working with very moderate size data sets I'll eventually need to handle >> something rather more massive, so I want to economize memory where possible >> and give each process only the data it needs. >> >> From the sounds of it, this pattern ultimately boils down to MPI_Gather >> being called P times where P is the size of the communicator. This will >> work okay when P is small, but it's much less efficient than calling >> MPI_Alltoall (or MPI_Alltoallv), which you can do by creating one PetscSF >> that ships the needed data to each task and PETSCSF_PATTERN_ALLTOALL. You >> can see an example. >> >> >> https://gitlab.com/petsc/petsc/-/blob/main/src/vec/is/sf/tests/ex3.c#L93-151 >> >
