This is a problem with MPI programming and optimization; I am unaware of a perfect solution.
Put the design variables into the solution vector on MPI rank 0, and when doing your objective/gradient, send the values to all the MPI processes where you use them. You can use a VecScatter to handle the communication you need or MPI_Scatter() etc whatever makes the most sense in your code. Barry > On Jan 30, 2024, at 10:53 AM, Guenther, Stefanie via petsc-users > <petsc-users@mcs.anl.gov> wrote: > > Hi Petsc team, > > I have a question regarding parallel layout of a Petsc vector to be used in > TAO optimizers for cases where the optimization variables split into ‘design’ > and ‘state’ variables (e.g. such as in PDE-constrained optimization as in > tao_lcl). In our case, the state variable naturally parallelizes evenly > amongst multiple processors and this distribution is fixed. The ‘design’ > vector however does not, it is very short compared to the state vector and it > is required on all state-processors when evaluating the objective function > and gradient. My question would be how the TAO optimization vector x = > [design,state] should be created in such a way that the ‘state’ part is > distributed as needed in our solver, while the design part is not. > > My only idea so far was to copy the design variables to all processors and > augment / interleave the optimization vector as x = [state_proc1,design, > state_proc2, design, … ] . When creating this vector in parallel on > PETSC_COMM_WORLD, each processor would then own the same number of variables > ( [state_proc<i>, design] ), as long as the numbers match up, and I would > only need to be careful when gathering the gradient wrt the design parts from > all processors. > > This seems cumbersome however, and I would be worried whether the > optimization problem is harder to solve this way. Is there any other way to > achieve this splitting, that I am missing here? Note that the distribution of > the state itself is given and can not be changed, and that the state vs > design vectors have very different (and independent) dimensions. > > Thanks for your help and thoughts! > Best, > Stefanie
