Am Fr., 22. Apr. 2022 um 16:04 Uhr schrieb Barry Smith <[email protected]>:
> > We would need more details as to exactly what goes wrong to determine > any kind of fix; my guess would be that the layout of the velocity vectors > and temperature vectors is slightly different since the DMDA uses nelx+1 > while the stag uses nelx and may split things up slightly differently in > parallel. You could try very small problems with say 2 or 4 ranks and put > known values into the vectors and look at the ghost point update locations > and exact locations in the local and global vectors to make sure everything > is where you expect it to be. > > There is also the possibility of using a DMStag for the temperature > instead of DMDA since DMDA provides essentially a subset of the > functionality of DMDA to get a more consistent layout of the unknowns in > the two vectors. > > This was going to be my first suggestion as well - one way to ensure compatibility would be to use DMStag for everything. E.g. you could create your temperature DMStag with only vertex/corner (dof30 degrees of freedom, and then create one or more a "compatible" DMStags (same elements on each MPI rank) for your other fields, using DMStagCreateCompatibleDMStag() . Finally, one could use a single DMStag with all the unknowns but treat > subvectors of the unknowns differently in your discretization and solve > process. So assign velocity values (I guess) to the cell faces and > temperature to the cell vertices, this will give consistent parallel > decomposition of values but you will have to manage the fact that the > unknowns are interlaced into a single vector so your solver portions of the > code may need to "pull" out appropriate subvectors. > > Barry > > > On Apr 22, 2022, at 9:45 AM, Carl-Johan Thore <[email protected]> > wrote: > > Hi! > > I'm working on a convection-diffusion heat transfer problem. The > temperature > is discretized using standard Q1 elements and a DMDA. The flow is modelled > using a stabilized Q1-Q0 method for which DMStag seemed like a good > choice. The codes for the temperature > and flow work fine separately (both in serial and parallel), but when > combined and running > in parallel, a problem sometimes arises in the assembly of the thermal > system matrix. > Here’s a rough sketch of the combined code: > > // Create dmda for thermal problem and dmstag for flow problem > DMDACreate3d(PETSC_COMM_WORLD, bx, by, bz, DMDA_STENCIL_BOX, nelx+1, > nely+1, nelz+1, PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, > 1, stencilWidth, 0, 0, 0, > &dmthermal); > … > // A bit of code to adjust Lx,Ly,Lz so that dmthermal and dmflow are > compatible in the sense of having the same > // local elements > … > DMStagCreate3d(PETSC_COMM_WORLD, bx, by, bz, nelx, nely, nelz, md,nd,pd, > 3,0,0,0,DMSTAG_STENCIL_BOX,stencilWidth,Lx,Ly,Lz,&dmflow); > > > PetscInt edofT[8]; // 8-noded element with 1 temp DOF per node > DMStagStencil edofF[24]; // 8 nodes with 3 velocity DOFs each > > // Assemble thermal system matrix K > for (PetscInt e=0 ...) // Loop over local elements > { > // Populate edofF, edofT > // Get element velocities in ue from local > velocity vector uloc > > DMStagVecGetValuesStencil(dmflow,uloc,24,edof,ue); > ... > Ke = Ke_diffusion + Ke_convection(ue) > ... > MatSetValuesLocal(K, 8, edofT, 8, edofT, Ke, > ADD_VALUES); > } > > This always works fine in serial, but depending on the mesh and the number > of ranks, > we don't always get the correct values in the element velocity vector ue. > I suspect > this has something to do with the ordering of the elements and/or the > DOFs, because > the elements in the global velocity vector are always the same but their > order may change > (judging from the output of VecView at least). > > Is it possible to ensure compatibility between the dm:s, or find some kind > of mapping > between them, so that something along the lines of the code above always > works? > > Kind regards, > Carl-Johan > > >
