On Tue, Oct 17, 2017 at 5:46 AM, Michael Werner <[email protected]> wrote:
> I'm not sure what you mean with this question? > The external CFD code, if that was what you referred to, can be run in > parallel. > Then why is it a problem that "in a parallel case, this call obviously gets called once per process"? Matt > Am 17.10.2017 um 11:11 schrieb Matthew Knepley: > > On Tue, Oct 17, 2017 at 4:21 AM, Michael Werner <[email protected]> > wrote: > >> That's something I'm still struggling with. In the serial case, I can >> simply extract the values from the original grid, and since the ordering of >> the Jacobian is the same there is no problem. In the parallel case this is >> still a more or less open question. That's why I thought about reordering >> the Jacobian. As long as the position of the individual IDs is the same for >> both, I don't have to care about their absolute position. >> >> I also wanted to thank you for your previous answer, it seems that the >> application ordering might be what I'm looking for. However, in the >> meantime I stumbled about another problem, that I have to solve first. My >> new problem is, that I call the external code within the shell matrix' >> multiply call. But in a parallel case, this call obviously gets called once >> per process. So right now I'm trying to circumvent this, so it might take a >> while before I'm able to come back to the original problem... >> > > I am not understanding. Is your original code parallel? > > Thanks, > > Matt > > >> Kind regards, >> Michael >> >> Am 16.10.2017 um 17:25 schrieb Praveen C: >> >> I am interested to learn more about how this works. How are the vectors >> created if the ids are not contiguous in a partition ? >> >> Thanks >> praveen >> >> On Mon, Oct 16, 2017 at 2:02 PM, Stefano Zampini < >> [email protected]> wrote: >> >>> >>> >>> 2017-10-16 10:26 GMT+03:00 Michael Werner <[email protected]>: >>> >>>> Hello, >>>> >>>> I'm having trouble with parallelizing a matrix-free code with PETSc. In >>>> this code, I use an external CFD code to provide the matrix-vector product >>>> for an iterative solver in PETSc. To increase convergence rate, I'm using >>>> an explicitly stored Jacobian matrix to precondition the solver. This works >>>> fine for serial runs. However, when I try to use multiple processes, I face >>>> the problem that PETSc decomposes the preconditioner matrix, and probably >>>> also the shell matrix, in a different way than the external CFD code >>>> decomposes the grid. >>>> >>>> The Jacobian matrix is built in a way, that its rows and columns >>>> correspond to the global IDs of the individual points in my CFD mesh >>>> >>>> The CFD code decomposes the domain based on the proximity of points to >>>> each other, so that the resulting subgrids are coherent. However, since its >>>> an unstructured grid, those subgrids are not necessarily made up of points >>>> with successive global IDs. This is a problem, since PETSc seems to >>>> partition the matrix in coherent slices. >>>> >>>> I'm not sure what the best approach to this problem might be. Is it >>>> maybe possible to exactly tell PETSc, which rows/columns it should assign >>>> to the individual processes? >>>> >>>> >>> If you are explicitly setting the values in your Jacobians via >>> MatSetValues(), you can create a ISLocalToGlobalMapping >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/ >>> IS/ISLocalToGlobalMappingCreate.html >>> >>> that maps the numbering you use for the Jacobians to their counterpart >>> in the CFD ordering, then call MatSetLocalToGlobalMapping and then use >>> MatSetValuesLocal with the same arguments you are calling MatSetValues now. >>> >>> Otherwise, you can play with the application ordering >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/ >>> AO/index.html >>> >>> >>> >>> >>> -- >>> Stefano >>> >> >> >> -- >> >> ____________________________________________________ >> >> Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR) >> Institut für Aerodynamik und Strömungstechnik | Bunsenstr. 10 | 37073 >> Göttingen >> <https://maps.google.com/?q=Bunsenstr.+10+%7C+37073+G%C3%B6ttingen&entry=gmail&source=g> >> >> Michael Werner >> Telefon 0551 709-2627 | Telefax 0551 709-2811 | [email protected] >> DLR.de >> >> >> >> >> >> >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/%7Emk51/> > > > -- > > ____________________________________________________ > > Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR) > Institut für Aerodynamik und Strömungstechnik | Bunsenstr. 10 | 37073 > Göttingen > <https://maps.google.com/?q=Bunsenstr.+10+%7C+37073+G%C3%B6ttingen&entry=gmail&source=g> > > Michael Werner > Telefon 0551 709-2627 | Telefax 0551 709-2811 | [email protected] > DLR.de > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>
