No problem, here is the code: // the numbers of processors per direction are (int) x_procs, y_procs, z_procs respectively // (no parallelization in direction 'dir' means dir_procs = 1)
MPI_Comm NewComm; int MPI_Rank, NewRank, x,y,z; // get rank from MPI ordering: MPI_Comm_rank(MPI_COMM_WORLD, &MPI_Rank); // calculate coordinates of cpus in MPI ordering: x = MPI_rank / (z_procs*y_procs); y = (MPI_rank % (z_procs*y_procs)) / z_procs; z = (MPI_rank % (z_procs*y_procs)) % z_procs; // set new rank according to PETSc ordering: NewRank = z*y_procs*x_procs + y*x_procs + x; // create communicator with new ranks according to PETSc ordering: MPI_Comm_split(PETSC_COMM_WORLD, 1, NewRank, &NewComm); // override the default communicator (was MPI_COMM_WORLD as default) PETSC_COMM_WORLD = NewComm; I hope, this will be useful for some of you. Ciao, Rolf ------------------------------------------------------- Rolf Kuiper Max-Planck Institute for Astronomy K?nigstuhl 17 69117 Heidelberg Office A5, Els?sser Labor Phone: 0049 (0)6221 528 350 Mail: kuiper at mpia.de Homepage: http://www.mpia.de/~kuiper ------------------------------------------------------- Am 04.07.2009 um 19:24 schrieb Barry Smith: > > Send us the code to do the conversion and we'll include as a > utility. > > Barry > > On Jul 4, 2009, at 6:08 AM, Rolf Kuiper wrote: > >> Thanks Barry! >> It's working. But by the way: You simply should offer such a second >> communicator inside the PETSc-library. >> >> Thanks for all your help, the support we got from this mailing list >> is amazing, >> Rolf >> >> >> Am 04.07.2009 um 01:44 schrieb Barry Smith: >>> >>> Use MPI_Comm_split() with the same color for all processors, then >>> use the second integer argument to indicate the new rank you want >>> for the process. >>> Choice the new rank so its x,y coordinate in the logical grid will >>> match the y,x coordinate in the cartesian grid. >>> >>> Barry >>> >>> On Jul 3, 2009, at 12:09 PM, Rolf Kuiper wrote: >>> >>>> Hi Barry, >>>> >>>> I tried that already with: >>>> First way by copying: >>>> MPI_Comm_dup(PETSC_COMM_WORLD, &MyComm); >>>> >>>> Second way by creating: >>>> int dims[3] = {0,0,0}; >>>> int ndims=3; >>>> MPI_Dims_create(NumberOfProcessors, ndims, dims); >>>> int false = 0; int true = 1; >>>> int periods[3] = { false, false, true }; >>>> int reorder = true; >>>> MPI_Comm MyComm; >>>> MPI_Cart_create(PETSC_COMM_WORLD, ndims, dims, periods, reorder, >>>> &MyComm); >>>> >>>> in the end then: >>>> PETSC_COMM_WORLD = MyComm; >>>> >>>> I test the MyComm with MPI_Topo_test(); and it is cartesian, yes. >>>> I can the coordinates of the cpus with MPI_Cart_coords(MyComm, >>>> LocalRank, ndims, coords); , but I found no way to set/rearrange >>>> these coordinates. >>>> >>>> Do you can help me in that case or have I to ask a MPI-support? >>>> >>>> Thanks for all, >>>> Rolf >>>> >>>> >>>> Am 03.07.2009 um 17:56 schrieb Barry Smith: >>>>> >>>>> In designing the PETSc DA I did not (by ignorance) follow the >>>>> layout approach of the MPI cartesian MPI_Cart_create (that gives >>>>> the first local cpus first in the y-direction). >>>>> I had it put the first cpus in the x-direction. >>>>> >>>>> What you need to do is create a new communicator that changes >>>>> the order of the processors so that when used by the PETSc DA >>>>> they lie out in the ordering that matches the other code. You >>>>> will need to read up on the MPI_Cart stuff. >>>>> >>>>> To change PETSC_COMM_WORLD you simply set PETSC_COMM_WORLD = >>>>> yournewcom BEFORE calling PetscInitialize(). >>>>> >>>>> Barry >>>>> >>>>> On Jul 3, 2009, at 3:52 AM, Rolf Kuiper wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Am 30.06.2009 um 02:24 schrieb Barry Smith: >>>>>>> >>>>>>> On Jun 29, 2009, at 7:07 PM, Rolf Kuiper wrote: >>>>>>> >>>>>>>> Hi PETSc users, >>>>>>>> >>>>>>>> I ran into trouble in combining my developed PETSc >>>>>>>> application with another code (based on another library >>>>>>>> called "ArrayLib"). >>>>>>>> The problem is the parallel layout for MPI, e.g. in 2D with 6 >>>>>>>> cpus the ArrayLib code gives the names/ranks of the local >>>>>>>> cpus first in y-direction, than in x (from last to first, in >>>>>>>> the same way the MPI arrays are called, like 3Darray[z][y][x]): >>>>>>>> >>>>>>>> y >>>>>>>> ^ >>>>>>>> | 2-4-6 >>>>>>>> | 1-3-5 >>>>>>>> |--------> x >>>>>>>> >>>>>>>> If I call DACreate() from PETSc, it will assume an ordering >>>>>>>> according to names/ranks first set in x-direction, than in y: >>>>>>>> >>>>>>>> y >>>>>>>> ^ >>>>>>>> | 4-5-6 >>>>>>>> | 1-2-3 >>>>>>>> |--------> x >>>>>>>> >>>>>>>> Of course, if I now communicate the boundary values, I mix up >>>>>>>> the domain (build by the other program). >>>>>>>> >>>>>>>> Is there a possibility / a flag to set the name of the ranks? >>>>>>>> Due to the fact that my application is written and working in >>>>>>>> curvilinear coordinates and not in cartesian, I cannot just >>>>>>>> switch the directions. >>>>>>> >>>>>>> What we recommend in this case is to just change the meaning >>>>>>> of x, y, and z when you use the PETSc DA. This does mean >>>>>>> changing your code that uses the PETSc DA. >>>>>> >>>>>> The code is used as a module for many codes, so I would prefer >>>>>> to not change the code (and the meaning of directions, that's >>>>>> not user-friendly), but 'just' change the communicator. >>>>>> >>>>>>> I do not understand why curvilinear coordinates has anything >>>>>>> to do with it. Another choice is to create a new MPI >>>>>>> communicator that has the different ordering of the ranks of >>>>>>> the processors and then using that comm to create the PETSc DA >>>>>>> objects; then you would not need to change your code that >>>>>>> calls PETSc. >>>>>> >>>>>> I tried some time before to use the PetscSetCommWorld() >>>>>> routine, but I can't find it anymore, how can I set a new >>>>>> communicator in PETSc3.0? >>>>>> The communicator, I want to use, is the MPI_COMM_WORLD, which >>>>>> takes the first described ordering. >>>>>> Now I read that the MPI_COMM_WORLD is the default communicator >>>>>> for PETSc. But why is the ordering than different? >>>>>> >>>>>> Sorry for all this question, but (as you can see) I really >>>>>> don't understand this comm problem at the moment, >>>>>> Thanks for all, >>>>>> Rolf >>>>>> >>>>>>> Unfortunately PETSc doesn't have any way to flip how the DA >>>>>>> handles the layout automatically. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>>> >>>>>>>> Thanks a lot for your help, >>>>>>>> Rolf >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20090704/28fceefe/attachment-0001.htm>
