Barry Smith wrote: > > > > > On Thu, 18 May 2006, Sean Dettrick wrote: > >> Hi Barry, >> the order is determined by MPI_Cart_create. > > > Do you mean that MPI_Cart_create() orders across the 2nd (y-axis) > fastest and then the first (x-axis)?
I am not sure that my interpretation is correct. I'll send you a small demo, so you can see for yourself if you agree with it. Sean > Hmmm, maybe we should change the > DA? Changing it once and for all (not supporting both) is probably > not a big deal and shouldn't break much (I hope). > Bill, what do you think? > > Barry > >> I've tried to swap x and y in the DA to make the problem go away but >> haven't succeeded yet. Not enough brain cells I suspect ... still >> thinking about it. >> Thanks for the response, >> Sean >> >> Barry Smith wrote: >> >>> >>> >>> Sean, >>> >>> I don't completely understand what goes wrong. Is it because YOUR >>> application orders the processors related to geometry in the >>> following way? >>> >>> ^ y direction >>> | >>> 2 5 8 >>> 1 4 7 >>> 0 3 6 >>> >>> -> x direction >>> >>> Or is this something inherent in MPI_Cart_create? >>> >>> PETSc does it so >>> >>> ^ y direction >>> | >>> 6 7 8 >>> 3 4 5 >>> 0 1 2 >>> >>> -> x direction >>> >>> If you want to "force" the PETSc DA to match the first case you >>> could just >>> "lie" to PETSc and treat the x direction as the y direction and the >>> x as the y. >>> Thus you would flip the i and j indices. You could do the same trick in >>> 3d. >>> >>> There is no easy way to change the DA to do the ordering as the first >>> case above or support both approaches (possible? yes, but ugly >>> duplicate code to handle the two cases). >>> >>> I've cc:ed to Bill since he would know more about the details of >>> MPI_cart_create(). >>> >>> Barry >>> >>> On Mon, 15 May 2006, Sean Dettrick wrote: >>> >>>> Hi, >>>> >>>> I'm trying to use DACreate2d and KSP in my existing MPI >>>> application. I already have a Cartesian communicator established, >>>> and I set PETSC_COMM_WORLD equal to it and then call PetscInitialize. >>>> >>>> This works fine on a prime number of CPUs, because there is only >>>> one possible ordered MPI layout in one dimension. But with a >>>> non-prime number there are two possible ordered layouts and it just >>>> happens that my 2D CPU layout (determined by MPI_Cart_create) is >>>> the transpose of the PETSc 2D CPU layout. Is there a way to >>>> organize the DA layout more explicitly than with DACreate2d? Or >>>> to tell PETSc to transpose its CPU order? I also wonder about the >>>> 3D case. >>>> >>>> thanks >>>> Sean >>>> >>>> >>> >>> >> >> > >
