----- Original Message ----- From: "Jed Brown" <[email protected]> To: "Joon Hee Choi" <[email protected]> Cc: [email protected] Sent: Thursday, September 12, 2013 7:51:21 PM Subject: Re: [petsc-users] About MatGetOwnershipRange()
Joon Hee Choi <[email protected]> writes: >> Hi Jed, >> >> Thank you for your reply. I tried to follow your method, but I think >> my code still has something wrong because "localsize" is the same as >> global row size("I"). >Did you run in parallel? Yes. I tested my code with 2 nodes and 4 ppn(processors per node). >> Could you let me know what I am missing? My new code is as follows: >> >> PetscErrorCode SetUpMatrix(Mat *X, vector< std::tr1::tuple< PetscInt, >> PetscInt, PetscInt > >, PetscInt I, PetscInt J) >> { >> PetscInt i, j; >> PetscScalar val; >> >> MatCreate(PETSC_COMM_WORLD, X); >> MatSetType(*X, MATMPIAIJ); >> MatSetSizes(*X, PETSC_DECIDE, PETSC_DECIDE, I, J); >> >> PetscInt begin, end, localsize=PETSC_DECIDE; >> PetscSplitOwnership(PETSC_COMM_WORLD, &localsize, &I); >> MPI_Scan(&localsize, &end, 1, MPI_INT, MPI_SUM, PETSC_COMM_WORLD); >> begin = end - localsize; >> PetscPrintf(PETSC_COMM_WORLD, "Local Size: %d, begin: %d, end: %d \n", >> localsize, begin, end); >> >> ... >> } >> >> Also, tuples are the total pairs of (row dimension, column dimension, >> element), values of sparse matrix, which are read from a file. >Using files this way is a massive bottleneck that you'll have to >eliminate if you want your code to be scalable. What do you mean a massive bottleneck? Actually, I succeeded in setting up the large matrix using seqaij type and calculating products between huge matrices. However, I need to use multiple machines because I have to run loops of the product many times. And I am a novice of mpi. >> The tuples are sorted and distributed. >When you distribute, are you sure that each process really gets the >entire row, or would it be possible to cut in the middle of a row? I think each process gets the entire row. The following is the code for getting tuples from text file: ierr = PetscFOpen(PETSC_COMM_WORLD, inputFile, "r", &file); CHKERRQ(ierr); while (fscanf(file, "%d %d %d", &ii, &jj, &vv) == 3) { tups.push_back(std::tr1::make_tuple (ii-1, jj-1, vv)); if (ii > I) I = ii; if (jj > J) J = jj; } ierr = PetscFClose(PETSC_COMM_WORLD, file); CHKERRQ(ierr);
