Call BOTH the AssemblyBegin() and End AFTER you have called the set values routines.
Barry Yes, you could argue it is confusing terminology. On Apr 22, 2014, at 11:32 AM, Niklas Fischer <[email protected]> wrote: > Hello Barry, > > Am 22.04.2014 18:08, schrieb Barry Smith: >> On Apr 22, 2014, at 10:23 AM, Niklas Fischer <[email protected]> wrote: >> >>> I have tracked down the problem further and it basically boils down to the >>> following question: Is it possible to use MatSetValue(s) to set values >>> which are owned by other processes? >> Yes it is certainly possible. You should not set a large percent of the >> values on the “wrong” process but setting some is fine. The values will also >> be added together if you use ADD_VALUES. >> >> Below have you called the MatAssemblyBegin/End after setting all the >> values? > It certainly is AssemblyBegin first, then set values, then AssemblyEnd > > CHKERRXX(MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY)); > CHKERRXX(VecAssemblyBegin(b)); > > > for(int i = 0; i < 4 * size; ++i){ > CHKERRXX(VecSetValue(b, i, 1, ADD_VALUES)); > } > > for(int i = 0; i < 4 * size; ++i){ > CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES)); > } > > CHKERRXX(MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY)); > CHKERRXX(VecAssemblyEnd(b)); > > My observation, setting the values does not work, also ties in with the > solution given by the solver which is the result of > > Diag [1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4]x=Constant(1,16) > > Process [0] > 1 > 1 > 1 > 1 > Process [1] > 0.5 > 0.5 > 0.5 > 0.5 > Process [2] > 0.333333 > 0.333333 > 0.333333 > 0.333333 > Process [3] > 0.25 > 0.25 > 0.25 > 0.25 >> >>> If I create a matrix with >>> >>> for(int i = 0; i < 4 * size; ++i){ >>> CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES)); >>> } >>> >>> for n=m=4, on four processes, one would expect each entry to be 1 + 2 + 3 + >>> 4 = 10, however, PETSc prints >>> Matrix Object: 1 MPI processes >>> type: mpiaij >>> row 0: (0, 1) >>> row 1: (1, 1) >>> row 2: (2, 1) >>> row 3: (3, 1) >>> row 4: (4, 2) >>> row 5: (5, 2) >>> row 6: (6, 2) >>> row 7: (7, 2) >>> row 8: (8, 3) >>> row 9: (9, 3) >>> row 10: (10, 3) >>> row 11: (11, 3) >>> row 12: (12, 4) >>> row 13: (13, 4) >>> row 14: (14, 4) >>> row 15: (15, 4) >>> which is exactly, what >>> >>> CHKERRXX(VecGetOwnershipRange(x, &ownership_start, &ownership_end)); >>> for(int i = ownership_start; i < ownership_end; ++i){ >>> CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES)); >>> } >>> would give us. >>> >>> Kind regards, >>> Niklas Fischer >>> >>> Am 22.04.2014 14:59, schrieb Niklas Fischer: >>>> I should probably note that everything is fine if I run the serial version >>>> of this (with the exact same matrix + right hand side). >>>> >>>> PETSc KSPSolve done, residual norm: 3.13459e-13, it took 6 iterations. >>>> >>>> Am 22.04.2014 14:12, schrieb Niklas Fischer: >>>>> Am 22.04.2014 13:57, schrieb Matthew Knepley: >>>>>> On Tue, Apr 22, 2014 at 6:48 AM, Niklas Fischer <[email protected]> >>>>>> wrote: >>>>>> Am 22.04.2014 13:08, schrieb Jed Brown: >>>>>> Niklas Fischer <[email protected]> writes: >>>>>> >>>>>> Hello, >>>>>> >>>>>> I have attached a small test case for a problem I am experiencing. What >>>>>> this dummy program does is it reads a vector and a matrix from a text >>>>>> file and then solves Ax=b. The same data is available in two forms: >>>>>> - everything is in one file (matops.s.0 and vops.s.0) >>>>>> - the matrix and vector are split between processes (matops.0, >>>>>> matops.1, vops.0, vops.1) >>>>>> >>>>>> The serial version of the program works perfectly fine but unfortunately >>>>>> errors occure, when running the parallel version: >>>>>> >>>>>> make && mpirun -n 2 a.out matops vops >>>>>> >>>>>> mpic++ -DPETSC_CLANGUAGE_CXX -isystem >>>>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/include -isystem >>>>>> /home/data/fischer/libs/petsc-3.4.3/include petsctest.cpp -Werror -Wall >>>>>> -Wpedantic -std=c++11 -L >>>>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib -lpetsc >>>>>> /usr/bin/ld: warning: libmpi_cxx.so.0, needed by >>>>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib/libpetsc.so, >>>>>> may conflict with libmpi_cxx.so.1 >>>>>> /usr/bin/ld: warning: libmpi.so.0, needed by >>>>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib/libpetsc.so, >>>>>> may conflict with libmpi.so.1 >>>>>> librdmacm: couldn't read ABI version. >>>>>> librdmacm: assuming: 4 >>>>>> CMA: unable to get RDMA device list >>>>>> -------------------------------------------------------------------------- >>>>>> [[43019,1],0]: A high-performance Open MPI point-to-point messaging >>>>>> module >>>>>> was unable to find any relevant network interfaces: >>>>>> >>>>>> Module: OpenFabrics (openib) >>>>>> Host: dornroeschen.igpm.rwth-aachen.de >>>>>> CMA: unable to get RDMA device list >>>>>> It looks like your MPI is either broken or some of the code linked into >>>>>> your application was compiled with a different MPI or different version. >>>>>> Make sure you can compile and run simple MPI programs in parallel. >>>>>> Hello Jed, >>>>>> >>>>>> thank you for your inputs. Unfortunately MPI does not seem to be the >>>>>> issue here. The attachment contains a simple MPI hello world program >>>>>> which runs flawlessly (I will append the output to this mail) and I have >>>>>> not encountered any problems with other MPI programs. My question still >>>>>> stands. >>>>>> >>>>>> This is a simple error. You created the matrix A using PETSC_COMM_WORLD, >>>>>> but you try to view it >>>>>> using PETSC_VIEWER_STDOUT_SELF. You need to use >>>>>> PETSC_VIEWER_STDOUT_WORLD in >>>>>> order to match. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> Greetings, >>>>>> Niklas Fischer >>>>>> >>>>>> mpirun -np 2 ./mpitest >>>>>> >>>>>> librdmacm: couldn't read ABI version. >>>>>> librdmacm: assuming: 4 >>>>>> CMA: unable to get RDMA device list >>>>>> -------------------------------------------------------------------------- >>>>>> [[44086,1],0]: A high-performance Open MPI point-to-point messaging >>>>>> module >>>>>> was unable to find any relevant network interfaces: >>>>>> >>>>>> Module: OpenFabrics (openib) >>>>>> Host: dornroeschen.igpm.rwth-aachen.de >>>>>> >>>>>> Another transport will be used instead, although this may result in >>>>>> lower performance. >>>>>> -------------------------------------------------------------------------- >>>>>> librdmacm: couldn't read ABI version. >>>>>> librdmacm: assuming: 4 >>>>>> CMA: unable to get RDMA device list >>>>>> Hello world from processor dornroeschen.igpm.rwth-aachen.de, rank 0 out >>>>>> of 2 processors >>>>>> Hello world from processor dornroeschen.igpm.rwth-aachen.de, rank 1 out >>>>>> of 2 processors >>>>>> [dornroeschen.igpm.rwth-aachen.de:128141] 1 more process has sent help >>>>>> message help-mpi-btl-base.txt / btl:no-nics >>>>>> [dornroeschen.igpm.rwth-aachen.de:128141] Set MCA parameter >>>>>> "orte_base_help_aggregate" to 0 to see all help / error messages >>>>>> >>>>> Thank you, Matthew, this solves my viewing problem. Am I doing something >>>>> wrong when initializing the matrices as well? The matrix' viewing output >>>>> starts with "Matrix Object: 1 MPI processes" and the Krylov solver does >>>>> not converge. >>>>> >>>>> Your help is really appreciated, >>>>> Niklas Fischer >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which >>>>>> their experiments lead. >>>>>> -- Norbert Wiener >
