Hello Barry,
Am 22.04.2014 18:08, schrieb Barry Smith:
On Apr 22, 2014, at 10:23 AM, Niklas Fischer <[email protected]> wrote:
I have tracked down the problem further and it basically boils down to the
following question: Is it possible to use MatSetValue(s) to set values which
are owned by other processes?
Yes it is certainly possible. You should not set a large percent of the
values on the “wrong” process but setting some is fine. The values will also be
added together if you use ADD_VALUES.
Below have you called the MatAssemblyBegin/End after setting all the values?
It certainly is AssemblyBegin first, then set values, then AssemblyEnd
CHKERRXX(MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY));
CHKERRXX(VecAssemblyBegin(b));
for(int i = 0; i < 4 * size; ++i){
CHKERRXX(VecSetValue(b, i, 1, ADD_VALUES));
}
for(int i = 0; i < 4 * size; ++i){
CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES));
}
CHKERRXX(MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY));
CHKERRXX(VecAssemblyEnd(b));
My observation, setting the values does not work, also ties in with the
solution given by the solver which is the result of
Diag [1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4]x=Constant(1,16)
Process [0]
1
1
1
1
Process [1]
0.5
0.5
0.5
0.5
Process [2]
0.333333
0.333333
0.333333
0.333333
Process [3]
0.25
0.25
0.25
0.25
If I create a matrix with
for(int i = 0; i < 4 * size; ++i){
CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES));
}
for n=m=4, on four processes, one would expect each entry to be 1 + 2 + 3 + 4 =
10, however, PETSc prints
Matrix Object: 1 MPI processes
type: mpiaij
row 0: (0, 1)
row 1: (1, 1)
row 2: (2, 1)
row 3: (3, 1)
row 4: (4, 2)
row 5: (5, 2)
row 6: (6, 2)
row 7: (7, 2)
row 8: (8, 3)
row 9: (9, 3)
row 10: (10, 3)
row 11: (11, 3)
row 12: (12, 4)
row 13: (13, 4)
row 14: (14, 4)
row 15: (15, 4)
which is exactly, what
CHKERRXX(VecGetOwnershipRange(x, &ownership_start, &ownership_end));
for(int i = ownership_start; i < ownership_end; ++i){
CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES));
}
would give us.
Kind regards,
Niklas Fischer
Am 22.04.2014 14:59, schrieb Niklas Fischer:
I should probably note that everything is fine if I run the serial version of
this (with the exact same matrix + right hand side).
PETSc KSPSolve done, residual norm: 3.13459e-13, it took 6 iterations.
Am 22.04.2014 14:12, schrieb Niklas Fischer:
Am 22.04.2014 13:57, schrieb Matthew Knepley:
On Tue, Apr 22, 2014 at 6:48 AM, Niklas Fischer <[email protected]> wrote:
Am 22.04.2014 13:08, schrieb Jed Brown:
Niklas Fischer <[email protected]> writes:
Hello,
I have attached a small test case for a problem I am experiencing. What
this dummy program does is it reads a vector and a matrix from a text
file and then solves Ax=b. The same data is available in two forms:
- everything is in one file (matops.s.0 and vops.s.0)
- the matrix and vector are split between processes (matops.0,
matops.1, vops.0, vops.1)
The serial version of the program works perfectly fine but unfortunately
errors occure, when running the parallel version:
make && mpirun -n 2 a.out matops vops
mpic++ -DPETSC_CLANGUAGE_CXX -isystem
/home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/include -isystem
/home/data/fischer/libs/petsc-3.4.3/include petsctest.cpp -Werror -Wall
-Wpedantic -std=c++11 -L
/home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib -lpetsc
/usr/bin/ld: warning: libmpi_cxx.so.0, needed by
/home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib/libpetsc.so,
may conflict with libmpi_cxx.so.1
/usr/bin/ld: warning: libmpi.so.0, needed by
/home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib/libpetsc.so,
may conflict with libmpi.so.1
librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
CMA: unable to get RDMA device list
--------------------------------------------------------------------------
[[43019,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:
Module: OpenFabrics (openib)
Host: dornroeschen.igpm.rwth-aachen.de
CMA: unable to get RDMA device list
It looks like your MPI is either broken or some of the code linked into
your application was compiled with a different MPI or different version.
Make sure you can compile and run simple MPI programs in parallel.
Hello Jed,
thank you for your inputs. Unfortunately MPI does not seem to be the issue
here. The attachment contains a simple MPI hello world program which runs
flawlessly (I will append the output to this mail) and I have not encountered
any problems with other MPI programs. My question still stands.
This is a simple error. You created the matrix A using PETSC_COMM_WORLD, but
you try to view it
using PETSC_VIEWER_STDOUT_SELF. You need to use PETSC_VIEWER_STDOUT_WORLD in
order to match.
Thanks,
Matt
Greetings,
Niklas Fischer
mpirun -np 2 ./mpitest
librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
CMA: unable to get RDMA device list
--------------------------------------------------------------------------
[[44086,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:
Module: OpenFabrics (openib)
Host: dornroeschen.igpm.rwth-aachen.de
Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
CMA: unable to get RDMA device list
Hello world from processor dornroeschen.igpm.rwth-aachen.de, rank 0 out of 2
processors
Hello world from processor dornroeschen.igpm.rwth-aachen.de, rank 1 out of 2
processors
[dornroeschen.igpm.rwth-aachen.de:128141] 1 more process has sent help message
help-mpi-btl-base.txt / btl:no-nics
[dornroeschen.igpm.rwth-aachen.de:128141] Set MCA parameter
"orte_base_help_aggregate" to 0 to see all help / error messages
Thank you, Matthew, this solves my viewing problem. Am I doing something wrong when
initializing the matrices as well? The matrix' viewing output starts with "Matrix
Object: 1 MPI processes" and the Krylov solver does not converge.
Your help is really appreciated,
Niklas Fischer
--
What most experimenters take for granted before they begin their experiments is
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
