Going over my older codes I found out that I have already tried the approach of
splitting PETSc.COMM_WORLD, but whenever I try to create a matrix using a
subcommuicator, the program fails. For example, executing the following python
code attached to this msg, I get the following output
time mpirun -n 5 python another_split_ex.py
petsc rank=2, petsc size=5
petsc rank=3, petsc size=5
petsc rank=0, petsc size=5
petsc rank=1, petsc size=5
petsc rank=4, petsc size=5
number of subcomms = 2
sub rank 0/3, color:0
sub rank 0/2, color:1
sub rank 1/3, color:0
sub rank 1/2, color:1
sub rank 2/3, color:0
creating A in subcomm 1= 2, 1
creating A in subcomm 1= 2, 0
Traceback (most recent call last):
File "another_split_ex.py", line 43, in <module>
Traceback (most recent call last):
File "another_split_ex.py", line 43, in <module>
A = PETSc.Mat().createDense([n,n], comm=subcomm)
File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense
(src/petsc4py.PETSc.c:113792)
A = PETSc.Mat().createDense([n,n], comm=subcomm)
File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense
(src/petsc4py.PETSc.c:113792)
File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create
(src/petsc4py.PETSc.c:25274)
File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create
(src/petsc4py.PETSc.c:25274)
File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout
(src/petsc4py.PETSc.c:13666)
File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout
(src/petsc4py.PETSc.c:13666)
petsc4py.PETSc.Error: petsc4py.PETSc.Errorerror code 608517
[1] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/psplit.c
: error code 134826245
[3] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/psplit.c
Checking the traceback, all I can say is that when the subcommunicator object
reaches psplit.c code it gets somehow corrupted, because PetscSplitOwnership()
fails to retrieve the size of the subcommunicator ... :-(
regards
Rodrigo
________________________________
This email and any files transmitted with it are confidential and are intended
solely for the use of the individual or entity to whom they are addressed. If
you are not the original recipient or the person responsible for delivering the
email to the intended recipient, be advised that you have received this email
in error, and that any use, dissemination, forwarding, printing, or copying of
this email is strictly prohibited. If you received this email in error, please
immediately notify the sender and delete the original.
________________________________
This email and any files transmitted with it are confidential and are intended
solely for the use of the individual or entity to whom they are addressed. If
you are not the original recipient or the person responsible for delivering the
email to the intended recipient, be advised that you have received this email
in error, and that any use, dissemination, forwarding, printing, or copying of
this email is strictly prohibited. If you received this email in error, please
immediately notify the sender and delete the original.
import petsc4py, sys
petsc4py.init(sys.argv)
from petsc4py import PETSc
import mpi4py.rc
mpi4py.rc.finalize=False
#from mpi4py import MPI
import numpy as np
pComm = PETSc.COMM_WORLD
pRank = pComm.getRank()
pSize = pComm.Get_size()
#PETSc.Sys.Print('petsc rank={}, petsc size={}'.format(pRank, pSize))
print('petsc rank={}, petsc size={}'.format(pRank, pSize))
# break into communicators
NumProcsPerSubComm = 2
color = pRank % NumProcsPerSubComm
NumSubComms = pSize/NumProcsPerSubComm
#subcomm = PETSc.Comm(MPI.COMM_WORLD.Split(color))
#subcomm = MPI.COMM_WORLD.Split(color)
subcomm = pComm.tompi4py().Split(color)
subRank = subcomm.Get_rank()
subSize = subcomm.Get_size()
PETSc.Sys.Print('number of subcomms = {}'.format(NumSubComms))
for i in range(pSize):
pComm.barrier()
#PETSc.Sys.Print('sub rank {}, psub size{}'.format(pRank, pSize))
if (pComm.rank == i):
print('sub rank {}/{}, color:{}'.format(subRank, subSize, color))
n =100
a = np.random.randn(n, n)
asym = a + a.T
# if subcomm != MPI.COMM_NULL:
if color == 1:
print('creating A in subcomm {}= {}, {}'.format(color, subSize, subRank))
A = PETSc.Mat().createDense([n,n], comm=subcomm)
A.setUp()
row1, row2 = A.getOwnershipRange()
PETSc.Sys.Print('rows={},{}'.format(row1, row2))
for ip in range(row1, row2):
for k in range(n):
A.setValues(ip, k, asym[ip, k])
A.assemble()
ksp = PETSc.KSP()
ksp.create(subcomm)
ksp.setType('cg')
ksp.getPC().setType('none')
x, b = A.getVecs()
x.set(0)
b.set(1)
ksp.setOperators(A)
ksp.setFromOptions()
ksp.view()
ksp.solve(b, x)
PETSc.Sys.Print('||x|| = {}'.format(x.norm()))
