Maybe try doing pComm=MPI.COMM_WORLD instead of the PETSc.COMM_WORLD. I know it shouldn't matter, but it's worth a shot. Also then you won't need the tompi4py() i guess.
Gaetan On Wed, Apr 12, 2017 at 10:17 AM, Gaetan Kenway <[email protected]> wrote: > Hi Rodrigo > > I just ran your example on Nasa's Pleiades system. Here's what I got: > > PBS r459i4n11:~> time mpiexec -n 5 python3.5 another_split_ex.py > number of subcomms = 2.5 > petsc rank=2, petsc size=5 > sub rank 1/3, color:0 > petsc rank=4, petsc size=5 > sub rank 2/3, color:0 > petsc rank=0, petsc size=5 > sub rank 0/3, color:0 > KSP Object: 2 MPI processes > type: cg > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using DEFAULT norm type for convergence test > PC Object: 2 MPI processes > type: none > PC has not been set up so information may be incomplete > linear system matrix = precond matrix: > Mat Object: 2 MPI processes > type: mpidense > rows=100, cols=100 > total: nonzeros=10000, allocated nonzeros=10000 > total number of mallocs used during MatSetValues calls =0 > petsc rank=1, petsc size=5 > sub rank 0/2, color:1 > creating A in subcomm 1= 2, 0 > petsc rank=3, petsc size=5 > sub rank 1/2, color:1 > creating A in subcomm 1= 2, 1 > > real 0m1.236s > user 0m0.088s > sys 0m0.008s > > So everything looks like it went through fine. I know this doesn't help > you directly, but we can confirm at least the python code itself is fine. > > Gaetan > > On Wed, Apr 12, 2017 at 10:10 AM, Rodrigo Felicio < > [email protected]> wrote: > >> Going over my older codes I found out that I have already tried the >> approach of splitting PETSc.COMM_WORLD, but whenever I try to create a >> matrix using a subcommuicator, the program fails. For example, executing >> the following python code attached to this msg, I get the following output >> >> time mpirun -n 5 python another_split_ex.py >> petsc rank=2, petsc size=5 >> petsc rank=3, petsc size=5 >> petsc rank=0, petsc size=5 >> petsc rank=1, petsc size=5 >> petsc rank=4, petsc size=5 >> number of subcomms = 2 >> sub rank 0/3, color:0 >> sub rank 0/2, color:1 >> sub rank 1/3, color:0 >> sub rank 1/2, color:1 >> sub rank 2/3, color:0 >> creating A in subcomm 1= 2, 1 >> creating A in subcomm 1= 2, 0 >> Traceback (most recent call last): >> File "another_split_ex.py", line 43, in <module> >> Traceback (most recent call last): >> File "another_split_ex.py", line 43, in <module> >> A = PETSc.Mat().createDense([n,n], comm=subcomm) >> File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense >> (src/petsc4py.PETSc.c:113792) >> A = PETSc.Mat().createDense([n,n], comm=subcomm) >> File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense >> (src/petsc4py.PETSc.c:113792) >> File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create >> (src/petsc4py.PETSc.c:25274) >> File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create >> (src/petsc4py.PETSc.c:25274) >> File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout >> (src/petsc4py.PETSc.c:13666) >> File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout >> (src/petsc4py.PETSc.c:13666) >> petsc4py.PETSc.Error: petsc4py.PETSc.Errorerror code 608517 >> [1] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/ >> psplit.c >> : error code 134826245 >> [3] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/ >> psplit.c >> >> >> Checking the traceback, all I can say is that when the subcommunicator >> object reaches psplit.c code it gets somehow corrupted, because >> PetscSplitOwnership() fails to retrieve the size of the subcommunicator ... >> :-( >> >> regards >> Rodrigo >> >> ________________________________ >> >> >> This email and any files transmitted with it are confidential and are >> intended solely for the use of the individual or entity to whom they are >> addressed. If you are not the original recipient or the person responsible >> for delivering the email to the intended recipient, be advised that you >> have received this email in error, and that any use, dissemination, >> forwarding, printing, or copying of this email is strictly prohibited. If >> you received this email in error, please immediately notify the sender and >> delete the original. >> >> >> ________________________________ >> >> >> This email and any files transmitted with it are confidential and are >> intended solely for the use of the individual or entity to whom they are >> addressed. If you are not the original recipient or the person responsible >> for delivering the email to the intended recipient, be advised that you >> have received this email in error, and that any use, dissemination, >> forwarding, printing, or copying of this email is strictly prohibited. If >> you received this email in error, please immediately notify the sender and >> delete the original. >> >> >
