Thanks, Gaetan, for your suggestions and for running the code…Now I know for sure there is something wrong with my installation! Cheers Rodrigo
From: Gaetan Kenway [mailto:[email protected]] Sent: Wednesday, April 12, 2017 12:20 PM To: Rodrigo Felicio Cc: [email protected] Subject: Re: [petsc-users] how to use petsc4py with mpi subcommunicators? Maybe try doing pComm=MPI.COMM_WORLD instead of the PETSc.COMM_WORLD. I know it shouldn't matter, but it's worth a shot. Also then you won't need the tompi4py() i guess. Gaetan On Wed, Apr 12, 2017 at 10:17 AM, Gaetan Kenway <[email protected]<mailto:[email protected]>> wrote: Hi Rodrigo I just ran your example on Nasa's Pleiades system. Here's what I got: PBS r459i4n11:~> time mpiexec -n 5 python3.5 another_split_ex.py number of subcomms = 2.5 petsc rank=2, petsc size=5 sub rank 1/3, color:0 petsc rank=4, petsc size=5 sub rank 2/3, color:0 petsc rank=0, petsc size=5 sub rank 0/3, color:0 KSP Object: 2 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: 2 MPI processes type: none PC has not been set up so information may be incomplete linear system matrix = precond matrix: Mat Object: 2 MPI processes type: mpidense rows=100, cols=100 total: nonzeros=10000, allocated nonzeros=10000 total number of mallocs used during MatSetValues calls =0 petsc rank=1, petsc size=5 sub rank 0/2, color:1 creating A in subcomm 1= 2, 0 petsc rank=3, petsc size=5 sub rank 1/2, color:1 creating A in subcomm 1= 2, 1 real 0m1.236s user 0m0.088s sys 0m0.008s So everything looks like it went through fine. I know this doesn't help you directly, but we can confirm at least the python code itself is fine. Gaetan On Wed, Apr 12, 2017 at 10:10 AM, Rodrigo Felicio <[email protected]<mailto:[email protected]>> wrote: Going over my older codes I found out that I have already tried the approach of splitting PETSc.COMM_WORLD, but whenever I try to create a matrix using a subcommuicator, the program fails. For example, executing the following python code attached to this msg, I get the following output time mpirun -n 5 python another_split_ex.py petsc rank=2, petsc size=5 petsc rank=3, petsc size=5 petsc rank=0, petsc size=5 petsc rank=1, petsc size=5 petsc rank=4, petsc size=5 number of subcomms = 2 sub rank 0/3, color:0 sub rank 0/2, color:1 sub rank 1/3, color:0 sub rank 1/2, color:1 sub rank 2/3, color:0 creating A in subcomm 1= 2, 1 creating A in subcomm 1= 2, 0 Traceback (most recent call last): File "another_split_ex.py", line 43, in <module> Traceback (most recent call last): File "another_split_ex.py", line 43, in <module> A = PETSc.Mat().createDense([n,n], comm=subcomm) File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense (src/petsc4py.PETSc.c:113792) A = PETSc.Mat().createDense([n,n], comm=subcomm) File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense (src/petsc4py.PETSc.c:113792) File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create (src/petsc4py.PETSc.c:25274) File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create (src/petsc4py.PETSc.c:25274) File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout (src/petsc4py.PETSc.c:13666) File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout (src/petsc4py.PETSc.c:13666) petsc4py.PETSc.Error: petsc4py.PETSc.Errorerror code 608517 [1] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/psplit.c : error code 134826245 [3] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/psplit.c Checking the traceback, all I can say is that when the subcommunicator object reaches psplit.c code it gets somehow corrupted, because PetscSplitOwnership() fails to retrieve the size of the subcommunicator ... :-( regards Rodrigo ________________________________ This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original. ________________________________ This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original. ________________________________ This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.
