Hi all, I'm solving a Navier Stokes system. I am having a problem whereby when I try to use asm as a preconditioner and make the number of blocks more than one per mpi proc I get an error from within parmetis (when I attempt the same with bjacobi I don't get an error). Also when I run with exactly one block per processor there are no issues. Any tips? Error message below.
Cheers, James PARMETIS ERROR: The sum of tpwgts for constraint #0 is not 1.0 PARMETIS ERROR: The sum of tpwgts for constraint #0 is not 1.0 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 76. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Error in external library! [1]PETSC ERROR: ParMETIS general error in ParMETIS_V3_PartKway! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: /home/james/libmesh-0.9.3/examples/fem_system/fem_system_ex1/.libs/lt-example-opt on a linux-opt named james-SATELLITE-C855-18D by james Fri Aug 1 10:45:36 2014 [1]PETSC ERROR: Libraries linked from /home/james/petsc-3.4.2/linux-opt/lib [1]PETSC ERROR: Configure run at Wed Jul 9 14:22:40 2014 [1]PETSC ERROR: Configure options ./configure --with-mpi=1 --with-x=0 --download-scalapack=1 --download-metis=1 --download-parmetis=1 --download-mumps=1 --download-hypre=1 --download-f-blas-lapack=1 --download-ml=1 --download-superlu_dist=1 --download-superlu=1 --download-umfpack=1 --download-ptscotch=1 --download-gamg=1 --download-pastix=1 --with-debugging=0 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: MatPartitioningApply_Parmetis() line 104 in /home/james/petsc-3.4.2/src/mat/partition/impls/pmetis/pmetis.c [0]PETSC ERROR: Error in external library! [0]PETSC ERROR: ParMETIS general error in ParMETIS_V3_PartKway! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /home/james/libmesh-0.9.3/examples/fem_system/fem_system_ex1/.libs/lt-example-opt on a linux-opt named james-SATELLITE-C855-18D by james Fri Aug 1 10:45:36 2014 [0]PETSC ERROR: Libraries linked from /home/james/petsc-3.4.2/linux-opt/lib [0]PETSC ERROR: Configure run at Wed Jul 9 14:22:40 2014 [0]PETSC ERROR: Configure options ./configure --with-mpi=1 --with-x=0 --download-scalapack=1 --download-metis=1 --download-parmetis=1 --download-mumps=1 --download-hypre=1 --download-f-blas-lapack=1 --download-ml=1 --download-superlu_dist=1 --download-superlu=1 --download-umfpack=1 --download-ptscotch=1 --download-gamg=1 --download-pastix=1 --with-debugging=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatPartitioningApply_Parmetis() line 104 in /home/james/petsc-3.4.2/src/mat/partition/impls/pmetis/pmetis.c [0]PETSC ERROR: MatPartitioningApply() line 225 in /home/james/petsc-3.4.2/src/mat/partition/partition.c [0]PETSC ERROR: PCASMCreateSubdomains() line 1210 in /home/james/petsc-3.4.2/src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: PCSetUp_ASM() line 232 in /home/james/petsc-3.4.2/src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: PCSetUp() line 890 in /home/james/petsc-3.4.2/src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 278 in /home/james/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: KSPSolve() line 399 in /home/james/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: solve() line 763 in "unknowndirectory/"src/solvers/petsc_linear_solver.C [1]PETSC ERROR: MatPartitioningApply() line 225 in /home/james/petsc-3.4.2/src/mat/partition/partition.c [1]PETSC ERROR: PCASMCreateSubdomains() line 1210 in /home/james/petsc-3.4.2/src/ksp/pc/impls/asm/asm.c [1]PETSC ERROR: PCSetUp_ASM() line 232 in /home/james/petsc-3.4.2/src/ksp/pc/impls/asm/asm.c [1]PETSC ERROR: PCSetUp() line 890 in /home/james/petsc-3.4.2/src/ksp/pc/interface/precon.c [1]PETSC ERROR: KSPSetUp() line 278 in /home/james/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: KSPSolve() line 399 in /home/james/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: solve() line 763 in "unknowndirectory/"src/solvers/petsc_linear_solver.C -------------------------------------------------------------------------- mpirun has exited due to process rank 1 with PID 5356 on node james-SATELLITE-C855-18D exiting without calling "finalize". This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported hPARMETIS ERROR: The sum of tpwgts for constraint #0 is not 1.0 PARMETIS ERROR: The sum of tpwgts for constraint #0 is not 1.0 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 76. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Error in external library! [1]PETSC ERROR: ParMETIS general error in ParMETIS_V3_PartKway! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: /home/james/libmesh-0.9.3/examples/fem_system/fem_system_ex1/.libs/lt-example-opt on a linux-opt named james-SATELLITE-C855-18D by james Fri Aug 1 10:45:36 2014 [1]PETSC ERROR: Libraries linked from /home/james/petsc-3.4.2/linux-opt/lib [1]PETSC ERROR: Configure run at Wed Jul 9 14:22:40 2014 [1]PETSC ERROR: Configure options ./configure --with-mpi=1 --with-x=0 --download-scalapack=1 --download-metis=1 --download-parmetis=1 --download-mumps=1 --download-hypre=1 --download-f-blas-lapack=1 --download-ml=1 --download-superlu_dist=1 --download-superlu=1 --download-umfpack=1 --download-ptscotch=1 --download-gamg=1 --download-pastix=1 --with-debugging=0 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: MatPartitioningApply_Parmetis() line 104 in /home/james/petsc-3.4.2/src/mat/partition/impls/pmetis/pmetis.c [0]PETSC ERROR: Error in external library! [0]PETSC ERROR: ParMETIS general error in ParMETIS_V3_PartKway! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /home/james/libmesh-0.9.3/examples/fem_system/fem_system_ex1/.libs/lt-example-opt on a linux-opt named james-SATELLITE-C855-18D by james Fri Aug 1 10:45:36 2014 [0]PETSC ERROR: Libraries linked from /home/james/petsc-3.4.2/linux-opt/lib [0]PETSC ERROR: Configure run at Wed Jul 9 14:22:40 2014 [0]PETSC ERROR: Configure options ./configure --with-mpi=1 --with-x=0 --download-scalapack=1 --download-metis=1 --download-parmetis=1 --download-mumps=1 --download-hypre=1 --download-f-blas-lapack=1 --download-ml=1 --download-superlu_dist=1 --download-superlu=1 --download-umfpack=1 --download-ptscotch=1 --download-gamg=1 --download-pastix=1 --with-debugging=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatPartitioningApply_Parmetis() line 104 in /home/james/petsc-3.4.2/src/mat/partition/impls/pmetis/pmetis.c [0]PETSC ERROR: MatPartitioningApply() line 225 in /home/james/petsc-3.4.2/src/mat/partition/partition.c [0]PETSC ERROR: PCASMCreateSubdomains() line 1210 in /home/james/petsc-3.4.2/src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: PCSetUp_ASM() line 232 in /home/james/petsc-3.4.2/src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: PCSetUp() line 890 in /home/james/petsc-3.4.2/src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 278 in /home/james/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: KSPSolve() line 399 in /home/james/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: solve() line 763 in "unknowndirectory/"src/solvers/petsc_linear_solver.C [1]PETSC ERROR: MatPartitioningApply() line 225 in /home/james/petsc-3.4.2/src/mat/partition/partition.c [1]PETSC ERROR: PCASMCreateSubdomains() line 1210 in /home/james/petsc-3.4.2/src/ksp/pc/impls/asm/asm.c [1]PETSC ERROR: PCSetUp_ASM() line 232 in /home/james/petsc-3.4.2/src/ksp/pc/impls/asm/asm.c [1]PETSC ERROR: PCSetUp() line 890 in /home/james/petsc-3.4.2/src/ksp/pc/interface/precon.c [1]PETSC ERROR: KSPSetUp() line 278 in /home/james/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: KSPSolve() line 399 in /home/james/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: solve() line 763 in "unknowndirectory/"src/solvers/petsc_linear_solver.C -------------------------------------------------------------------------- mpirun has exited due to process rank 1 with PID 5356 on node james-SATELLITE-C855-18D exiting without calling "finalize". This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- [james-SATELLITE-C855-18D:05354] 1 more process has sent help message help-mpi-api.txt / mpi-abort [james-SATELLITE-C855-18D:05354] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages ere). -------------------------------------------------------------------------- [james-SATELLITE-C855-18D:05354] 1 more process has sent help message help-mpi-api.txt / mpi-abort [james-SATELLITE-C855-18D:05354] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
