[petsc-users] How to use multigrid?
1. *Always* send -log_summary when asking about performance. 2. AMG setup costs more, the solve should be faster, especially for large problems. 3. 30k degrees of freedom is not large. On Sat, Nov 3, 2012 at 10:27 AM, w_ang_temp w_ang_temp at 163.com wrote: Hello, I have tried AMG, but there are some problems. I use the command: mpiexec -n 4 ./ex4f -ksp_type gmres -pc_type gamg -pc_gamg_agg_nsmooths 1 -ksp_gmres_restart 170 -ksp_rtol 1.0e-15 -ksp_converged_reason. The matrix has a size of 3. However, compared with -pc_type asm, the amg need more time:asm needs 4.9s, amg needs 13.7s. I did several tests and got the same conclusion. When it begins, the screen shows the information: [0]PCSetData_AGG bs=1 MM=7601. I do not know the meaning. And if there is some parameters that affect the performance of AMG? Besides, I want to confirm a conception. In my view, AMG itself can be a solver like gmres. It can also be used as a preconditioner like jacobi and is used by combining with other solver. Is it right? If it is right, how use AMG solver? My codes are attached. Thanks. Jim ---codes-- call MatCreate(PETSC_COMM_WORLD,A,ierr) call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr) call MatSetType(A, MATMPIAIJ,ierr) call MatSetFromOptions(A,ierr) !premalloc !find the max non-zero numbers of all rows maxnonzero=0 do 19,II=1,m !no-zero numbers of this row maxnonzeroII=NROWIN(II+1)-NROWIN(II) if (maxnonzeroIImaxnonzero) then maxnonzero=maxnonzeroII endif 19 continue nbs p; call MatMPIAIJSetPreallocation(A,maxnonzero,PETSC_NULL_INTEGER, maxnonzero,PETSC_NULL_INTEGER,ierr) call MatGetOwnershipRange(A,Istart,Iend,ierr) !set values per row do 10,II=Istart+1,Iend !no-zero numbers of this row rowNum=NROWIN(II+1)-NROWIN(II) allocate(nColPerRow(rowNum)) allocate(valuePerRow(rowNum)) kValStart=NROWIN(II)+1-1 kValEnd=NROWIN(II)+rowNum-1 !column index nColPerRow=NNZIJ(kValStart:kValEnd)-1 valuePerRow=VALUE(kValStart:kValEnd) nb sp; nRow=II-1 call MatSetValues(A,ione,nRow,rowNum,nColPerRow,valuePerRow, INSERT_VALUES,ierr) deallocate(nColPerRow) deallocate(valuePerRow) 10 continue call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,m,b,ierr) call VecSetFromOptions(b,ierr) call VecDuplicate(b,u,ierr) call VecDuplicate(b,x,ierr) call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) call KSPSetOperators(ksp,A,A,DIF FERENT_NONZERO_PATTERN,ierr) call PetscOptionsHasName(PETSC_NULL_CHARACTER,'-my_ksp_monitor', flg,ierr) if (flg) then call KSPMonitorSet(ksp,MyKSPMonitor,PETSC_NULL_OBJECT, PETSC_NULL_FUNCTION,ierr) endif call KSPSetFromOptions(ksp,ierr) call PetscOptionsHasName(PETSC_NULL_CHARACTER, '-my_ksp_convergence',flg,ierr) if (flg) then call KSPSetConvergenceTest(ksp,MyKSPConverged, PETSC_NULL_OBJECT,PETSC_NULL_FUNCTION,ierr) endif !Assing values to 'b' bTemp=Iend-Istart ioneb=1 do 12,II=Istart,Iend-1 voneb=F(II+1) call VecSetValues(b,ioneb,II,voneb,INSERT_VALUES,ierr) 12continue call VecAssemblyBegin(b,ierr) call VecAssemblyEnd(b,ierr) call KSPSolve(ksp,b,x,ierr) --codes- At 2012-11-01 22:00:28,Jed Brown jedbrown at mcs.anl.gov wrote: Yes, it's faster to understand this error message than to have mysteriously slow performance. ** Preallocation routines now automatically set MAT_NEW_NONZERO_ALLOCATION_ERR, if you intentionally preallocate less than necessary then **use MatSetOption(mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) to disable the error generation. * http://www.mcs.anl.gov/petsc/documentation/changes/33.html On Thu, Nov 1, 2012 at 8:57 AM, w_ang_temp w_ang_temp at 163.com wrote: Do you mean that the two versions have a different in this point? If I use the new version, I have to make some modifications on my codes? -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121103/c4ad41d3/attachment.html
[petsc-users] How to use multigrid?
Just pass it as a command line option. It gives profiling output in PetscFinalize(). On Sat, Nov 3, 2012 at 10:52 AM, w_ang_temp w_ang_temp at 163.com wrote: Is there something that need attention when setting up PETSc? The -log_summary is no use in my system. At 2012-11-03 23:31:52,Jed Brown jedbrown at mcs.anl.gov wrote: 1. *Always* send -log_summary when asking about performance. 2. AMG setup costs more, the solve should be faster, especially for large problems. 3. 30k degrees of freedom is not large. On Sat, Nov 3, 2012 at 10:27 AM, w_ang_temp w_ang_temp at 163.com wrote: Hello, I have tried AMG, but there are some problems. I use the command: mpiexec -n 4 ./ex4f -ksp_type gmres -pc_type gamg -pc_gamg_agg_nsmooths 1 -ksp_gmres_restart 170 -ksp_rtol 1.0e-15 -ksp_converged_reason. The matrix has a size of 3. However, compared with -pc_type asm, the amg need more time:asm needs 4.9s, amg needs 13.7s. I did several tests and got the same conclusion. When it begins, the screen shows the information: [0]PCSetData_AGG bs=1 MM=7601. I do not know the meaning. And if there is some parameters that affect the performance of AMG? Besides, I want to confirm a conception. In my view, AMG itself can be a solver like gmres. It can also be used as a preconditioner like jacobi and is used by combining with other solver. Is it right? If it is right, how use AMG solver? My codes are attached. Thanks. Jim At 2012-11-01 22:00:28,Jed Brown jedbrown at mcs.anl.gov wrote: Yes, it's faster to understand this error message than to have mysteriously slow performance. ** Preallocation routines now automatically set MAT_NEW_NONZERO_ALLOCATION_ERR, if you intentionally preallocate less than necessary then **use MatSetOption(mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) to disable the error generation. * http://www.mcs.anl.gov/petsc/documentation/changes/33.html On Thu, Nov 1, 2012 at 8:57 AM, w_ang_temp w_ang_temp at 163.com wrote: Do you mean that the two versions have a different in this point? If I use the new version, I have to make some modifications on my codes? -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121103/5a087a4a/attachment.html
[petsc-users] How to use multigrid?
/include -I/home/ubu/soft/petsc/petsc-3.3-p4/include -I/home/ubu/soft/petsc/petsc-3.3-p4/arch-linux2-c-debug/include -I/home/ubu/soft/mpich2/include - Using C linker: /home/ubu/soft/mpich2/bin/mpicc Using Fortran linker: /home/ubu/soft/mpich2/bin/mpif90 Using libraries: -Wl,-rpath,/home/ubu/soft/petsc/petsc-3.3-p4/arch-linux2-c-debug/lib -L/home/ubu/soft/petsc/petsc-3.3-p4/arch-linux2-c-debug/lib -lpetsc -lpthread -Wl,-rpath,/home/u bu/soft/petsc/petsc-3.3-p4/arch-linux2-c-debug/lib -L/home/ubu/soft/petsc/petsc-3.3-p4/arch-linux2-c-debug/lib -lflapack -lfblas -L/home/ubu/soft/mpich2/lib -L/opt/intel/composer_xe_2011_sp1.10.319/compiler/lib/ia32 -L/opt/intel/composer_xe_2011_sp1.10.319/ipp/lib/ia32 -L/opt/intel/composer_xe_2011_sp1.10.319/mkl/lib/ia32 -L/opt/intel/composer_xe_2011_sp1.10.319/tbb/lib/ia32/cc4.1.0_libc2.4_kernel2.6.16.21 -L/opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/ia32 -L/opt/intel/composer_xe_2011_sp1.9.293/mkl/lib/ia32 -L/usr/lib/gcc/i486-linux-gnu/4.4.3 -L/usr/lib/i486-linux-gnu -lmpichf90 -lifport -lifcore -lm -lm -ldl -lmpich -lopa -lmpl -lrt -lpthread -limf -lsvml -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s -ldl - time 5.724358 time 5.404338 time 5.392337 time 5.516344 At 2012-11-03 23:53:42,Jed Brown jedbrown at mcs.anl.gov wrote: Just pass it as a command line option. It gives profiling output in PetscFinalize(). On Sat, Nov 3, 2012 at 10:52 AM, w_ang_temp w_ang_temp at 163.com wrote: Is there something that need attention when setting up PETSc? The -log_summary is no use in my system. At 2012-11-03 23:31:52,Jed Brown jedbrown at mcs.anl.gov wrote: 1. *Always* send -log_summary when asking about performance. 2. AMG setup costs more, the solve should be faster, especially for large problems. 3. 30k degrees of freedom is not large. On Sat, Nov 3, 2012 at 10:27 AM, w_ang_temp w_ang_temp at 163.comwrote: Hello, I have tried AMG, but there are some problems. I use the command: mpiexec -n 4 ./ex4f -ksp_type gmres -pc_type gamg -pc_gamg_agg_nsmooths 1 -ksp_gmres_restart 170 -ksp_rtol 1.0e-15 -ksp_converged_reason. The matrix has a size of 3. However, compared with -pc_type asm, the amg need more time:asm needs 4.9s, amg needs 13.7s. I did several tests and got the same conclusion. When it begins, the screen shows the information: [0]PCSetData_AGG bs=1 MM=7601. I do not know the meaning. And if there is some parameters that affect the performance of AMG? Besides, I want to confirm a conception. In my view, AMG itself can be a solver like gmres. It can also be used as a preconditioner like jacobi and is used by combining with other solver. Is it right? If it is right, how use AMG solver? My codes are attached. Thanks. Jim At 2012-11-01 22:00:28,Jed Brown jedbrown at mcs.anl.gov wrote: Yes, it's faster to understand this error message than to have mysteriously slow performance. ** Preallocation routines now automatically set MAT_NEW_NONZERO_ALLOCATION_ERR, if you intentionally preallocate less than necessary then **use MatSetOption(mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) to disable the error generation. * http://www.mcs.anl.gov/petsc/documentation/changes/33.html On Thu, Nov 1, 2012 at 8:57 AM, w_ang_temp w_ang_temp at 163.com wrote: Do you mean that the two versions have a different in this point? If I use the new version, I have to make some modifications on my codes? -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121103/49ef8f87/attachment-0001.html
[petsc-users] How to use multigrid?
On Sat, Nov 3, 2012 at 1:17 PM, w_ang_temp w_ang_temp at 163.com wrote: At 2012-11-04 01:08:26,Jed Brown jedbrown at mcs.anl.gov wrote: 1. What kind of equation are you solving? AMG is not working well if it takes that many iterations. I just deal with the typical soil-water coupled geotechnical problems. It is a typical finite element equation. The matrix is 3X3 and ill-conditioned. We are now at the root of your problem. Solvers do not work on discretizations, they work on equations. No solver is designed for finite elements, and there is no typical finite element problem. Multigrid works best on elliptic equations with smooth coefficients. Without that, you have to do special things. I can tell from the above discussion that you have not spent a lot of time researching successful preconditioning strategies for your problem in the literature. This is always the first step to building a high performance solver. Thanks, Matt 2. * ##* * ##* * # WARNING!!!#* * ##* * # This code was compiled with a debugging option, #* * # To get timing results run ./configure#* * # using --with-debugging=no, the performance will #* * # be generally two or three times faster. #* * ##* * ##* It is true a debugging version. And I used the same version dealing with the same problem, one preconditioner is asm and the other is amg. The time with amg is about 3 times as with asm. I do not know the reason. And I also do not know the meaning of '[0]PCSetData_AGG bs=1 MM=7601'. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121103/cbe1a43c/attachment.html
[petsc-users] How to use multigrid?
On Sat, Nov 3, 2012 at 1:38 PM, w_ang_temp w_ang_temp at 163.com wrote: Hello, Matthew I just mean that the problem that I am resolving is a finite element problem. The linear system of it is true elliptic equations. I heared that AMG was an efficient solver, so I just want to have a try about AMG and find that if it is efficient. And I meant it when I said, you MUST look it up .Next time you ask us what AMG can do, please include a reference for a paper in which they are attacking this problem with it and we can help. By the way, I want to confirm a conception. In my view, AMG itself can be a solver like gmres. It can also be used as a preconditioner like jacobi and is used by combining with other solver. Is it right? If it is right, how use AMG solver? This is true of almost all KSP and PC objects. These are all jsut approximate solvers. Matt Thanks. Jim ? 2012-11-04 01:21:59?Matthew Knepley knepley at gmail.com ??? On Sat, Nov 3, 2012 at 1:17 PM, w_ang_temp w_ang_temp at 163.com wrote: At 2012-11-04 01:08:26,Jed Brown jedbrown at mcs.anl.gov wrote: 1. What kind of equation are you solving? AMG is not working well if it takes that many iterations. I just deal with the typical soil-water coupled geotechnical problems. It is a typical finite element equation. The matrix is 3X3 and ill-conditioned. We are now at the root of your problem. Solvers do not work on discretizations, they work on equations. No solver is designed for finite elements, and there is no typical finite element problem. Multigrid works best on elliptic equations with smooth coefficients. Without that, you have to do special things. I can tell from the above discussion that you have not spent a lot of time researching successful preconditioning strategies for your problem in the literature. This is always the first step to building a high performance solver. Thanks, Matt 2. * ##* * ##* * # WARNING!!!#* * ##* * # This code was compiled with a debugging option, #* * # To get timing results run ./configure#* * # using --with-debugging=no, the performance will #* * # be generally two or three times faster. #* * ##* * ##* It is true a debugging version. And I used the same version dealing with the same problem, one preconditioner is asm and the other is amg. The time with amg is about 3 times as with asm. I do not know the reason. And I also do not know the meaning of '[0]PCSetData_AGG bs=1 MM=7601'. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121103/e971e292/attachment-0001.html
[petsc-users] thread model
Hum ... That did not help. I found this -threadcomm_affinities 0,1,2,3,4,5,6,7 but that also did nothing. Is that option any relevant? Another question: the linux box I'm testing on has 2 quad-cores (8 cores/2 sockets). If I run the code with mpirun -np 8, I get about 3X which makes sense to me. However, If I run with either pthread (which seems to use all cores) or openmp (which always defaults to 1 no matter what) I get the same performance as serial. Does this mean there is something messed up with the hardware and/or how mpi/openmp/pthread is set up? On Fri, Nov 2, 2012 at 8:24 PM, Shri abhyshr at mcs.anl.gov wrote: On Nov 2, 2012, at 9:54 PM, Mohammad Mirzadeh wrote: Hi, To use the new thread model in PETSc, does it suffice to run the code with the following? -threadcomm_type openmp/pthread -threadcomm_nthreads # When I run the code with openmp, only 1 processor/core is active (looking at top). When using pthread, all cores are active. Am I missing something? OpenMP defaults to binding all threads to a single core if the cpu affinity is not specified explicitly. If you are using GNU OpenMP then you can bind the threads to specific cores using the environment variable GOMP_CPU_AFFINITY http://gcc.gnu.org/onlinedocs/libgomp/GOMP_005fCPU_005fAFFINITY.html If you are some other OpenMP implementation then check its manual to see how to set cpu affinity. Shri -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121103/3b9d5b67/attachment.html
