Dear Team, I ran into some issues using Petsc with Boomeramg and FieldSplit as PC on the ARCHER2 cluster.
These are my options for solving a Navier-Stokes-like system and it ran fine on other clusters. -ksp_type fgmres -ksp_pc_side right -ksp_initial_guess_nonzero 0 -ksp_lag_norm -ksp_gmres_restart 100 -ksp_monitor :block_system_ksp_its.log -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type full -pc_fieldsplit_schur_precondition selfp -fieldsplit_0_ksp_type preonly -fieldsplit_0_pc_type hypre -fieldsplit_0_pc_hypre_type boomeramg -fieldsplit_0_pc_hypre_boomeramg_max_iter 1 -fieldsplit_0_pc_hypre_boomeramg_coarsen_type HMIS -fieldsplit_0_pc_hypre_boomeramg_tol 0.0 -fieldsplit_0_pc_hypre_boomeramg_interp_type ext+i -fieldsplit_0_pc_hypre_boomeramg_strong_threshold 0.7 -fieldsplit_0_pc_hypre_boomeramg_relax_type_all l1-Gauss-Seidel -fieldsplit_1_mat_schur_complement_ainv_type blockdiag -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type sor -fieldsplit_1_ksp_max_it 30 -fieldsplit_1_ksp_rtol 1e-1 -fieldsplit_1_ksp_atol 0.0 -fieldsplit_1_ksp_pc_side right However on ARCHER2 I ended up with OOM errors and the SLURM scheduler quit my jobs. I did some digging with valgrind I found out that in PCApply when BoomerAMG is called and HYPRE is compiled with the option "--enable-persistent 1" there are a lot of calloc's on HYPRE's side that are not free'd even after the PC gets destroyed from Petsc side. This seems to be the case on the vendor-shipped Petsc on ARCHER2. Building my own Petsc and Hypre on the cluster resolves this issue with the downside of loosing around 15% of speed. I thought the developers might be interested in this bug, but I do not know who is to be adressed here, the hypre developers, the petsc developers, or the ARCHER2 admins. Best regards Elias -- Dr. Elias Karabelas Research Associate University of Graz Institute of Mathematics and Scientific Computing Heinrichstraße 36 A-8010 Graz Austria Phone: +43 316 380 8546 Email: [email protected]<mailto:[email protected]> Web: https://ccl.medunigraz.at/
