Hi,
In the past, we have successfully launched large sized (finite elements)
computations using PARMetis as mesh partitioner.
It was first in 2012 with OpenMPI (v2.?) and secondly in March 2019 with
OpenMPI 3.1.2 that we succeeded.
Today, we have a bunch of nightly (small) tests running nicely and
testing all of OpenMPI (4.0.x, 4.1.x and 5.0x), MPICH-3.3.2 and IntelMPI
2021.6.
Preparing for launching the same computation we did in 2012, and even
larger ones, we compiled with bot OpenMPI 4.0.3+ucx-1.8.0 and OpenMPI
4.1.2+ucx-1.11.2 and launched computation from small to large problems
(meshes).
For small meshes, it goes fine.
But when we reach near 2^31 faces into the 3D mesh we are using and call
ParMETIS_V3_PartMeshKway, we always get a segfault with the same
backtrace pointing into ucx library:
Wed Jun 1 23:04:54
2022<stdout>:chrono::InterfaceParMetis::ParMETIS_V3_PartMeshKway::debut
VmSize: 1202304 VmRSS: 349456 VmPeak: 1211736 VmData: 500764 VmHWM:
359012 <etiq_18>
Wed Jun 1 23:07:07 2022<stdout>:Erreur : MEF++ Signal recu : 11 :
segmentation violation
Wed Jun 1 23:07:07 2022<stdout>:Erreur :
Wed Jun 1 23:07:07 2022<stdout>:------------------------------ (Début
des informations destinées aux développeurs C++)
------------------------------
Wed Jun 1 23:07:07 2022<stdout>:La pile d'appels contient 27 symboles.
Wed Jun 1 23:07:07 2022<stdout>:# 000:
reqBacktrace(std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >&) >>> probGD.opt
(probGD.opt(_Z12reqBacktraceRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x71)
[0x4119f1])
Wed Jun 1 23:07:07 2022<stdout>:# 001: attacheDebugger() >>>
probGD.opt (probGD.opt(_Z15attacheDebuggerv+0x29a) [0x41386a])
Wed Jun 1 23:07:07 2022<stdout>:# 002:
/gpfs/fs0/project/d/deteix/ericc/GIREF/lib/libgiref_opt_Util.so(traitementSignal+0x1f9f)
[0x2ab3aef0e5cf]
Wed Jun 1 23:07:07 2022<stdout>:# 003: /lib64/libc.so.6(+0x36400)
[0x2ab3bd59a400]
Wed Jun 1 23:07:07 2022<stdout>:# 004:
/scinet/niagara/software/2022a/opt/gcc-11.2.0/ucx/1.11.2/lib/libucp.so.0(ucp_dt_pack+0x123)
[0x2ab3c966e353]
Wed Jun 1 23:07:07 2022<stdout>:# 005:
/scinet/niagara/software/2022a/opt/gcc-11.2.0/ucx/1.11.2/lib/libucp.so.0(+0x536b7)
[0x2ab3c968d6b7]
Wed Jun 1 23:07:07 2022<stdout>:# 006:
/scinet/niagara/software/2022a/opt/gcc-11.2.0/ucx/1.11.2/lib/ucx/libuct_ib.so.0(uct_dc_mlx5_ep_am_bcopy+0xd7)
[0x2ab3ca712137]
Wed Jun 1 23:07:07 2022<stdout>:# 007:
/scinet/niagara/software/2022a/opt/gcc-11.2.0/ucx/1.11.2/lib/libucp.so.0(+0x52d3c)
[0x2ab3c968cd3c]
Wed Jun 1 23:07:07 2022<stdout>:# 008:
/scinet/niagara/software/2022a/opt/gcc-11.2.0/ucx/1.11.2/lib/libucp.so.0(ucp_tag_send_nbx+0x5ad)
[0x2ab3c9696dcd]
Wed Jun 1 23:07:07 2022<stdout>:# 009:
/scinet/niagara/software/2022a/opt/gcc-11.2.0/openmpi/4.1.2+ucx-1.11.2/lib/openmpi/mca_pml_ucx.so(mca_pml_ucx_send+0xf2)
[0x2ab3c922e0b2]
Wed Jun 1 23:07:07 2022<stdout>:# 010:
/scinet/niagara/software/2022a/opt/gcc-11.2.0/openmpi/4.1.2+ucx-1.11.2/lib/libmpi.so.40(ompi_coll_base_sendrecv_actual+0x92)
[0x2ab3bbca5a32]
Wed Jun 1 23:07:07 2022<stdout>:# 011:
/scinet/niagara/software/2022a/opt/gcc-11.2.0/openmpi/4.1.2+ucx-1.11.2/lib/libmpi.so.40(ompi_coll_base_alltoallv_intra_pairwise+0x141)
[0x2ab3bbcad941]
Wed Jun 1 23:07:07 2022<stdout>:# 012:
/scinet/niagara/software/2022a/opt/gcc-11.2.0/openmpi/4.1.2+ucx-1.11.2/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_alltoallv_intra_dec_fixed+0x42)
[0x2ab3d4836da2]
Wed Jun 1 23:07:07 2022<stdout>:# 013:
/scinet/niagara/software/2022a/opt/gcc-11.2.0/openmpi/4.1.2+ucx-1.11.2/lib/libmpi.so.40(PMPI_Alltoallv+0x29)
[0x2ab3bbc7bdf9]
Wed Jun 1 23:07:07 2022<stdout>:# 014:
/scinet/niagara/software/2022a/opt/gcc-11.2.0-openmpi-4.1.2+ucx-1.11.2/petsc-64bits/3.17.1/lib/libparmetis.so(libparmetis__gkMPI_Alltoallv+0x106)
[0x2ab3bb0e1c06]
Wed Jun 1 23:07:07 2022<stdout>:# 015:
/scinet/niagara/software/2022a/opt/gcc-11.2.0-openmpi-4.1.2+ucx-1.11.2/petsc-64bits/3.17.1/lib/libparmetis.so(ParMETIS_V3_Mesh2Dual+0xdd6)
[0x2ab3bb0f10b6]
Wed Jun 1 23:07:07 2022<stdout>:# 016:
/scinet/niagara/software/2022a/opt/gcc-11.2.0-openmpi-4.1.2+ucx-1.11.2/petsc-64bits/3.17.1/lib/libparmetis.so(ParMETIS_V3_PartMeshKway+0x100)
[0x2ab3bb0f1ac0]
PARMetis is compiled as part of PETSc-3.17.1 with 64bit indices. Here
are PETSc configure options:
--prefix=/scinet/niagara/software/2022a/opt/gcc-11.2.0-openmpi-4.1.2+ucx-1.11.2/petsc-64bits/3.17.1
COPTFLAGS=\"-O2 -march=native\"
CXXOPTFLAGS=\"-O2 -march=native\"
FOPTFLAGS=\"-O2 -march=native\"
--download-fftw=1
--download-hdf5=1
--download-hypre=1
--download-metis=1
--download-mumps=1
--download-parmetis=1
--download-plapack=1
--download-prometheus=1
--download-ptscotch=1
--download-scotch=1
--download-sprng=1
--download-superlu_dist=1
--download-triangle=1
--with-avx512-kernels=1
--with-blaslapack-dir=/scinet/intel/oneapi/2021u4/mkl/2021.4.0
--with-cc=mpicc
--with-cxx=mpicxx
--with-cxx-dialect=C++11
--with-debugging=0
--with-fc=mpifort
--with-mkl_pardiso-dir=/scinet/intel/oneapi/2021u4/mkl/2021.4.0
--with-scalapack=1
--with-scalapack-lib=\"[/scinet/intel/oneapi/2021u4/mkl/2021.4.0/lib/intel64/libmkl_scalapack_lp64.so,/scinet/intel/oneapi/2021u4/mkl/2021.4.0/lib/intel64/libmkl_blacs_openmpi_lp64.so]\"
--with-x=0
--with-64-bit-indices=1
--with-memalign=64
and OpenMPI configure options:
'--prefix=/scinet/niagara/software/2022a/opt/gcc-11.2.0/openmpi/4.1.2+ucx-1.11.2'
'--enable-mpi-cxx'
'--enable-mpi1-compatibility'
'--with-hwloc=internal'
'--with-knem=/opt/knem-1.1.3.90mlnx1'
'--with-libevent=internal'
'--with-platform=contrib/platform/mellanox/optimized'
'--with-pmix=internal'
'--with-slurm=/opt/slurm'
'--with-ucx=/scinet/niagara/software/2022a/opt/gcc-11.2.0/ucx/1.11.2'
I am then wondering:
1) Is UCX library considered "stable" for production use with very large
sized problems ?
2) Is there a way to "bypass" UCX at runtime?
3) Any idea for debugging this?
Of course, I do not yet have a "minimum reproducer" that bugs, since it
happens only on "large" problems, but I think I could export the data
for a 512 processes reproducer with PARMetis call only...
Thanks for helping,
Eric
--
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42
