It looks like some form of resource exhaustion, possibly exceeding the number of entries into the mmap table. What is the value of `vm.max_map_count` on this system ? You can obtain it with `sysctl vm.max_map_count` or `cat /proc/sys/vm/max_map_count`.
George On Wed, Mar 4, 2026 at 4:25 AM Christoph Niethammer <[email protected]> wrote: > Dear all, > > We are hitting the following error when running a simple Open MPI “Hello > World” with UCX 1.16 or newer and Open MPI 5.0.x and some 4.1.5+ versions > on a single node: > > rcache.c:248 UCX ERROR mmap(size=151552) failed: Cannot allocate memory > pgtable.c:75 Fatal: Failed to allocate page table directory > *** Process received signal *** > Signal: Aborted (6) > Signal code: (-6) > > This is on CentOS 8.10, kernel 4.18, 192 GB RAM, Intel Xeon Gold 6138 > (dual-socket Skylake, 40 cores). The failure is reproducible only when > using more than 20-24 MPI ranks; fewer than 20 ranks work fine. Older UCX > versions on the same system (e.g. 1.12) do not show this issue. > > The issue also goes away if we run Open MPI with the ob1 PML (without UCX) > or disable for the UCX PML some of the TLS with UCX_TLS=^shm or UCX_TLS=^ib. > > Has anyone seen similar "mmap failed / Failed to allocate page table > directory" errors with UCX > 1.15 and Open MPI 4.1.x/5.0.x, or is aware of > known regressions or configuration pitfalls (e.g. rcache, huge pages, > memtype cache, or other UCX/Open MPI memory-related settings)? Are there > specific UCX environment variables or OMPI MCA parameters you would > recommend trying to diagnose this further? > > I can provide full ompi_info, ucx_info, build options, and more complete > logs if that is helpful. > > > Many thanks in advance for any hints or suggestions. > > > Best regards, > Christoph Niethammer > > -- > > Dr.-Ing. Christoph Niethammer > High Performance Computing Center Stuttgart (HLRS) > Nobelstrasse 19 > 70569 Stuttgart > > Tel: ++49(0)711-685-87203 > email: [email protected] > https://www.hlrs.de/people/christoph-niethammer > > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
