Diego, the mpirun command line starts 2 MPI task, but the error log mentions rank 56, so unless there is a copy/paste error, this is highly suspicious.
I invite you to check the filesystem usage on this node, and make sure there is a similar amount of available space in /tmp and /dev/shm (or other filesystem if you use a non standard $TMPDIR Cheers, Gilles On Fri, Jan 29, 2021 at 10:50 PM Diego Zuccato via users <users@lists.open-mpi.org> wrote: > > Hello all. > > I'm having a problem with a job: if it gets scheduled on a specific node > of our cluster, it fails with: > -8<-- > -------------------------------------------------------------------------- > Primary job terminated normally, but 1 process returned > a non-zero exit code. Per user-direction, the job has been aborted. > -------------------------------------------------------------------------- > [str957-mtx-10:38099] *** Process received signal *** > [str957-mtx-10:38099] Signal: Segmentation fault (11) > [str957-mtx-10:38099] Signal code: Address not mapped (1) > [str957-mtx-10:38099] Failing at address: 0x7f98cb266008 > [str957-mtx-10:38099] [ 0] > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f98ca553730] > [str957-mtx-10:38099] [ 1] > /usr/lib/x86_64-linux-gnu/pmix/lib/pmix/mca_gds_ds21.so(+0x2936)[0x7f98c8a99936] > [str957-mtx-10:38099] [ 2] > /lib/x86_64-linux-gnu/libmca_common_dstore.so.1(pmix_common_dstor_init+0x9d3)[0x7f98c8a82733] > [str957-mtx-10:38099] [ 3] > /usr/lib/x86_64-linux-gnu/pmix/lib/pmix/mca_gds_ds21.so(+0x25b4)[0x7f98c8a995b4] > [str957-mtx-10:38099] [ 4] > /lib/x86_64-linux-gnu/libpmix.so.2(pmix_gds_base_select+0x12e)[0x7f98c8bdc46e] > [str957-mtx-10:38099] [ 5] > /lib/x86_64-linux-gnu/libpmix.so.2(pmix_rte_init+0x8cd)[0x7f98c8b9488d] > [str957-mtx-10:38099] [ 6] > /lib/x86_64-linux-gnu/libpmix.so.2(PMIx_Init+0xdc)[0x7f98c8b50d7c] > [str957-mtx-10:38099] [ 7] > /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_pmix_ext2x.so(ext2x_client_init+0xc4)[0x7f98c8c3afe4] > [str957-mtx-10:38099] [ 8] > /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_ess_pmi.so(+0x2656)[0x7f98c946d656] > [str957-mtx-10:38099] [ 9] > /lib/x86_64-linux-gnu/libopen-rte.so.40(orte_init+0x29a)[0x7f98ca2c111a] > [str957-mtx-10:38099] [10] > /lib/x86_64-linux-gnu/libmpi.so.40(ompi_mpi_init+0x252)[0x7f98cae1ce62] > [str957-mtx-10:38099] [11] > /lib/x86_64-linux-gnu/libmpi.so.40(MPI_Init+0x6e)[0x7f98cae4b17e] > [str957-mtx-10:38099] [12] Arepo(+0x3940)[0x561b45905940] > [str957-mtx-10:38099] [13] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f98ca3a409b] > [str957-mtx-10:38099] [14] Arepo(+0x3d3a)[0x561b45905d3a] > [str957-mtx-10:38099] *** End of error message *** > -------------------------------------------------------------------------- > mpiexec noticed that process rank 56 with PID 37999 on node > str957-mtx-10 exited on signal 11 (Segmentation fault). > -------------------------------------------------------------------------- > slurmstepd-str957-mtx-00: error: *** JOB 12129 ON str957-mtx-00 > CANCELLED AT 2021-01-28T14:11:33 *** > -8<-- > [I cut out the other repetitions of the stack trace for brevity.] > > The command used to launch it is: > mpirun --mca mpi_leave_pinned 0 --mca oob_tcp_listen_mode listen_thread > -np 2 --map-by socket Arepo someargs > > The same job, when scheduled to run on another node, works w/o problems. > For what I could check, the nodes are configured the same (actually > installed from the same series of scripts and following the same > procedure: it was a set of 16 nodes and just one is giving troubles). > I tried with simpler MPI codes and could not reproduce the error. Other > users are using the same node w/o problems with different codes. > Packages are the same on all nodes. I already double-checked that kernel > module config is the same and memlock is unlimited. > Any hint where to look? > > Tks. > > -- > Diego Zuccato > DIFA - Dip. di Fisica e Astronomia > Servizi Informatici > Alma Mater Studiorum - Università di Bologna > V.le Berti-Pichat 6/2 - 40127 Bologna - Italy > tel.: +39 051 20 95786
