Hi Pierre,
MPI is allowed to pipeline the collective communications. This explains why
the MPI_Op takes the len of the buffers as an argument. Because your MPI_Op
ignores this length it alters data outside the temporary buffer we use for
the segment. Other versions of the MPI_Allreduce implementation might
choose not to pipeline in which case applying the MPI_Op on the entire
length of the buffer (as you manually did in your code) is correct.
George.
On Sat, Mar 13, 2021 at 4:47 AM Pierre Jolivet via users <
users@lists.open-mpi.org> wrote:
> Hello,
> The following piece of code generates Valgrind errors with OpenMPI 4.1.0,
> while it is Valgrind-clean with MPICH and OpenMPI 4.0.5.
> I don’t think I’m doing anything illegal, so could this be a regression
> introduced in 4.1.0?
>
> Thanks,
> Pierre
>
> $ /opt/openmpi-4.1.0/bin/mpicxx ompi.cxx -g -O0 -std=c++11
> $ /opt/openmpi-4.1.0/bin/mpirun -n 4 valgrind --log-file=dump.%p.log
> ./a.out
>
>
>
> ==528== Invalid read of size 2
> ==528== at 0x4011EB: main::{lambda(void*, void*, int*,
> ompi_datatype_t**)#1}::operator()(void*, void*, int*, ompi_datatype_t**)
> const (ompi.cxx:15)
> ==528== by 0x40127B: main::{lambda(void*, void*, int*,
> ompi_datatype_t**)#1}::_FUN(void*, void*, int*, ompi_datatype_t**)
> (ompi.cxx:19)
> ==528== by 0x48EFFED: ompi_coll_base_allreduce_intra_ring (in
> /opt/openmpi-4.1.0/lib/libmpi.so.40.30.0)
> ==528== by 0x77CD93A: ompi_coll_tuned_allreduce_intra_dec_fixed (in
> /opt/openmpi-4.1.0/lib/openmpi/mca_coll_tuned.so)
> ==528== by 0x48AAF00: PMPI_Allreduce (in
> /opt/openmpi-4.1.0/lib/libmpi.so.40.30.0)
> ==528== by 0x401317: main (ompi.cxx:21)
> ==528== Address 0x7139f74 is 0 bytes after a block of size 4 alloc'd
> ==528== at 0x4839809: malloc (vg_replace_malloc.c:307)
> ==528== by 0x48EF940: ompi_coll_base_allreduce_intra_ring (in
> /opt/openmpi-4.1.0/lib/libmpi.so.40.30.0)
> ==528== by 0x77CD93A: ompi_coll_tuned_allreduce_intra_dec_fixed (in
> /opt/openmpi-4.1.0/lib/openmpi/mca_coll_tuned.so)
> ==528== by 0x48AAF00: PMPI_Allreduce (in
> /opt/openmpi-4.1.0/lib/libmpi.so.40.30.0)
> ==528== by 0x401317: main (ompi.cxx:21)
> ==528==
> ==528== Invalid read of size 2
> ==528== at 0x40120E: main::{lambda(void*, void*, int*,
> ompi_datatype_t**)#1}::operator()(void*, void*, int*, ompi_datatype_t**)
> const (ompi.cxx:16)
> ==528== by 0x40127B: main::{lambda(void*, void*, int*,
> ompi_datatype_t**)#1}::_FUN(void*, void*, int*, ompi_datatype_t**)
> (ompi.cxx:19)
> ==528== by 0x48EFFED: ompi_coll_base_allreduce_intra_ring (in
> /opt/openmpi-4.1.0/lib/libmpi.so.40.30.0)
> ==528== by 0x77CD93A: ompi_coll_tuned_allreduce_intra_dec_fixed (in
> /opt/openmpi-4.1.0/lib/openmpi/mca_coll_tuned.so)
> ==528== by 0x48AAF00: PMPI_Allreduce (in
> /opt/openmpi-4.1.0/lib/libmpi.so.40.30.0)
> ==528== by 0x401317: main (ompi.cxx:21)
> ==528== Address 0x7139f76 is 2 bytes after a block of size 4 alloc'd
> ==528== at 0x4839809: malloc (vg_replace_malloc.c:307)
> ==528== by 0x48EF940: ompi_coll_base_allreduce_intra_ring (in
> /opt/openmpi-4.1.0/lib/libmpi.so.40.30.0)
> ==528== by 0x77CD93A: ompi_coll_tuned_allreduce_intra_dec_fixed (in
> /opt/openmpi-4.1.0/lib/openmpi/mca_coll_tuned.so)
> ==528== by 0x48AAF00: PMPI_Allreduce (in
> /opt/openmpi-4.1.0/lib/libmpi.so.40.30.0)
> ==528== by 0x401317: main (ompi.cxx:21)
> ==528==
> ==528== Invalid read of size 2
> ==528== at 0x401231: main::{lambda(void*, void*, int*,
> ompi_datatype_t**)#1}::operator()(void*, void*, int*, ompi_datatype_t**)
> const (ompi.cxx:18)
> ==528== by 0x40127B: main::{lambda(void*, void*, int*,
> ompi_datatype_t**)#1}::_FUN(void*, void*, int*, ompi_datatype_t**)
> (ompi.cxx:19)
> ==528== by 0x48EFFED: ompi_coll_base_allreduce_intra_ring (in
> /opt/openmpi-4.1.0/lib/libmpi.so.40.30.0)
> ==528== by 0x77CD93A: ompi_coll_tuned_allreduce_intra_dec_fixed (in
> /opt/openmpi-4.1.0/lib/openmpi/mca_coll_tuned.so)
> ==528== by 0x48AAF00: PMPI_Allreduce (in
> /opt/openmpi-4.1.0/lib/libmpi.so.40.30.0)
> ==528== by 0x401317: main (ompi.cxx:21)
> ==528== Address 0x7139f78 is 4 bytes after a block of size 4 alloc'd
> ==528== at 0x4839809: malloc (vg_replace_malloc.c:307)
> ==528== by 0x48EF940: ompi_coll_base_allreduce_intra_ring (in
> /opt/openmpi-4.1.0/lib/libmpi.so.40.30.0)
> ==528== by 0x77CD93A: ompi_coll_tuned_allreduce_intra_dec_fixed (in
> /opt/openmpi-4.1.0/lib/openmpi/mca_coll_tuned.so)
> ==528== by 0x48AAF00: PMPI_Allreduce (in
> /opt/openmpi-4.1.0/lib/libmpi.so.40.30.0)
> ==528== by 0x401317: main (ompi.cxx:21)
