Re: [OMPI users] 1.8.6 w/ CUDA 7.0 & GDR Huge Memory Leak

2015-07-06 Thread Rolf vandeVaart
eper into this issue. Thanks for reporting. Rolf From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Stefan Paquay Sent: Wednesday, July 01, 2015 11:43 AM To: us...@open-mpi.org<mailto:us...@open-mpi.org> Subject: Re: [OMPI users] 1.8.6 w/ CUDA 7.0 & GDR Huge Memory Leak Hi

Re: [OMPI users] 1.8.6 w/ CUDA 7.0 & GDR Huge Memory Leak

2015-07-01 Thread Rolf vandeVaart
into this issue. Thanks for reporting. Rolf From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Stefan Paquay Sent: Wednesday, July 01, 2015 11:43 AM To: us...@open-mpi.org Subject: Re: [OMPI users] 1.8.6 w/ CUDA 7.0 & GDR Huge Memory Leak Hi all, Hopefully this mail gets posted in the r

Re: [OMPI users] 1.8.6 w/ CUDA 7.0 & GDR Huge Memory Leak

2015-07-01 Thread Stefan Paquay
Hi all, Hopefully this mail gets posted in the right thread... I have noticed the (I guess same) leak using OpenMPI 1.8.6 with LAMMPS, a molecular dynamics program, without any use of CUDA. I am not that familiar with how the internal memory management of LAMMPS works, but it does not appear

Re: [OMPI users] 1.8.6 w/ CUDA 7.0 & GDR Huge Memory Leak

2015-06-30 Thread Steven Eliuk
Ramps up over time, we had a bunch of locked up nodes over the weekend and have traced it back to this. Let me see if I can share more details, I will review with everyone tomorrow and get back to you, Rolf vandeVaart wrote: Hi Steven, Thanks for the report. Very

Re: [OMPI users] 1.8.6 w/ CUDA 7.0 & GDR Huge Memory Leak

2015-06-30 Thread Rolf vandeVaart
Hi Steven, Thanks for the report. Very little has changed between 1.8.5 and 1.8.6 within the CUDA-aware specific code so I am perplexed. Also interesting that you do not see the issue with 1.8.5 and CUDA 7.0. You mentioned that it is hard to share the code on this but maybe you could share

[OMPI users] 1.8.6 w/ CUDA 7.0 & GDR Huge Memory Leak

2015-06-30 Thread Steven Eliuk
Hi All, Looks like we have found a large memory leak, Very difficult to share code on this but here are some details, 1.8.5 w/ Cuda 7.0 — no memory leak 1.8.5 w/ cuda 6.5 — no memory leak 1.8.6 w/ cuda 7.0 — large memory leak 1.8.5 w/ cuda 6.5 — no memory leak mvapich2 2.1 GDR — no issue on