There have been two reports on the user list about memory leaks. I have reproduced this leak with LAMMPS. Note that this has nothing to do with CUDA-aware features. The steps that Stefan has provided make it easy to reproduce.
Here are some more specific steps to reproduce derived from Stefan. 1. clone LAMMPS (git clone git://git.lammps.org/lammps-ro.git<http://git.lammps.org/lammps-ro.git> lammps) 2. cd src/, compile with openMPI 1.8.6. To do this, set your path to Open MPI and type "make mpi" 3. run the example listed in lammps/examples/melt. To do this, first copy "lmp_mpi" from the src directory into the melt directory. Then you need to modify the in.melt file so that it will run for a while. Change "run 25" to "run 250000" 4. you can run by mpirun -np 2 lmp_mpi < in.melt For reference, here is both 1.8.5 and 1.8.6 memory consumption. 1.8.5 stays very stable where 1.8.6 almost triples after 6 minutes of running. Open MPI 1.8.5 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 59.0 0.0 329672 14584 pts/16 Rl 16:24 0:00 ./lmp_mpi_185_nocuda 32341 26908 60.0 0.0 329672 14676 pts/16 Rl 16:24 0:00 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 98.3 0.0 329672 14932 pts/16 Rl 16:24 0:30 ./lmp_mpi_185_nocuda 32341 26908 98.5 0.0 329672 14932 pts/16 Rl 16:24 0:30 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 98.9 0.0 329672 14960 pts/16 Rl 16:24 1:00 ./lmp_mpi_185_nocuda 32341 26908 99.1 0.0 329672 14952 pts/16 Rl 16:24 1:00 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 99.1 0.0 329672 14960 pts/16 Rl 16:24 1:30 ./lmp_mpi_185_nocuda 32341 26908 99.3 0.0 329672 14952 pts/16 Rl 16:24 1:30 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 99.2 0.0 329672 14960 pts/16 Rl 16:24 2:00 ./lmp_mpi_185_nocuda 32341 26908 99.4 0.0 329672 14952 pts/16 Rl 16:24 2:00 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 99.3 0.0 329672 14960 pts/16 Rl 16:24 2:30 ./lmp_mpi_185_nocuda 32341 26908 99.5 0.0 329672 14952 pts/16 Rl 16:24 2:30 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 99.4 0.0 329672 14960 pts/16 Rl 16:24 2:59 ./lmp_mpi_185_nocuda 32341 26908 99.5 0.0 329672 14952 pts/16 Rl 16:24 3:00 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 99.4 0.0 329672 14960 pts/16 Rl 16:24 3:29 ./lmp_mpi_185_nocuda 32341 26908 99.6 0.0 329672 14956 pts/16 Rl 16:24 3:30 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 99.4 0.0 329672 14960 pts/16 Rl 16:24 3:59 ./lmp_mpi_185_nocuda 32341 26908 99.6 0.0 329672 14956 pts/16 Rl 16:24 4:00 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 99.4 0.0 329672 14960 pts/16 Rl 16:24 4:29 ./lmp_mpi_185_nocuda 32341 26908 99.6 0.0 329672 14956 pts/16 Rl 16:24 4:30 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 99.5 0.0 329672 14960 pts/16 Rl 16:24 4:59 ./lmp_mpi_185_nocuda 32341 26908 99.6 0.0 329672 14956 pts/16 Rl 16:24 5:00 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 99.5 0.0 329672 14960 pts/16 Rl 16:24 5:29 ./lmp_mpi_185_nocuda 32341 26908 99.6 0.0 329672 14956 pts/16 Rl 16:24 5:29 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26907 99.5 0.0 329672 14960 pts/16 Rl 16:24 5:59 ./lmp_mpi_185_nocuda 32341 26908 99.6 0.0 329672 14956 pts/16 Rl 16:24 5:59 ./lmp_mpi_185_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Open MPI 1.8.6 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 0.0 0.0 330288 15368 pts/16 Rl 16:10 0:00 ./lmp_mpi_186_nocuda 32341 26756 0.0 0.0 330284 15376 pts/16 Rl 16:10 0:00 ./lmp_mpi_186_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 100 0.0 409856 94976 pts/16 Rl 16:10 0:30 ./lmp_mpi_186_nocuda 32341 26756 100 0.0 409848 94904 pts/16 Rl 16:10 0:30 ./lmp_mpi_186_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 100 0.1 489292 174320 pts/16 Rl 16:10 1:00 ./lmp_mpi_186_nocuda 32341 26756 100 0.1 489288 174536 pts/16 Rl 16:10 1:00 ./lmp_mpi_186_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 99.9 0.1 568860 253928 pts/16 Rl 16:10 1:29 ./lmp_mpi_186_nocuda 32341 26756 100 0.1 568984 254168 pts/16 Rl 16:10 1:30 ./lmp_mpi_186_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 99.9 0.2 648620 333800 pts/16 Rl 16:10 1:59 ./lmp_mpi_186_nocuda 32341 26756 100 0.2 648616 333868 pts/16 Rl 16:10 2:00 ./lmp_mpi_186_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 99.8 0.3 728444 413516 pts/16 Rl 16:10 2:29 ./lmp_mpi_186_nocuda 32341 26756 100 0.3 728376 413800 pts/16 Rl 16:10 2:30 ./lmp_mpi_186_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 99.8 0.3 808332 493388 pts/16 Rl 16:10 2:59 ./lmp_mpi_186_nocuda 32341 26756 99.9 0.3 808328 493432 pts/16 Rl 16:10 2:59 ./lmp_mpi_186_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 99.7 0.4 888156 573260 pts/16 Rl 16:10 3:29 ./lmp_mpi_186_nocuda 32341 26756 99.9 0.4 888088 573328 pts/16 Rl 16:10 3:29 ./lmp_mpi_186_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 99.7 0.4 968108 653396 pts/16 Rl 16:10 3:59 ./lmp_mpi_186_nocuda 32341 26756 99.9 0.4 968232 653488 pts/16 Rl 16:10 3:59 ./lmp_mpi_186_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 99.7 0.5 1048252 733268 pts/16 Rl 16:10 4:29 ./lmp_mpi_186_nocuda 32341 26756 99.9 0.5 1048248 733384 pts/16 Rl 16:10 4:29 ./lmp_mpi_186_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 99.7 0.6 1128396 813404 pts/16 Rl 16:10 4:59 ./lmp_mpi_186_nocuda 32341 26756 99.9 0.6 1128328 813544 pts/16 Rl 16:10 4:59 ./lmp_mpi_186_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 99.7 0.6 1208736 893804 pts/16 Rl 16:10 5:29 ./lmp_mpi_186_nocuda 32341 26756 99.9 0.6 1208668 893968 pts/16 Rl 16:10 5:29 ./lmp_mpi_186_nocuda USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 32341 26755 99.7 0.7 1288880 973940 pts/16 Rl 16:10 5:59 ./lmp_mpi_186_nocuda 32341 26756 99.9 0.7 1288812 974128 pts/16 Rl 16:10 5:59 ./lmp_mpi_186_nocuda ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
