Your GPUs are in process exclusive mode which makes it impossible for multiple ranks to use a GPU; see nvidia-smi --compute-mode option.
-- Szilárd On Fri, Apr 6, 2018 at 10:14 PM, Hollingsworth, Bobby <louishollingswo...@g.harvard.edu> wrote: > Hello all, > > I'm tuning MDrun on a node with 24 intel skylake cores (2X12) and 2 V100 > GPUs. MPI-enabled (no thread-mpi) Gromacs 2018 ("2018.0") is compiled with > GCC, CUDA, OpenMPI, and OpenBLAS. I am trying to assign the two GPUs to > four ranks. My run command is: > > mpirun -np 4 mdrun_mpi_s_g -ntomp 6 -pme cpu -nb gpu -gputasks 0011 -deffnm > test_2018 > > However, I'm getting the error: > > Mapping of GPU IDs to the 4 GPU tasks in the 4 ranks on this node: > PP:0,PP:0,PP:1,PP:1 > > NOTE: You assigned the same GPU ID(s) to multiple ranks, which is a good > idea if you have measured the performance of alternatives. > > ------------------------------------------------------- > Program: mdrun_mpi_s_g, version 2018 > Source file: src/gromacs/gpu_utils/gpu_utils.cu (line 127) > MPI rank: 1 (out of 4) > > Fatal error: > cudaFuncGetAttributes failed: all CUDA-capable devices are busy or > unavailable > > > The launch configuration works with -np 2 and -ntomp 12. Presumably, there > is an issue with GPUs being split across ranks. Any advice here? Thanks! > > Best, > Bobby > -- > Louis "Bobby" Hollingsworth > Ph.D. Student, Biological and Biomedical Sciences, Harvard University > B.S. Chemical Engineering, B.S. Biochemistry, B.A. Chemistry, Virginia Tech > Honors College '17 > <http://www.linkedin.com/pub/louis-hollingsworth/77/aaa/a47> > -- > Gromacs Users mailing list > > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > * For (un)subscribe requests visit > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a > mail to gmx-users-requ...@gromacs.org. -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org.