Thank you Mark for the suggestion. I will convey the message to our system admin. to solve the issue. thanks again for the valuable inputs.
On Thu, Apr 7, 2016 at 7:02 PM, Mark Abraham <[email protected]> wrote: > Hi, > > As you can see in that log file, you are only getting a single rank, so > that's all that GROMACS can use. You need to troubleshoot your use of PBS > and mpiexec so that you get four ranks placed two on each node. We can't > read your cluster's docs or talk to your sysadmins :-) > > (And while you're at it, get them to compile with something less > prehistoric than gcc 4.4. For many runs, you're likely to be bound by CPU > performance with two K40s per node, and more recent compilers will be > better...) > > Mark > > On Thu, Apr 7, 2016 at 3:16 PM Venkat Reddy <[email protected]> wrote: > > > Dear Szilárd Páll, > > Thanks for the response. > > > > My PBS script to launch the run is: > > > > #! /bin/bash > > #PBS -l cput=5000:00:00 > > #PBS -l select=2:ncpus=16:ngpus=2 > > #PBS -e errorfile.err > > #PBS -o logfile.log > > tpdir=`echo $PBS_JOBID | cut -f 1 -d .` > > tempdir=$HOME/work/job$tpdir > > mkdir -p $tempdir > > cd $tempdir > > cp -R $PBS_O_WORKDIR/* . > > mpiexec.hydra -np 4 -hostfile $PBS_NODEFILE /Apps/gromacs512/bin/gmx_mpi > > mdrun -v -dlb yes -ntomp 16 -s equilibration3.tpr > > > > Interestingly, I am using the same script to run CPU only jobs, which are > > not creating any problems. > > > > Please check the generated log file here: > > https://www.dropbox.com/s/dtfsuh6dv635n6q/md.log?dl=0 > > > > > > > > > > On Thu, Apr 7, 2016 at 6:18 PM, Szilárd Páll <[email protected]> > > wrote: > > > > > On Thu, Apr 7, 2016 at 2:35 PM, Venkat Reddy <[email protected]> > > wrote: > > > > Thank you Mark for the quick response. > > > > I tried to change -np option to 4. But it seems that mdrun is using > > only > > > > one GPU in single node with four ranks. The nvidia-smi command shows > > > > > > > > | 0 7977 C /Apps/gromacs512/bin/gmx_mpi > > > > 130MiB | > > > > | 0 7978 C /Apps/gromacs512/bin/gmx_mpi > > > > 130MiB | > > > > | 0 7979 C /Apps/gromacs512/bin/gmx_mpi > > > > 130MiB | > > > > | 0 7980 C /Apps/gromacs512/bin/gmx_mpi > > > > 130MiB | > > > > > > No command line, no log file shown nothing to comment on. > > > > > > Additionally, if all four ranks you requested are on the same node > > > rather than split over two nodes, that likely means you're using an > > > incorrect job script -- definitely not a GROMACS issue. Please make > > > sure you can launch an MPI "Hello world" program over multiple nodes > > > first. > > > > > > > > > > Also the job folder has four backed up copies of same run. > > > > > > > > On Thu, Apr 7, 2016 at 5:07 PM, Mark Abraham < > [email protected] > > > > > > > wrote: > > > > > > > >> Hi, > > > >> > > > >> mpiexec.hydra -np 1 asks for a single MPI rank, which is what you > got. > > > But > > > >> you need at least two, ie. at least one on each rank, and at least > > four > > > if > > > >> you want to make use of the two GPUs on each of two nodes. > > > >> > > > >> Mark > > > >> > > > >> > > > >> On Thu, Apr 7, 2016 at 1:14 PM Venkat Reddy <[email protected]> > > > wrote: > > > >> > > > >> > Dear all, > > > >> > > > > >> > Please neglect my previous mail which was incomplete. > > > >> > > > > >> > I am trying to execute mdrun our GPU cluster with 7 nodes where > each > > > node > > > >> > is populated by 16 processors and two K40 GPU cards. I have no > > problem > > > >> with > > > >> > mdrun on single node. However, when I try to execute parallel run > on > > > two > > > >> > nodes with gmx_mpi executable (gromacs-5.1.2), the performance is > > > very > > > >> > slow. When I logged into individual nodes, I found that mdrun is > not > > > >> > utilizing both GPUs. The generated log file shows the following > > > message. > > > >> > > > > >> > Using 1 MPI process > > > >> > Using 16 OpenMP threads > > > >> > > > > >> > 2 compatible GPUs are present, with IDs 0,1 > > > >> > 1 GPU auto-selected for this run. > > > >> > Mapping of GPU ID to the 1 PP rank in this node: 0 > > > >> > > > > >> > > > > >> > NOTE: potentially sub-optimal launch configuration, gmx_mpi > started > > > with > > > >> > less > > > >> > PP MPI process per node than GPUs available. > > > >> > Each PP MPI process can use only one GPU, 1 GPU per node > will > > be > > > >> > used. > > > >> > > > > >> > I read the manual and instructions in > > > >> > > > > >> > > > > >> > > > > > > http://manual.gromacs.org/documentation/5.1/user-guide/mdrun-performance.html > > > >> > to > > > >> > execute the parallel run. But I couldn't find the right flags to > > > initiate > > > >> > it. Please help me in this aspect. The script I used to execute > the > > > >> > parallel run is given below. > > > >> > > > > >> > #! /bin/bash > > > >> > #PBS -l cput=5000:00:00 > > > >> > #PBS -l select=2:ncpus=16:ngpus=2 > > > >> > #PBS -e errorfile.err > > > >> > #PBS -o logfile.log > > > >> > tpdir=`echo $PBS_JOBID | cut -f 1 -d .` > > > >> > tempdir=$HOME/work/job$tpdir > > > >> > mkdir -p $tempdir > > > >> > cd $tempdir > > > >> > cp -R $PBS_O_WORKDIR/* . > > > >> > mpiexec.hydra -np 1 -hostfile $PBS_NODEFILE > > > /Apps/gromacs512/bin/gmx_mpi > > > >> > mdrun -v -dlb yes -ntomp 16 -s equilibration3.tpr > > > >> > > > > >> > > > > >> > > > > >> > -- > > > >> > With Best Wishes > > > >> > Venkat Reddy Chirasani > > > >> > PhD student > > > >> > Laboratory of Computational Biophysics > > > >> > Department of Biotechnology > > > >> > IIT Madras > > > >> > Chennai > > > >> > INDIA-600036 > > > >> > -- > > > >> > Gromacs Users mailing list > > > >> > > > > >> > * Please search the archive at > > > >> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List > before > > > >> > posting! > > > >> > > > > >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > >> > > > > >> > * For (un)subscribe requests visit > > > >> > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users > > or > > > >> > send a mail to [email protected]. > > > >> > > > > >> -- > > > >> Gromacs Users mailing list > > > >> > > > >> * Please search the archive at > > > >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > > > >> posting! > > > >> > > > >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > >> > > > >> * For (un)subscribe requests visit > > > >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users > or > > > >> send a mail to [email protected]. > > > >> > > > > > > > > > > > > > > > > -- > > > > With Best Wishes > > > > Venkat Reddy Chirasani > > > > PhD student > > > > Laboratory of Computational Biophysics > > > > Department of Biotechnology > > > > IIT Madras > > > > Chennai > > > > INDIA-600036 > > > > -- > > > > Gromacs Users mailing list > > > > > > > > * Please search the archive at > > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > > > posting! > > > > > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > > > > > > * For (un)subscribe requests visit > > > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users > or > > > send a mail to [email protected]. > > > -- > > > Gromacs Users mailing list > > > > > > * Please search the archive at > > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > > > posting! > > > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > > > > * For (un)subscribe requests visit > > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > > > send a mail to [email protected]. > > > > > > > > > > > -- > > With Best Wishes > > Venkat Reddy Chirasani > > PhD student > > Laboratory of Computational Biophysics > > Department of Biotechnology > > IIT Madras > > Chennai > > INDIA-600036 > > -- > > Gromacs Users mailing list > > > > * Please search the archive at > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > > posting! > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > > * For (un)subscribe requests visit > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > > send a mail to [email protected]. > -- > Gromacs Users mailing list > > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > posting! > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > * For (un)subscribe requests visit > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > send a mail to [email protected]. > -- With Best Wishes Venkat Reddy Chirasani PhD student Laboratory of Computational Biophysics Department of Biotechnology IIT Madras Chennai INDIA-600036 -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to [email protected].
