Hi Szilárd, Thanks, option 3 was in my mind but I need to figure out now how :) Manually fixing pinoffset as of now seems working with some quick test. I think option 1 would require to ask the admin but I can try option 3 myself. As there are other users from different places who may not bother using option 3. I think I would need to ask the admin to force option 1 but before that I will try option 3.
JIom On Wed, Sep 13, 2017 at 7:10 PM, Szilárd Páll <[email protected]> wrote: > J, > > You have a few options: > > * Use SLURM to assign not only the set of GPUs, but also the correct > set of CPU cores to each mdrun process. If you do so, mdrun will > respect the affinity mask it will inherit and your two mdrun jobs > should be running on the right set of cores. This has the drawback > that (AFAIK) SLURM/aprun (or srun) will not allow you to bind each > application thread to a core/hardware thread (which is what mdrun > does), only a process to a group of cores/hw threads which can > sometimes lead to performance loss. (You might be able to compensate > using some OpenMP library environment variables, though.) > > * Run multiple jobs with mdrun "-multi"/"-multidir" (either two per > node or mulitple across nodes) and benefit from the rank/thread to > core/hw thread assignment that's supported also across multiple > simulations part of a multi-run; e.g.: > mpirun -np 4 gmx mdrun -multi 4 -ntomp N -multidir my_input_dir{1,2,3,4} > will launch 4 ranks and start 4 simulations in each of the four > directories passed. > > * Write a wrapper script around gmx mdrun which will be what you > launch with SLURM; you can then inspect the node and decide what > pinoffset value to pass to your mdrun launch command. > > > I hope one of these will deliver the desired results :) > > Cheers, > -- > Szilárd > > > On Wed, Sep 13, 2017 at 7:47 PM, gromacs query <[email protected]> > wrote: > > Hi Szilárd, > > > > Thanks for your reply. This is useful but now am thinking because the > slurm > > launches job in an automated way it is not really in my control to choose > > the node. So following things can happen; say for two mdrun jobs I set > > -pinoffset 0 and -pinoffset 4; > > > > - if they are running on the same node this is good > > - if jobs run on different nodes (partially occupied or free) whether > these > > chosen pinoffsets will make sense or not as I don't know what pinoffset I > > would need to set > > - if I have to submit many jobs together and slurm chooses different/same > > node itself then I think it is difficult to define pinoffset. > > > > - > > J > > > > On Wed, Sep 13, 2017 at 6:14 PM, Szilárd Páll <[email protected]> > > wrote: > > > >> My guess is that the two jobs are using the same cores -- either all > >> cores/threads or only half of them, but the same set. > >> > >> You should use -pinoffset; see: > >> > >> - Docs and example: > >> http://manual.gromacs.org/documentation/2016/user-guide/ > >> mdrun-performance.html > >> > >> - More explanation on the thread pinning behavior on the old website: > >> http://www.gromacs.org/Documentation/Acceleration_ > >> and_parallelization#Pinning_threads_to_physical_cores > >> > >> Cheers, > >> -- > >> Szilárd > >> > >> > >> On Wed, Sep 13, 2017 at 6:35 PM, gromacs query <[email protected]> > >> wrote: > >> > Sorry forgot to add; we thought the two jobs are using same GPU ids > but > >> > cuda visible devices show both jobs are using different ids (0,1 and > 2,3) > >> > > >> > - > >> > J > >> > > >> > On Wed, Sep 13, 2017 at 5:33 PM, gromacs query < > [email protected]> > >> > wrote: > >> > > >> >> Hi All, > >> >> > >> >> I have some issues with gromacs performance. There are many nodes and > >> each > >> >> node has number of gpus and the batch process is controlled by slurm. > >> >> Although I get good performance with some settings of number of gpus > and > >> >> nprocs but when I submit same job twice on the same node then the > >> >> performance is reduced drastically. e.g > >> >> > >> >> For 2 GPUs I get 300 ns per day when there is no other job running on > >> the > >> >> node. When I submit same job twice on the same node & at the same > time, > >> I > >> >> get only 17 ns/day for both the jobs. I am using this: > >> >> > >> >> mpirun -np 4 gmx_mpi mdrun -deffnm test -ntomp 2 -maxh 0.12 > >> >> > >> >> Any suggestions highly appreciated. > >> >> > >> >> Thanks > >> >> > >> >> Jiom > >> >> > >> > -- > >> > Gromacs Users mailing list > >> > > >> > * Please search the archive at http://www.gromacs.org/ > >> Support/Mailing_Lists/GMX-Users_List before posting! > >> > > >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > >> > > >> > * For (un)subscribe requests visit > >> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > >> send a mail to [email protected]. > >> -- > >> Gromacs Users mailing list > >> > >> * Please search the archive at http://www.gromacs.org/ > >> Support/Mailing_Lists/GMX-Users_List before posting! > >> > >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > >> > >> * For (un)subscribe requests visit > >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > >> send a mail to [email protected]. > > -- > > Gromacs Users mailing list > > > > * Please search the archive at http://www.gromacs.org/Support > /Mailing_Lists/GMX-Users_List before posting! > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > > * For (un)subscribe requests visit > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > send a mail to [email protected]. > -- > Gromacs Users mailing list > > * Please search the archive at http://www.gromacs.org/Support > /Mailing_Lists/GMX-Users_List before posting! > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > * For (un)subscribe requests visit > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > send a mail to [email protected]. > -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to [email protected].
