On Thu, Apr 26, 2018 at 12:23 PM, Salman Zarrini <[email protected]> wrote: > > Dear Christoph, > > On Friday, April 20, 2018 at 1:11:03 PM UTC-4, Christoph Junghans wrote: >> >> On Fri, Apr 20, 2018 at 9:33 AM, Salman Zarrini >> <[email protected]> wrote: >> > Thanks Christoph. >> > >> > On Friday, April 20, 2018 at 8:41:41 AM UTC-4, Christoph Junghans wrote: >> >> >> >> On Fri, Apr 20, 2018 at 3:35 AM, Salman Zarrini >> >> <[email protected]> wrote: >> >> > Dear Votca developers, >> >> > >> >> > My Votca calculations are conducted in a cluster which has 82 >> >> > machines >> >> > each with 16 slots. >> >> > While the Gromacs can parallel either inter-nodes or intranode, the >> >> > Votca is only multithreaded so that only 16 slots can be used >> >> > maximally. >> >> > Consequently the Gromacs as the most time consuming part of a >> >> > IBI-Votca job >> >> > would be limited to 16 slots as well. >> >> > So, I wonder if there is any possibility to use a higher number of >> >> > slots >> >> > for at least the Gromacs part of a IBI calculation in such situation? >> >> Sure nothing prevents you from running VOTCA and gromacs on a >> >> different number of cores, e.g. using 4 nodes (4*16 cores) for the >> >> gromacs part (with mpi) and only 1 node (16 cores) for VOTCA's >> >> csg_stat. >> > >> > Probably doable for the csg_stat by using --nt 16, but how about the >> > csg_inverse in IBI; >> > As far as I know, the allocation for running a IBI job (csg_inverse >> > --options settings.xml) in cluster is addressed by two below commands in >> > job >> > submission script and setting.xml, respectively. >> > >> > #$ -pe shm n ## Or other types of parallel environment. >> > >> > <mdrun> >> > <command> mpirun -np m gmx_mpi mdrun </command> ### (m =< >> > n) >> > </mdrun> >> > >> > The n slots indicated in the job submission script would be assigned to >> > the >> > both VOTCA and Gromacs commands. Then how can I order to use 4 nodes >> > (4*16 >> > cores) for the gromacs and 1 node (16 cores) for VOTCA (probably the >> > other >> > way round was much easier in general)? And How VOTCA knows that those >> > 16 >> > cores are all belong to a single node? because if only 1 core from the >> > 16 >> > cores is contributed by another node the VOTCA part crashes. >> The only compute-heavy part in csg_inverse besides mdrun is csg_stat, >> all the other things take basically no time. >> And csg_stat when called inside csg_inverse will automatically be >> given the max. number of threads with the --nt options (check your >> inverse.log, you can overwrite that with the >> cg.inverse.simulation.tasks option in settings.xml). >> On the other hand the behavior of mdrun can be changed by >> cg.inverse.gromacs.mdrun.command, as you said above. >> >> So in your batch script you should set the number in "-pe" option to >> same number as you want to use for gromacs (and set in >> cg.inverse.gromacs.mdrun.command, too), but then in body of the >> script, just call csg_inverse without mpirun/mpiexec. Then csg_inverse >> will run serially, but call gromacs with mpi options and csg_stat will >> automatically make use of all threads on the node where csg_inverse >> got started. > > > The cg.inverse.simulation.tasks helps to allocate different number of slot > for the "csg_inverse" and "mpirun gmx_mpi mdrun", thanks. slot for *csg_stat", csg_inverse is never parallel.
> > Nevertheless, when I call Groamcs via the VOTCA (settings.xml) to be used in > multiple nodes the speed drastically decreases even for a small system, > whereas the Groamcs-only job free of VOTCA, works normally fine. After > discussing the issue with our system administrator, we found out that the > reason probably is because the Gromacs command is hardcoded into the > settings.xml file so it cannot dynamically contain the hostlist that was > decided by the scheduler at run time, whereas in Groamcs-only job the > environment is readable directly by mpirun which is aware of Grid Engine > environment variables. > > So, I was wondering if there are any other command that should be considered > in the settings.xml or any solution in general to somehow figure out the > issue? You could dump the hostlist to a file in batch script before starting csg_inverse. And then use that hostfile by running gmx with "mpirun --hostfile .." through the setting settings.xml. E.g. in the slurm batching system, you can use $SLURM_HOSTFILE for that. Christoph > > Sincerely, > > Salman Zarrini > >> >> Christoph >> > >> > Thank you. >> > Regards, >> > Salman Zarrini >> >> >> >> >> >> Christoph >> >> >> >> > BTW, Votca launches Gromacs directly with "mpirun” now. >> >> > >> >> > Regards, >> >> > >> >> > Salman Zarrini >> >> > Department of Chemical Engineering >> >> > Drexel University >> >> > Philadelphia, USA >> >> > >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups "votca" group. >> >> > To unsubscribe from this group and stop receiving emails from it, >> >> > send >> >> > an email to [email protected]. >> >> > To post to this group, send email to [email protected]. >> >> > Visit this group at https://groups.google.com/group/votca. >> >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> >> >> >> >> -- >> >> Christoph Junghans >> >> Web: http://www.compphys.de >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "votca" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to [email protected]. >> > To post to this group, send email to [email protected]. >> > Visit this group at https://groups.google.com/group/votca. >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> Christoph Junghans >> Web: http://www.compphys.de > > -- > You received this message because you are subscribed to the Google Groups > "votca" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/votca. > For more options, visit https://groups.google.com/d/optout. -- Christoph Junghans Web: http://www.compphys.de -- You received this message because you are subscribed to the Google Groups "votca" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/votca. For more options, visit https://groups.google.com/d/optout.
