Dear Christoph, On Friday, April 20, 2018 at 1:11:03 PM UTC-4, Christoph Junghans wrote: > > On Fri, Apr 20, 2018 at 9:33 AM, Salman Zarrini > <[email protected] <javascript:>> wrote: > > Thanks Christoph. > > > > On Friday, April 20, 2018 at 8:41:41 AM UTC-4, Christoph Junghans wrote: > >> > >> On Fri, Apr 20, 2018 at 3:35 AM, Salman Zarrini > >> <[email protected]> wrote: > >> > Dear Votca developers, > >> > > >> > My Votca calculations are conducted in a cluster which has 82 > machines > >> > each with 16 slots. > >> > While the Gromacs can parallel either inter-nodes or intranode, the > >> > Votca is only multithreaded so that only 16 slots can be used > maximally. > >> > Consequently the Gromacs as the most time consuming part of a > IBI-Votca job > >> > would be limited to 16 slots as well. > >> > So, I wonder if there is any possibility to use a higher number of > slots > >> > for at least the Gromacs part of a IBI calculation in such situation? > >> Sure nothing prevents you from running VOTCA and gromacs on a > >> different number of cores, e.g. using 4 nodes (4*16 cores) for the > >> gromacs part (with mpi) and only 1 node (16 cores) for VOTCA's > >> csg_stat. > > > > Probably doable for the csg_stat by using --nt 16, but how about the > > csg_inverse in IBI; > > As far as I know, the allocation for running a IBI job (csg_inverse > > --options settings.xml) in cluster is addressed by two below commands in > job > > submission script and setting.xml, respectively. > > > > #$ -pe shm n ## Or other types of parallel environment. > > > > <mdrun> > > <command> mpirun -np m gmx_mpi mdrun </command> ### (m =< > n) > > </mdrun> > > > > The n slots indicated in the job submission script would be assigned to > the > > both VOTCA and Gromacs commands. Then how can I order to use 4 nodes > (4*16 > > cores) for the gromacs and 1 node (16 cores) for VOTCA (probably the > other > > way round was much easier in general)? And How VOTCA knows that those > 16 > > cores are all belong to a single node? because if only 1 core from the > 16 > > cores is contributed by another node the VOTCA part crashes. > The only compute-heavy part in csg_inverse besides mdrun is csg_stat, > all the other things take basically no time. > And csg_stat when called inside csg_inverse will automatically be > given the max. number of threads with the --nt options (check your > inverse.log, you can overwrite that with the > cg.inverse.simulation.tasks option in settings.xml). > On the other hand the behavior of mdrun can be changed by > cg.inverse.gromacs.mdrun.command, as you said above. > > So in your batch script you should set the number in "-pe" option to > same number as you want to use for gromacs (and set in > cg.inverse.gromacs.mdrun.command, too), but then in body of the > script, just call csg_inverse without mpirun/mpiexec. Then csg_inverse > will run serially, but call gromacs with mpi options and csg_stat will > automatically make use of all threads on the node where csg_inverse > got started. > The cg.inverse.simulation.tasks helps to allocate different number of slot for the "csg_inverse" and "mpirun gmx_mpi mdrun", thanks.
Nevertheless, when I call Groamcs via the VOTCA (settings.xml) to be used in multiple nodes the speed drastically decreases even for a small system, whereas the Groamcs-only job free of VOTCA, works normally fine. After discussing the issue with our system administrator, we found out that the reason probably is because the Gromacs command is hardcoded into the settings.xml file so it cannot dynamically contain the hostlist that was decided by the scheduler at run time, whereas in Groamcs-only job the environment is readable directly by mpirun which is aware of Grid Engine environment variables. So, I was wondering if there are any other command that should be considered in the settings.xml or any solution in general to somehow figure out the issue? Sincerely, Salman Zarrini > Christoph > > > > Thank you. > > Regards, > > Salman Zarrini > >> > >> > >> Christoph > >> > >> > BTW, Votca launches Gromacs directly with "mpirun” now. > >> > > >> > Regards, > >> > > >> > Salman Zarrini > >> > Department of Chemical Engineering > >> > Drexel University > >> > Philadelphia, USA > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups "votca" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send > >> > an email to [email protected]. > >> > To post to this group, send email to [email protected]. > >> > Visit this group at https://groups.google.com/group/votca. > >> > For more options, visit https://groups.google.com/d/optout. > >> > >> > >> > >> -- > >> Christoph Junghans > >> Web: http://www.compphys.de > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "votca" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > To post to this group, send email to [email protected] > <javascript:>. > > Visit this group at https://groups.google.com/group/votca. > > For more options, visit https://groups.google.com/d/optout. > > > > -- > Christoph Junghans > Web: http://www.compphys.de > -- You received this message because you are subscribed to the Google Groups "votca" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/votca. For more options, visit https://groups.google.com/d/optout.
