Hi Michael, Can you post full logs, please? This is likely related to a known issue where CPU cores (and in some cases GPUs too) may take longer to clock up and get a stable performance than the time the auto-tuner takes to do a few cycles of measurements.
Unfortunately we do not have a good solution for this, but what you can do make runs more consistent is: - try "warming up" the CPU/GPU before production runs (e.g. stress -c or just a dummy 30 sec mdrun run) - repeat the benchmark a few times, see which cutoff / grid setting is best, set that in the mdp options and run with -notunepme Of course the latter may be too tedious if you have a variety of systems/inputs to run. Regarding tune_pme: that issue is related to resetting timings too early (for -resetstep see mdrun -h -hidden); not sure if we have a fix, but either way tune_pme is more suited for parallel runs' separate PME rank count tuning. Cheers, -- Szilárd On Thu, Mar 1, 2018 at 7:11 PM, Michael Brunsteiner <mbx0...@yahoo.com> wrote: > Hi,I ran a few MD runs with identical input files (the SAME tpr file. mdp > included below) on the same computer > with gmx 2018 and observed rather large performance variations (~50%) as > in: > grep Performance */mcz1.log7/mcz1.log:Performance: 98.510 > 0.244 > 7d/mcz1.log:Performance: 140.733 0.171 > 7e/mcz1.log:Performance: 115.586 0.208 > 7f/mcz1.log:Performance: 139.197 0.172 > > turns out the load balancing effort that is done at the beginning gives > quite different results: > grep "optimal pme grid" */mcz1.log > 7/mcz1.log: optimal pme grid 32 32 28, coulomb cutoff 1.394 > 7d/mcz1.log: optimal pme grid 36 36 32, coulomb cutoff 1.239 > 7e/mcz1.log: optimal pme grid 25 24 24, coulomb cutoff 1.784 > 7f/mcz1.log: optimal pme grid 40 36 32, coulomb cutoff 1.200 > > next i tried tune_pme as in:gmx tune_pme -mdrun 'gmx mdrun' -nt 6 -ntmpi 1 > -ntomp 6 -pin on -pinoffset 0 -s mcz1.tpr -pmefft cpu -pinstride 1 -r 10 > which didn't work ... in some log file it says:Fatal error: > PME tuning was still active when attempting to reset mdrun counters at step > 1500. Try resetting counters later in the run, e.g. with gmx mdrun > -resetstep. > > i found no documentation regarding "-resetstep" ... > > i could of course optimize the PME grid manually but since i plan to run a > large numberof jobs with different systems and sizes this would be a lot of > work and if possible i'd like to avoid that. > is there any way to ask gmx to perform more tests at the beginning of > therun when optimizing the PME grid?or is using "-notunepme -dlb yes" an > option, and does the latter require aconcurrent optimization of the domain > decomposition, if so how is this done? > thanks for any help! > michael > > > mdp: > integrator = md > dt = 0.001 > nsteps = 500000 > comm-grps = System > ; > nstxout = 0 > nstvout = 0 > nstfout = 0 > nstlog = 1000 > nstenergy = 1000 > ; > nstlist = 40 > ns_type = grid > pbc = xyz > rlist = 1.2 > cutoff-scheme = Verlet > ; > coulombtype = PME > rcoulomb = 1.2 > vdw_type = cut-off > rvdw = 1.2 > ; > constraints = none > ; > tcoupl = v-rescale > tau-t = 0.1 > ref-t = 300 > tc-grps = System > ; > pcoupl = berendsen > pcoupltype = anisotropic > tau-p = 2.0 > compressibility = 4.5e-5 4.5e-5 4.5e-5 0 0 0 > ref-p = 1 1 1 0 0 0 > ; > annealing = single > annealing-npoints = 2 > annealing-time = 0 500 > annealing-temp = 500 480 > > > -- > Gromacs Users mailing list > > * Please search the archive at http://www.gromacs.org/ > Support/Mailing_Lists/GMX-Users_List before posting! > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > * For (un)subscribe requests visit > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > send a mail to gmx-users-requ...@gromacs.org. -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org.