Re: [gmx-users] GMX 2020 - COMM Removal Issue

[Daniel Kozuch]

Thanks Paul! Unfortunately while 2020.1 seems to work for the membrane system 
(no protein), for the protein-membrane system I am still having significant COM 
motion in the xy plane (i.e. the membrane translates several nm/ns sideways in 
its own plane). This happens regardless of coupling the membrane and protein 
independently (3 COMM groups: protein, membrane, solvent) or together (2 COMM 
groups: protein-membrane, solvent). Any suggestions on how to track down why a 
protein imbedded in the membrane is causing this problem?

Best,
Dan

-Original Message-
From: gromacs.org_gmx-users-boun...@maillist.sys.kth.se 
 On Behalf Of Paul bauer
Sent: Friday, March 6, 2020 1:02 PM
To: gmx-us...@gromacs.org
Subject: Re: [gmx-users] GMX 2020 - COMM Removal Issue

Hello,

we resolved some issues with COM removal in the latest release (please see the 
patch notes for more details).
So it might be that you where affected by this before.

Please let us know if the issue shows up again.

Cheers

Paul

On 06/03/2020 18:58, Daniel Kozuch wrote:
> Additional (good) news, the problem appears to be resolved in the 2020.1 
> update (at least for the membrane only system). I'll conduct some additional 
> tests to see if there is any lingering problems.
>
> Dan
>
> -Original Message-
> From: gromacs.org_gmx-users-boun...@maillist.sys.kth.se 
>  On Behalf Of 
> Justin Lemkul
> Sent: Friday, March 6, 2020 11:02 AM
> To: gmx-us...@gromacs.org
> Subject: Re: [gmx-users] GMX 2020 - COMM Removal Issue
>
>
>
> On 3/6/20 10:00 AM, Daniel Kozuch wrote:
>> [Somehow my response got put in a different thread - hopefully this 
>> works]
>>
>> Justin,
>>
>> Thanks for your reply. I agree that some COM motion is normal.
>> However, this was a very short simulation (1 ns), so the size of the 
>> drift (several nm) was unexpected. To test, I repeated the simulation 
>> with GROMACS 2019.6 using all the same settings (but without the new
>> ones: GMX_GPU_DD_COMMS, GMX_GPU_PME_PP_COMMS, GMX_FORCE_UPDATE_DEFAULT_GPU), 
>> and I don't see the same drift.
> Does a membrane system (no protein) also cause the same issue?
>
> -Justin
>
>> Best,
>> Dan
>>
>> -Original Message-
>> From: gromacs.org_gmx-users-boun...@maillist.sys.kth.se
>>  On Behalf Of 
>> Justin Lemkul
>> Sent: Tuesday, March 3, 2020 3:02 PM
>> To: gmx-us...@gromacs.org
>> Subject: Re: [gmx-users] GMX 2020 - COMM Removal Issue
>>
>>
>>
>> On 3/2/20 9:53 PM, Daniel Kozuch wrote:
>>> Hello,
>>>
>>> I am experimenting with GROMACS 2020. I have compiled the mpi 
>>> threaded version and am using the new settings (GMX_GPU_DD_COMMS, 
>>> GMX_GPU_PME_PP_COMMS, GMX_FORCE_UPDATE_DEFAULT_GPU) as suggested on 
>>> at the following link:
>>> https://devblogs.nvidia.com/creating-faster-molecular-dynamics-simulations-with-gromacs-2020/.
>>> I am running mdrun  with the suggested options:
>>> "-pin on -nb gpu -bonded gpu -pme gpu -npme 1 -nstlist 400" on 4 
>>> GPUs and
>>> 28 CPUs with "-ntmpi 4 -ntomp 7".
>>>
>>> I am currently running a membrane system with a transmembrane 
>>> protein in water solvent. I am using the following settings for COM removal:
>>>
>>> comm_mode  = linear
>>> comm_grps  = PROT_MEMB SOL_ION
>>>
>>> Here I choose to couple the the protein and the membrane from the 
>>> advice in this thread:
>>> https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2016-Se
>>> p
>>> t
>>> ember/108584.html
>>>
>>> Unfortunately, I still see a large drift in the z-dimension for the 
>>> entire membrane/protein group. Currently I have 
>>> nstcalcenergy/nstcomm set to 100, as decreasing them leads to poor 
>>> performance. (Hopefully it is unnecessary to set them to 1)
>> Removing artificial contributions to COM motion does not mean that the 
>> entities cannot diffuse over time. Depending on the length of your 
>> simulation, drift in absolute position can be perfectly normal.
>>
>> -Justin
>>
>>> Does anyone have suggestions for avoiding the COM drift? I know this 
>>> issue has been discussed before
>>> (https://redmine.gromacs.org/issues/2867) but it looks like it was 
>>> resolved in earlier GROMACS versions. As a note, I am using a CHARMM force 
>>> field with CMAP dihedrals.
>>>
>>>
>>> Here are some other potentially relevant mdp options (from CHARMM) 
>>> in case they help:
>>>
>>> integrator  = md
>>&

Re: [gmx-users] GMX 2020 - COMM Removal Issue

[Daniel Kozuch]

Additional (good) news, the problem appears to be resolved in the 2020.1 update 
(at least for the membrane only system). I'll conduct some additional tests to 
see if there is any lingering problems.

Dan

-Original Message-
From: gromacs.org_gmx-users-boun...@maillist.sys.kth.se 
 On Behalf Of Justin Lemkul
Sent: Friday, March 6, 2020 11:02 AM
To: gmx-us...@gromacs.org
Subject: Re: [gmx-users] GMX 2020 - COMM Removal Issue



On 3/6/20 10:00 AM, Daniel Kozuch wrote:
> [Somehow my response got put in a different thread - hopefully this 
> works]
>
> Justin,
>
> Thanks for your reply. I agree that some COM motion is normal. 
> However, this was a very short simulation (1 ns), so the size of the 
> drift (several nm) was unexpected. To test, I repeated the simulation 
> with GROMACS 2019.6 using all the same settings (but without the new
> ones: GMX_GPU_DD_COMMS, GMX_GPU_PME_PP_COMMS, GMX_FORCE_UPDATE_DEFAULT_GPU), 
> and I don't see the same drift.

Does a membrane system (no protein) also cause the same issue?

-Justin

> Best,
> Dan
>
> -Original Message-
> From: gromacs.org_gmx-users-boun...@maillist.sys.kth.se 
>  On Behalf Of 
> Justin Lemkul
> Sent: Tuesday, March 3, 2020 3:02 PM
> To: gmx-us...@gromacs.org
> Subject: Re: [gmx-users] GMX 2020 - COMM Removal Issue
>
>
>
> On 3/2/20 9:53 PM, Daniel Kozuch wrote:
>> Hello,
>>
>> I am experimenting with GROMACS 2020. I have compiled the mpi 
>> threaded version and am using the new settings (GMX_GPU_DD_COMMS, 
>> GMX_GPU_PME_PP_COMMS, GMX_FORCE_UPDATE_DEFAULT_GPU) as suggested on 
>> at the following link:
>> https://devblogs.nvidia.com/creating-faster-molecular-dynamics-simulations-with-gromacs-2020/.
>> I am running mdrun  with the suggested options:
>> "-pin on -nb gpu -bonded gpu -pme gpu -npme 1 -nstlist 400" on 4 GPUs 
>> and
>> 28 CPUs with "-ntmpi 4 -ntomp 7".
>>
>> I am currently running a membrane system with a transmembrane protein 
>> in water solvent. I am using the following settings for COM removal:
>>
>> comm_mode  = linear
>> comm_grps  = PROT_MEMB SOL_ION
>>
>> Here I choose to couple the the protein and the membrane from the 
>> advice in this thread:
>> https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2016-Sep
>> t
>> ember/108584.html
>>
>> Unfortunately, I still see a large drift in the z-dimension for the 
>> entire membrane/protein group. Currently I have nstcalcenergy/nstcomm 
>> set to 100, as decreasing them leads to poor performance. (Hopefully 
>> it is unnecessary to set them to 1)
> Removing artificial contributions to COM motion does not mean that the 
> entities cannot diffuse over time. Depending on the length of your 
> simulation, drift in absolute position can be perfectly normal.
>
> -Justin
>
>> Does anyone have suggestions for avoiding the COM drift? I know this 
>> issue has been discussed before
>> (https://redmine.gromacs.org/issues/2867) but it looks like it was 
>> resolved in earlier GROMACS versions. As a note, I am using a CHARMM force 
>> field with CMAP dihedrals.
>>
>>
>> Here are some other potentially relevant mdp options (from CHARMM) in 
>> case they help:
>>
>> integrator  = md
>> dt  = 0.002
>> nstcalcenergy   = 100
>> ;
>> cutoff-scheme   = Verlet
>> nstlist = 20
>> rlist   = 1.2
>> coulombtype = pme
>> rcoulomb= 1.2
>> vdwtype = Cut-off
>> vdw-modifier= Force-switch
>> rvdw_switch = 1.0
>> rvdw= 1.2
>> ;
>> tcoupl  = v-rescale
>> tc_grps = PROT_MEMB   SOL_ION
>> tau_t   = 1.01.0
>> ref_t   = 303.15 303.15
>> ;
>> pcoupl  = Berendsen
>> pcoupltype  = semiisotropic
>> tau_p   = 5.0
>> compressibility = 4.5e-5  4.5e-5
>> ref_p   = 1.0 1.0
>> ;
>> constraints = h-bonds
>> constraint_algorithm= LINCS
>> continuation= yes
>>
>> Best,
>> Dan
> --
> ==
>
> Justin A. Lemkul, Ph.D.
> Assistant Professor
> Office: 301 Fralin Hall
> Lab: 303 Engel Hall
>
> Virginia Tech Department of Biochemistry
> 340 West Campus Dr.
> Blacksburg, VA 24061
>
> jalem...@vt.edu | (540) 231-3129
> http://www.thelemkullab.com
>
> ===

Re: [gmx-users] GMX 2020 - COMM Removal Issue

[Daniel Kozuch]

Yes, a smaller system with only membrane/solvent (no protein) causes the same 
issue.

Best,
Dan

-Original Message-
From: gromacs.org_gmx-users-boun...@maillist.sys.kth.se 
 On Behalf Of Justin Lemkul
Sent: Friday, March 6, 2020 11:02 AM
To: gmx-us...@gromacs.org
Subject: Re: [gmx-users] GMX 2020 - COMM Removal Issue



On 3/6/20 10:00 AM, Daniel Kozuch wrote:
> [Somehow my response got put in a different thread - hopefully this 
> works]
>
> Justin,
>
> Thanks for your reply. I agree that some COM motion is normal. 
> However, this was a very short simulation (1 ns), so the size of the 
> drift (several nm) was unexpected. To test, I repeated the simulation 
> with GROMACS 2019.6 using all the same settings (but without the new
> ones: GMX_GPU_DD_COMMS, GMX_GPU_PME_PP_COMMS, GMX_FORCE_UPDATE_DEFAULT_GPU), 
> and I don't see the same drift.

Does a membrane system (no protein) also cause the same issue?

-Justin

> Best,
> Dan
>
> -Original Message-
> From: gromacs.org_gmx-users-boun...@maillist.sys.kth.se 
>  On Behalf Of 
> Justin Lemkul
> Sent: Tuesday, March 3, 2020 3:02 PM
> To: gmx-us...@gromacs.org
> Subject: Re: [gmx-users] GMX 2020 - COMM Removal Issue
>
>
>
> On 3/2/20 9:53 PM, Daniel Kozuch wrote:
>> Hello,
>>
>> I am experimenting with GROMACS 2020. I have compiled the mpi 
>> threaded version and am using the new settings (GMX_GPU_DD_COMMS, 
>> GMX_GPU_PME_PP_COMMS, GMX_FORCE_UPDATE_DEFAULT_GPU) as suggested on 
>> at the following link:
>> https://devblogs.nvidia.com/creating-faster-molecular-dynamics-simulations-with-gromacs-2020/.
>> I am running mdrun  with the suggested options:
>> "-pin on -nb gpu -bonded gpu -pme gpu -npme 1 -nstlist 400" on 4 GPUs 
>> and
>> 28 CPUs with "-ntmpi 4 -ntomp 7".
>>
>> I am currently running a membrane system with a transmembrane protein 
>> in water solvent. I am using the following settings for COM removal:
>>
>> comm_mode  = linear
>> comm_grps  = PROT_MEMB SOL_ION
>>
>> Here I choose to couple the the protein and the membrane from the 
>> advice in this thread:
>> https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2016-Sep
>> t
>> ember/108584.html
>>
>> Unfortunately, I still see a large drift in the z-dimension for the 
>> entire membrane/protein group. Currently I have nstcalcenergy/nstcomm 
>> set to 100, as decreasing them leads to poor performance. (Hopefully 
>> it is unnecessary to set them to 1)
> Removing artificial contributions to COM motion does not mean that the 
> entities cannot diffuse over time. Depending on the length of your 
> simulation, drift in absolute position can be perfectly normal.
>
> -Justin
>
>> Does anyone have suggestions for avoiding the COM drift? I know this 
>> issue has been discussed before
>> (https://redmine.gromacs.org/issues/2867) but it looks like it was 
>> resolved in earlier GROMACS versions. As a note, I am using a CHARMM force 
>> field with CMAP dihedrals.
>>
>>
>> Here are some other potentially relevant mdp options (from CHARMM) in 
>> case they help:
>>
>> integrator  = md
>> dt  = 0.002
>> nstcalcenergy   = 100
>> ;
>> cutoff-scheme   = Verlet
>> nstlist = 20
>> rlist   = 1.2
>> coulombtype = pme
>> rcoulomb= 1.2
>> vdwtype = Cut-off
>> vdw-modifier= Force-switch
>> rvdw_switch = 1.0
>> rvdw= 1.2
>> ;
>> tcoupl  = v-rescale
>> tc_grps = PROT_MEMB   SOL_ION
>> tau_t   = 1.01.0
>> ref_t   = 303.15 303.15
>> ;
>> pcoupl  = Berendsen
>> pcoupltype  = semiisotropic
>> tau_p   = 5.0
>> compressibility = 4.5e-5  4.5e-5
>> ref_p   = 1.0 1.0
>> ;
>> constraints = h-bonds
>> constraint_algorithm= LINCS
>> continuation= yes
>>
>> Best,
>> Dan
> --
> ==
>
> Justin A. Lemkul, Ph.D.
> Assistant Professor
> Office: 301 Fralin Hall
> Lab: 303 Engel Hall
>
> Virginia Tech Department of Biochemistry
> 340 West Campus Dr.
> Blacksburg, VA 24061
>
> jalem...@vt.edu | (540) 231-3129
> http://www.thelemkullab.com
>
> ==
>
> --
> Gromacs Users mailing list
>
> * Please sear

Re: [gmx-users] GMX 2020 - COMM Removal Issue

[Daniel Kozuch]

[Somehow my response got put in a different thread - hopefully this works]

Justin,

Thanks for your reply. I agree that some COM motion is normal. However, this 
was a very short simulation (1 ns), so the size of the drift (several nm) was 
unexpected. To test, I repeated the simulation with GROMACS 2019.6 using all 
the same settings (but without the new
ones: GMX_GPU_DD_COMMS, GMX_GPU_PME_PP_COMMS, GMX_FORCE_UPDATE_DEFAULT_GPU), 
and I don't see the same drift.

Best,
Dan

-Original Message-
From: gromacs.org_gmx-users-boun...@maillist.sys.kth.se 
 On Behalf Of Justin Lemkul
Sent: Tuesday, March 3, 2020 3:02 PM
To: gmx-us...@gromacs.org
Subject: Re: [gmx-users] GMX 2020 - COMM Removal Issue



On 3/2/20 9:53 PM, Daniel Kozuch wrote:
> Hello,
>
> I am experimenting with GROMACS 2020. I have compiled the mpi threaded 
> version and am using the new settings (GMX_GPU_DD_COMMS, 
> GMX_GPU_PME_PP_COMMS, GMX_FORCE_UPDATE_DEFAULT_GPU) as suggested on at 
> the following link:
> https://devblogs.nvidia.com/creating-faster-molecular-dynamics-simulations-with-gromacs-2020/.
> I am running mdrun  with the suggested options:
> "-pin on -nb gpu -bonded gpu -pme gpu -npme 1 -nstlist 400" on 4 GPUs 
> and
> 28 CPUs with "-ntmpi 4 -ntomp 7".
>
> I am currently running a membrane system with a transmembrane protein 
> in water solvent. I am using the following settings for COM removal:
>
> comm_mode  = linear
> comm_grps  = PROT_MEMB SOL_ION
>
> Here I choose to couple the the protein and the membrane from the 
> advice in this thread:
> https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2016-Sept
> ember/108584.html
>
> Unfortunately, I still see a large drift in the z-dimension for the 
> entire membrane/protein group. Currently I have nstcalcenergy/nstcomm 
> set to 100, as decreasing them leads to poor performance. (Hopefully 
> it is unnecessary to set them to 1)

Removing artificial contributions to COM motion does not mean that the entities 
cannot diffuse over time. Depending on the length of your simulation, drift in 
absolute position can be perfectly normal.

-Justin

> Does anyone have suggestions for avoiding the COM drift? I know this 
> issue has been discussed before 
> (https://redmine.gromacs.org/issues/2867) but it looks like it was 
> resolved in earlier GROMACS versions. As a note, I am using a CHARMM force 
> field with CMAP dihedrals.
>
>
> Here are some other potentially relevant mdp options (from CHARMM) in 
> case they help:
>
> integrator  = md
> dt  = 0.002
> nstcalcenergy   = 100
> ;
> cutoff-scheme   = Verlet
> nstlist = 20
> rlist   = 1.2
> coulombtype = pme
> rcoulomb= 1.2
> vdwtype = Cut-off
> vdw-modifier= Force-switch
> rvdw_switch = 1.0
> rvdw= 1.2
> ;
> tcoupl  = v-rescale
> tc_grps = PROT_MEMB   SOL_ION
> tau_t   = 1.01.0
> ref_t   = 303.15 303.15
> ;
> pcoupl  = Berendsen
> pcoupltype  = semiisotropic
> tau_p   = 5.0
> compressibility = 4.5e-5  4.5e-5
> ref_p   = 1.0 1.0
> ;
> constraints = h-bonds
> constraint_algorithm= LINCS
> continuation= yes
>
> Best,
> Dan

--
==

Justin A. Lemkul, Ph.D.
Assistant Professor
Office: 301 Fralin Hall
Lab: 303 Engel Hall

Virginia Tech Department of Biochemistry
340 West Campus Dr.
Blacksburg, VA 24061

jalem...@vt.edu | (540) 231-3129
http://www.thelemkullab.com

==

--
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] GMX 2020 - COMM Removal Issue

[Daniel Kozuch]

I agree, but this was a very short simulation (1 ns), so the size of the
drift (several nm) was unexpected. To test, I repeated the simulation with
GROMACS 2019.6 using all the same settings (but without the new
ones: GMX_GPU_DD_COMMS, GMX_GPU_PME_PP_COMMS,
GMX_FORCE_UPDATE_DEFAULT_GPU), and I don't see the same drift.

Best,
Dan

On Tue, Mar 3, 2020 at 3:03 PM Justin Lemkul  wrote:

>
>
> On 3/2/20 9:53 PM, Daniel Kozuch wrote:
> > Hello,
> >
> > I am experimenting with GROMACS 2020. I have compiled the mpi threaded
> > version and am using the new settings
> > (GMX_GPU_DD_COMMS, GMX_GPU_PME_PP_COMMS, GMX_FORCE_UPDATE_DEFAULT_GPU) as
> > suggested on at the following link:
> >
> https://devblogs.nvidia.com/creating-faster-molecular-dynamics-simulations-with-gromacs-2020/
> .
> > I am running mdrun  with the suggested options:
> > "-pin on -nb gpu -bonded gpu -pme gpu -npme 1 -nstlist 400" on 4 GPUs and
> > 28 CPUs with "-ntmpi 4 -ntomp 7".
> >
> > I am currently running a membrane system with a transmembrane protein in
> > water solvent. I am using the following settings for COM removal:
> >
> > comm_mode  = linear
> > comm_grps  = PROT_MEMB SOL_ION
> >
> > Here I choose to couple the the protein and the membrane from the advice
> in
> > this thread:
> >
> https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2016-September/108584.html
> >
> > Unfortunately, I still see a large drift in the z-dimension for the
> entire
> > membrane/protein group. Currently I have nstcalcenergy/nstcomm set to
> 100,
> > as decreasing them leads to poor performance. (Hopefully it is
> unnecessary
> > to set them to 1)
>
> Removing artificial contributions to COM motion does not mean that the
> entities cannot diffuse over time. Depending on the length of your
> simulation, drift in absolute position can be perfectly normal.
>
> -Justin
>
> > Does anyone have suggestions for avoiding the COM drift? I know this
> issue
> > has been discussed before (https://redmine.gromacs.org/issues/2867) but
> it
> > looks like it was resolved in earlier GROMACS versions. As a note, I am
> > using a CHARMM force field with CMAP dihedrals.
> >
> >
> > Here are some other potentially relevant mdp options (from CHARMM) in
> case
> > they help:
> >
> > integrator  = md
> > dt  = 0.002
> > nstcalcenergy   = 100
> > ;
> > cutoff-scheme   = Verlet
> > nstlist = 20
> > rlist   = 1.2
> > coulombtype = pme
> > rcoulomb= 1.2
> > vdwtype = Cut-off
> > vdw-modifier= Force-switch
> > rvdw_switch = 1.0
> > rvdw= 1.2
> > ;
> > tcoupl  = v-rescale
> > tc_grps = PROT_MEMB   SOL_ION
> > tau_t   = 1.01.0
> > ref_t   = 303.15 303.15
> > ;
> > pcoupl  = Berendsen
> > pcoupltype  = semiisotropic
> > tau_p   = 5.0
> > compressibility = 4.5e-5  4.5e-5
> > ref_p   = 1.0 1.0
> > ;
> > constraints = h-bonds
> > constraint_algorithm= LINCS
> > continuation= yes
> >
> > Best,
> > Dan
>
> --
> ==
>
> Justin A. Lemkul, Ph.D.
> Assistant Professor
> Office: 301 Fralin Hall
> Lab: 303 Engel Hall
>
> Virginia Tech Department of Biochemistry
> 340 West Campus Dr.
> Blacksburg, VA 24061
>
> jalem...@vt.edu | (540) 231-3129
> http://www.thelemkullab.com
>
> ==
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-requ...@gromacs.org.
>
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] GMX 2020 - COMM Removal Issue

[Daniel Kozuch]

Hello,

I am experimenting with GROMACS 2020. I have compiled the mpi threaded
version and am using the new settings
(GMX_GPU_DD_COMMS, GMX_GPU_PME_PP_COMMS, GMX_FORCE_UPDATE_DEFAULT_GPU) as
suggested on at the following link:
https://devblogs.nvidia.com/creating-faster-molecular-dynamics-simulations-with-gromacs-2020/.
I am running mdrun  with the suggested options:
"-pin on -nb gpu -bonded gpu -pme gpu -npme 1 -nstlist 400" on 4 GPUs and
28 CPUs with "-ntmpi 4 -ntomp 7".

I am currently running a membrane system with a transmembrane protein in
water solvent. I am using the following settings for COM removal:

comm_mode  = linear
comm_grps  = PROT_MEMB SOL_ION

Here I choose to couple the the protein and the membrane from the advice in
this thread:
https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2016-September/108584.html

Unfortunately, I still see a large drift in the z-dimension for the entire
membrane/protein group. Currently I have nstcalcenergy/nstcomm set to 100,
as decreasing them leads to poor performance. (Hopefully it is unnecessary
to set them to 1)

Does anyone have suggestions for avoiding the COM drift? I know this issue
has been discussed before (https://redmine.gromacs.org/issues/2867) but it
looks like it was resolved in earlier GROMACS versions. As a note, I am
using a CHARMM force field with CMAP dihedrals.


Here are some other potentially relevant mdp options (from CHARMM) in case
they help:

integrator  = md
dt  = 0.002
nstcalcenergy   = 100
;
cutoff-scheme   = Verlet
nstlist = 20
rlist   = 1.2
coulombtype = pme
rcoulomb= 1.2
vdwtype = Cut-off
vdw-modifier= Force-switch
rvdw_switch = 1.0
rvdw= 1.2
;
tcoupl  = v-rescale
tc_grps = PROT_MEMB   SOL_ION
tau_t   = 1.01.0
ref_t   = 303.15 303.15
;
pcoupl  = Berendsen
pcoupltype  = semiisotropic
tau_p   = 5.0
compressibility = 4.5e-5  4.5e-5
ref_p   = 1.0 1.0
;
constraints = h-bonds
constraint_algorithm= LINCS
continuation= yes

Best,
Dan
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] Lambda Weights from Expanded Ensemble Code

[Daniel Kozuch]

Okay, thanks, I created a redmine request: #3304

On Wed, Jan 15, 2020 at 11:23 PM Michael Shirts  wrote:

> The simulated tempering options haven't been as well tested as the
> hamiltonian expanded ensemble version.  The weights SHOULD be showing up in
> the column that says -nan, but clearly they aren't.  If you file a redmine
> issue, I may be able to take a look, but it might take a while to address.
>
> On Wed, Jan 15, 2020 at 8:52 PM Daniel Kozuch 
> wrote:
>
> > Hello,
> >
> > I am interested in using simulated tempering in GROMACS (2019.5) under
> the
> > expanded ensemble options. Is there a way to monitor the ensemble weights
> > as the simulation progresses? I think in theory they are supposed to be
> > printed out in the log file, but it is only printing 0, -nan, and inf:
> >
> > MC-lambda information
> >   N  Temp.(K)Count   G(in kT)  dG(in kT)
> > ...
> >  36  359.105  118   -nan   -nan <<
> >  37  366.880   96   -nan   -nan
> >  38  374.852  107   -nan   -nan
> >  39  383.026  129   -nan   -nan
> >  40  391.407  166   -nan   -nan
> >  41  400.000  199   -nan0.0
> >
> > Here are my relevant mdp settings:
> > simulated-tempering = yes
> > nstexpanded = 500
> > simulated-tempering-scaling = exponential
> > lmc-stats   = metropolis-transition
> > lmc-move= metropolis
> >
> > Any suggestions?
> >
> > Best,
> > Dan Kozuch
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-requ...@gromacs.org.
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-requ...@gromacs.org.
>
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] Lambda Weights from Expanded Ensemble Code

[Daniel Kozuch]

Hello,

I am interested in using simulated tempering in GROMACS (2019.5) under the
expanded ensemble options. Is there a way to monitor the ensemble weights
as the simulation progresses? I think in theory they are supposed to be
printed out in the log file, but it is only printing 0, -nan, and inf:

MC-lambda information
  N  Temp.(K)Count   G(in kT)  dG(in kT)
...
 36  359.105  118   -nan   -nan <<
 37  366.880   96   -nan   -nan
 38  374.852  107   -nan   -nan
 39  383.026  129   -nan   -nan
 40  391.407  166   -nan   -nan
 41  400.000  199   -nan0.0

Here are my relevant mdp settings:
simulated-tempering = yes
nstexpanded = 500
simulated-tempering-scaling = exponential
lmc-stats   = metropolis-transition
lmc-move= metropolis

Any suggestions?

Best,
Dan Kozuch
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] Incompatible subsytems for REMD when running on multiple nodes

[Daniel Kozuch]

Hello,

I am running GROMACS 2019.4 (with GPUs) using the following command on two
nodes (each with 28 processors, and 4 GPUs).

srun -n 56 gmx mdrun -s sim -cpi sim -append no -deffnm sim -plumed
plumed.dat -multidir $mydirs -replex 500 -ntomp 1

It starts fine, but when I restart I get the incompatible systems error
(see below). I looked at the following discussion:
https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2013-November/085739.html,
but using -append no does not produce numbered cpt files and I would like
to avoid wasting every run this happens for. Is there a more current
solution?


Initializing Replica Exchange
Repl  There are 56 replicas:
Multi-checking the number of atoms ... OK
Multi-checking the integrator ... OK
Multi-checking init_step+nsteps ...
init_step+nsteps is not equal for all subsystems
  subsystem 0: 33291099
  subsystem 1: 33291099
  subsystem 2: 33291099
  subsystem 3: 33291099
  subsystem 4: 33291099
  subsystem 5: 33291099
  subsystem 6: 33291099
  subsystem 7: 33291099
  subsystem 8: 33291099
  subsystem 9: 33291000
  subsystem 10: 33291000
  subsystem 11: 33291000
  subsystem 12: 33291000
  subsystem 13: 33291099
  subsystem 14: 33291099
  subsystem 15: 33291000
  subsystem 16: 33291000
  subsystem 17: 33291000
  subsystem 18: 33291000
  subsystem 19: 33291000
  subsystem 20: 33291000
  subsystem 21: 33291099
  subsystem 22: 33291099
  subsystem 23: 33291099
  subsystem 24: 33291099
  subsystem 25: 33291099
  subsystem 26: 33291099
  subsystem 27: 33291000
  subsystem 28: 33291000
  subsystem 29: 33291099
  subsystem 30: 33291039
  subsystem 31: 33291039
  subsystem 32: 33291039
  subsystem 33: 33291039
  subsystem 34: 33291039
  subsystem 35: 33291039
  subsystem 36: 33291039
  subsystem 37: 33291039
  subsystem 38: 33291039
  subsystem 39: 33291039
  subsystem 40: 33291039
  subsystem 41: 33291039
  subsystem 42: 33291039
  subsystem 43: 33291039
  subsystem 44: 33291039
  subsystem 45: 33291000
  subsystem 46: 33291000
  subsystem 47: 33291039
  subsystem 48: 33291039
  subsystem 49: 33291039
  subsystem 50: 33291039
  subsystem 51: 33291039
  subsystem 52: 33291049
  subsystem 53: 33291049
  subsystem 54: 33291049
  subsystem 55: 33291049

---
Program: gmx mdrun, version 2019.4
Source file: src/gromacs/gmxlib/network.cpp (line 745)
MPI rank:0 (out of 56)

Fatal error:
The 56 subsystems are not compatible
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] OPLS AA/M pdb2gmx error with inter

[Daniel Kozuch]

I am having a problem similar to that mentioned in a previous thread (
https://www.mail-archive.com/gromacs.org_gmx-users@maillist.sys.kth.se/msg35369.html),
but I could not find a solution from that discussion. I am using pdb2gmx
with the flag -inter and the OPLS AA/M force field:

> gmx pdb2gmx -f ala.pdb -o ala.gro -inter
>> 23: OPLS-AA/M all-atom force field (2015 aminoacid dihedrals)
>>  7: None
>> 1: ZWITTERION_NH3+ (only use with zwitterions containing exactly one
residue)
>>  1: ZWITTERION_COO- (only use with zwitterions containing exactly one
residue)

Fatal error:
tpA = 49549, i= 4 in print_atoms

The input structure is a pdb file from ALA from AmberTools (below) and I
have not modified the forcefield files. Was there any solution to this
error?

Best regards,
Dan

ATOM  1  N   ALA 1   3.326   1.548  -0.000  1.00  0.00
ATOM  2  H   ALA 1   3.909   0.724  -0.000  1.00  0.00
ATOM  3  CA  ALA 1   3.970   2.846  -0.000  1.00  0.00
ATOM  4  HA  ALA 1   3.672   3.400  -0.890  1.00  0.00
ATOM  5  CB  ALA 1   3.577   3.654   1.232  1.00  0.00
ATOM  6  HB1 ALA 1   3.877   3.116   2.131  1.00  0.00
ATOM  7  HB2 ALA 1   4.075   4.623   1.206  1.00  0.00
ATOM  8  HB3 ALA 1   2.497   3.801   1.241  1.00  0.00
ATOM  9  C   ALA 1   5.486   2.705  -0.000  1.00  0.00
ATOM 10  O   ALA 1   6.009   1.593  -0.000  1.00  0.00
TER
END
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] tune_pme error with GROMACS 2018

[Daniel Kozuch]

Carsten,

I was hoping yo use tune_pme to optimize a REMD job across multiple nodes,
so I think I need it compiled with MPI (please let me know if that logic is
incorrect).
I am indeed running on GPU nodes (I tested mdrun on the nodes and and there
was no issue).

Best,
Dan

On Mon, Feb 12, 2018 at 8:32 AM, Kutzner, Carsten <ckut...@gwdg.de> wrote:

> Hi Dan,
>
> > On 11. Feb 2018, at 20:13, Daniel Kozuch <dan.koz...@gmail.com> wrote:
> >
> > Hello,
> >
> > I was recently trying to use the tune_pme tool with GROMACS 2018 with the
> > following command:
> >
> > gmx tune_pme -np 84 -s my_tpr.tpr -mdrun 'gmx mdrun’
> Maybe you need to compile gmx without MPI (so that gmx tune_pme
> is a serial executable), and a separate mdrun which naturally
> needs to be MPI-enabled (the latter you pass via -mdrun argument).
>
> >
> > but I'm getting the following error:
> >
> > "Fatal error:
> > Cannot execute mdrun. Please check benchtest.log for problems!"
> >
> > Unfortunately benchtest.log is an empty file. I saw that there were some
> > similar problems with GROMACS 5. Is this still an issue?
> What output is there on the command line?
>
> Are you running on GPU nodes? Are all shared libraries needed by
> mdrun actually found?
>
> Best,
>   Carsten
>
>
> >
> > Thanks,
> > Dan
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at http://www.gromacs.org/
> Support/Mailing_Lists/GMX-Users_List before posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-requ...@gromacs.org.
>
> --
> Dr. Carsten Kutzner
> Max Planck Institute for Biophysical Chemistry
> Theoretical and Computational Biophysics
> Am Fassberg 11, 37077 Goettingen, Germany
> Tel. +49-551-2012313, Fax: +49-551-2012302
> http://www.mpibpc.mpg.de/grubmueller/kutzner
> http://www.mpibpc.mpg.de/grubmueller/sppexa
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/
> Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-requ...@gromacs.org.
>
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] Gromacs 2018 and GPU PME

[Daniel Kozuch]

Szilárd,

If I may jump in on this conversation, I am having the reverse problem
(which I assume others may encounter also) where I am attempting a large
REMD run (84 replicas) and I have access to say 12 GPUs and 84 CPUs.

Basically I have less GPUs than simulations. Is there a logical approach to
using gputasks and other new options in GROMACS 2018 for this setup? I read
through the available documentation,but as you mentioned it seems to be
targeted for a single-GPU runs.

Thanks so much,
Dan



On Fri, Feb 9, 2018 at 10:27 AM, Szilárd Páll 
wrote:

> On Fri, Feb 9, 2018 at 4:25 PM, Szilárd Páll 
> wrote:
>
> > Hi,
> >
> > First of all,have you read the docs (admittedly somewhat brief):
> > http://manual.gromacs.org/documentation/2018/user-guide/
> > mdrun-performance.html#types-of-gpu-tasks
> >
> > The current PME GPU was optimized for single-GPU runs. Using multiple
> GPUs
> > with PME offloaded works, but this mode hasn't been an optimization
> target
> > and it will often not give very good performance. Using multiple GPUs
> > requires a separate PME rank (as you have realized), only one can be used
> > (as we don't support PME decomposition on GPUs) and it comes some
> > inherent scaling drawbacks. For this reason, unless you _need_ your
> single
> > run to be as fast as possible, you'll be better off running multiple
> > simulations side-by side.
> >
>
> PS: You can of course also run on two GPUs and run two simulations
> side-by-side (on half of the cores for each) to improve the overall
> aggregate throughput you get out of the hardware.
>
>
> >
> > A few tips for tuning the performance of a multi-GPU run with PME
> offload:
> > * expect to get at best 1.5 scaling to 2 GPUs (rarely 3 if the tasks
> allow)
> > * generally it's best to use about the same decomposition that you'd use
> > with nonbonded-only offload, e.g. in your case 6-8 ranks
> > * map the GPU task alone or at most together with 1 PP rank to a GPU,
> i.e.
> > use the new -gputasks option
> > e.g. for your case I'd expect the following to work ~best:
> > gmx mdrun -v -deffnm md -pme gpu -nb gpu -ntmpi 8 -ntomp 6 -npme 1
> > -gputasks 0001
> > or
> > gmx mdrun -v -deffnm md -pme gpu -nb gpu -ntmpi 8 -ntomp 6 -npme 1
> > -gputasks 0011
> >
> >
> > Let me know if that gave some improvement.
> >
> > Cheers,
> >
> > --
> > Szilárd
> >
> > On Fri, Feb 9, 2018 at 8:51 AM, Gmx QA  wrote:
> >
> >> Hi list,
> >>
> >> I am trying out the new gromacs 2018 (really nice so far), but have a
> few
> >> questions about what command line options I should specify, specifically
> >> with the new gnu pme implementation.
> >>
> >> My computer has two CPUs (with 12 cores each, 24 with hyper threading)
> and
> >> two GPUs, and I currently (with 2018) start simulations like this:
> >>
> >> $ gmx mdrun -v -deffnm md -pme gpu -nb gpu -ntmpi 2 -npme 1 -ntomp 24
> >> -gpu_id 01
> >>
> >> this works, but gromacs prints the message that 24 omp threads per mpi
> >> rank
> >> is likely inefficient. However, trying to reduce the number of omp
> threads
> >> I see a reduction in performance. Is this message no longer relevant
> with
> >> gpu pme or am I overlooking something?
> >>
> >> Thanks
> >> /PK
> >> --
> >> Gromacs Users mailing list
> >>
> >> * Please search the archive at http://www.gromacs.org/Support
> >> /Mailing_Lists/GMX-Users_List before posting!
> >>
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >> * For (un)subscribe requests visit
> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-requ...@gromacs.org.
> >>
> >
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/
> Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-requ...@gromacs.org.
>
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] Volume-Temperature Replica Exchange

[Daniel Kozuch]

Hello,

I am performing constant pressure replica exchange across a phase
transition, and as one might expect the associated change in volume is
causing exchange issues and many of my replicas are not efficiently
crossing the phase transition.

I noticed some papers that claim volume-temperature replica exchange
(VTREMD) can help solve this issue (https://doi.org/10.1063/1.1652015 and
https://doi.org/10.1016/j.cpc.2014.11.021). The main idea is to have a grid
of replicas in volume-temperature space that can exchange. Dietmar Paschek
published a code some time ago for performing VTREMD with GROMACS 3, but I
have not seen any recent activity.

My question: is there an easy way of implementing VTREMD with recent
versions of GROMACS? If not, is there a better way of tackling this issue?

Best regards,
Dan Kozuch
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] Dynamic Load Balancing Crash

[Daniel Kozuch]

Hello,

I recently started experiencing a error with GROMACS 2016.3 during a
replica exchange simulation with 80 replicas, 480 cpus, and 40 GPUs:

Assertion failed:
Condition: comm->cycl_n[ddCyclStep] > 0
When we turned on DLB, we should have measured cycles

The simulation then crashes. I turned off DLB with the flag "-dlb no" and
the error did not resurface so I have to assume it really is the DLB
causing the issue.

Does anyone know what might be causing this?

Thanks,
Dan
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] Free Energy Calculations with Position Restraints

[Daniel Kozuch]

Hello all,

Is there a way to use the free energy code with position restraints
(similar to the way that the free energy code interacts with the pull
code)? From the manual all I can see that might be relevant is
"restraint-lambdas" but that is apparently only for "dihedral restraints,
and the pull code restraints".

I would like to use position restraints instead of other methods (like pull
or umbrella) because I am using the position restraints to maintain an ice
layer with many atoms while also using NPT ensemble.

Any suggestions are appreciated.

Thanks,
Dan
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] Using the Pull Code to Restrain an Ice Layer

[Daniel Kozuch]

Hello,

I am attempting to restrain an ice layer in a system with liquid water. I
initially considered using position restraints, but it seems like GROMACS
has a few quirks that make that difficult: you have to create a new .itp
and define the crystal water as different from the liquid water, then use
constraints instead of settle, etc.

So I was wondering, is there was a way to achieve this (restraint of the
ice layer) with the pull code? This would be extra helpful since I want to
use the free energy code also to slowly relax the restraint and allow the
ice to become liquid water. This requires the restraint of many atoms
though, not just one center of mass, so I don't know if this is possible.

Any suggestions are welcome, and apologies if this is a duplicate (I had
trouble with the mailing system).

Thanks,
Dan
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] Using the Pull Code to Restrain Ice

[Daniel Kozuch]

Hello,

I am attempting to restrain an ice sheet in a system with liquid water. I
initially considered using position restraints, but it seems like GROMACS
has a few quirks that make that difficult: you have to create a new .itp
and define the crystal water as different from the liquid water and then
use constraints instead of settle, etc.

I was hoping that the pull code might be an easier alternative by using the
initial coordinates as reference coordinates (pull-coord1-groups= 0 1) and
pull-geometry=distance. This has the added bonus of the pull code allowing
for slow tuning of the force constant (via the free energy code) which I
would like to use.

However, this of course gives lots of warnings about using absolute
reference coordinates and artifacts from translation/rotation. Is there a
good way to go about doing this?

I know ideally the pull code would use some other part of the system as a
reference, but I would like to keep pressure coupling and I cannot do that
with a reference molecule since I would need to use
pull-geometry=direction-periodic which does not allow for changes in the
box size.

Any suggestions are welcome.

Thanks,
Dan
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] Pull Code: direction-periodic with 3 dimensions

[Daniel Kozuch]

Hello,

I am using a pull code to increase the end-to-end distance of a protein
(included below). I am using direction-periodic and would like the distance
between the COM groups to be calculated in three dimensions. However,
setting pull_coord1_dim = Y Y Y appears to have no effect and the distance
printed to the pull file is still only in the pull_vec dimension (as
verified by checking with gmx distance). Is there any way to use
pull_coord1_dim = Y Y Y with direction-periodic?

pull= yes
pull_ngroups= 2
pull_ncoords= 1
pull_group1_name= Chain_Begin
pull_group2_name= Chain_End
pull_coord1_type= umbrella  ; harmonic biasing force
pull_coord1_geometry= direction-periodic
pull_coord1_groups  = 1 2
pull_coord1_dim = Y Y Y
pull_coord1_rate= 0.0
pull_coord1_k   = 5000  ; kJ mol^-1 nm^-2
pull_coord1_start   = no   ; define initial COM distance > 0
pull_coord1_init = 3.5
pull_coord1_vec = 0 0 1


Thanks,
Dan
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] Umbrella Sampling with Direction-Periodic

[Daniel Kozuch]

Hello,

I am using a pull code with geometry=direction-periodic and attempting to
use gmx wham to construct the free energy. The pulling code is doing what I
would like it to, but as might be expected from direction-periodic, when
the pull distance is more than half the box length the distance is written
as negative.

Is there some way to have the pull files written with positive (greater
than half the box length) distance so that gmx wham will work properly? For
example, if the box is 10 nm long, I would like the pull distance to be
written as 7 nm, not -3 nm.

I have tried editing the pullx.xvg files, but as the .tpr files must still
include the original shortest periodic distance, this does not correct the
issue.

Thanks,
Dan
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] Umbrella Sampling with Direction-Periodic Pulling

[Daniel Kozuch]

Hello,

I am using a pull code with geometry=direction-periodic and attempting to
use gmx wham to construct the free energy. I believe the pulling code is
doing what I would like it to, but as might be expected from
direction-periodic, when the pull distance is more than half the box length
the distance is written as negative.

Is there some way to have the pull files written with positive (greater
than half the box length) distance so that gmx wham will work properly? For
example, if the box is 10 nm long, I would like the pull distance to be
written as 7 nm, not -3 nm. Or is the best option to simply edit the pullx
files?

Thanks,
Dan
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] Non-periodic COM Pulling

[Daniel Kozuch]

Justin,

It was my understanding that direction-periodic would not allow for the NPT
ensemble (which I would like to use if possible, and why I did not use it
in the first place). Is there a way around this?

Thanks,
Dan

On Thu, Jul 13, 2017 at 8:58 AM, Justin Lemkul <jalem...@vt.edu> wrote:

>
>
> On 7/12/17 3:05 PM, Daniel Kozuch wrote:
> > Hello,
> >
> > Is it possible to do non-periodic COM pulling using the distance function
> > in GMX 5.14 (i.e. where the distance between the two groups is calculated
> > ignoring pbc)?
> >
>
> No, but this is what direction-periodic is for.
>
> -Justin
>
> > In the tutorials/online the solution seems to be to simply use a box
> twice
> > the size of the largest pulling distance, but that would be very
> > computationally expensive for me.
> >
> > Best,
> > Dan
> >
>
> --
> ==
>
> Justin A. Lemkul, Ph.D.
> Ruth L. Kirschstein NRSA Postdoctoral Fellow
>
> Department of Pharmaceutical Sciences
> School of Pharmacy
> Health Sciences Facility II, Room 629
> University of Maryland, Baltimore
> 20 Penn St.
> Baltimore, MD 21201
>
> jalem...@outerbanks.umaryland.edu | (410) 706-7441
> http://mackerell.umaryland.edu/~jalemkul
>
> ==
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/
> Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-requ...@gromacs.org.
>
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] Non-periodic COM Pulling

[Daniel Kozuch]

Hello,

Is it possible to do non-periodic COM pulling using the distance function
in GMX 5.14 (i.e. where the distance between the two groups is calculated
ignoring pbc)?

In the tutorials/online the solution seems to be to simply use a box twice
the size of the largest pulling distance, but that would be very
computationally expensive for me.

Best,
Dan
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] Legacy Test Failure with GMX 5.1.4

[Daniel Kozuch]

Hello,

I am recompiling GROMACS on a new compute node and I am getting a unit test
failure (shown below). I am compiling with GNU 4.8.5 and the following
cmake commands:
cmake .. -DCMAKE_INSTALL_PREFIX=[redacted] -DGMX_MPI=on -DGMX_GPU=off
-DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_THREAD_MPI=off.

I am unsure how to fix this error, does anyone have any suggestions?

Best,
Dan

 Start  1: TestUtilsUnitTests
 1/26 Test  #1: TestUtilsUnitTests ...   Passed0.57 sec
  Start  2: GmxlibTests
 2/26 Test  #2: GmxlibTests ..   Passed0.56 sec
  Start  3: MdlibUnitTest
 3/26 Test  #3: MdlibUnitTest    Passed0.56 sec
  Start  4: CommandLineUnitTests
 4/26 Test  #4: CommandLineUnitTests .   Passed0.58 sec
  Start  5: FFTUnitTests
 5/26 Test  #5: FFTUnitTests .   Passed0.63 sec
  Start  6: MathUnitTests
 6/26 Test  #6: MathUnitTests    Passed0.56 sec
  Start  7: RandomUnitTests
 7/26 Test  #7: RandomUnitTests ..   Passed0.56 sec
  Start  8: OnlineHelpUnitTests
 8/26 Test  #8: OnlineHelpUnitTests ..   Passed0.57 sec
  Start  9: OptionsUnitTests
 9/26 Test  #9: OptionsUnitTests .   Passed0.56 sec
  Start 10: UtilityUnitTests
10/26 Test #10: UtilityUnitTests .   Passed0.57 sec
  Start 11: FileIOTests
11/26 Test #11: FileIOTests ..   Passed0.56 sec
  Start 12: SimdUnitTests
12/26 Test #12: SimdUnitTests    Passed0.58 sec
  Start 13: LegacyToolsTests
13/26 Test #13: LegacyToolsTests .***Failed0.56 sec
[redacted] no hfi units are active (err=23)
[==] Running 12 tests from 2 test cases.
[--] Global test environment set-up.
[--] 6 tests from NoFatalErrorWhenWritingFrom/GmxTraj
[ RUN  ] NoFatalErrorWhenWritingFrom/GmxTraj.WithDifferentInputFormats/0
Group 0 ( System) has 6 elements
Group 1 (  Water) has 6 elements
Group 2 (SOL) has 6 elements
Select a group:
---
Program legacy-tools-test, VERSION 5.1.4
Source code file: [redacted]/gromacs-5.1.4/src/gromacs/topology/index.cpp,
line: 917

Fatal error:
Cannot read from input
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
---

Halting program legacy-tools-test
--
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--

  Start 14: GmxPreprocessTests
14/26 Test #14: GmxPreprocessTests ...   Passed0.66 sec
  Start 15: CorrelationsTest
15/26 Test #15: CorrelationsTest .   Passed0.84 sec
  Start 16: AnalysisDataUnitTests
16/26 Test #16: AnalysisDataUnitTests    Passed0.59 sec
  Start 17: SelectionUnitTests
17/26 Test #17: SelectionUnitTests ...   Passed0.69 sec
  Start 18: TrajectoryAnalysisUnitTests
18/26 Test #18: TrajectoryAnalysisUnitTests ..   Passed0.89 sec
  Start 19: MdrunTests
19/26 Test #19: MdrunTests ...   Passed2.10 sec
  Start 20: MdrunMpiTests
20/26 Test #20: MdrunMpiTests    Passed0.96 sec
  Start 21: regressiontests/simple
21/26 Test #21: regressiontests/simple ...   Passed   47.05 sec
  Start 22: regressiontests/complex
22/26 Test #22: regressiontests/complex ..   Passed  149.94 sec
  Start 23: regressiontests/kernel
23/26 Test #23: regressiontests/kernel ...   Passed  436.31 sec
  Start 24: regressiontests/freeenergy
24/26 Test #24: regressiontests/freeenergy ...   Passed   35.35 sec
  Start 25: regressiontests/pdb2gmx
25/26 Test #25: regressiontests/pdb2gmx ..   Passed  109.15 sec
  Start 26: regressiontests/rotation
26/26 Test #26: regressiontests/rotation .   Passed   36.78 sec

96% tests passed, 1 tests failed out of 26

Label Time Summary:
GTest =  10.52 sec
IntegrationTest   =   2.65 sec
MpiIntegrationTest=   0.96 sec
UnitTest  =  10.52 sec

Total Test time (real) = 828.73 sec

The following tests FAILED:
 13 - LegacyToolsTests (Failed)
Errors while running CTest
make[3]: *** [CMakeFiles/run-ctest] Error 8
make[2]: *** [CMakeFiles/run-ctest.dir/all] Error 2
make[1]: *** [CMakeFiles/check.dir/rule] Error 2
make: *** [check] Error 2
-- 
Gromacs Users mailing list

* Please 

[gmx-users] GMX GPU Rest Time

[Daniel Kozuch]

Hello,

I recently changed the number of cpus I was pairing with each gpu and I
noticed a significant slowdown, more than I would have expected simply due
to a reduction in the number of cpus.

>From the log file it appears that the GPU is resting for a large amount of
time. Is there something I can do about this?

I have attached parts of the log file. For reference this is a REMD
simulation with 60 replicas on 360 cpus and 60 gpus. I have set the local
variable OMP_NUM_THREADS to six in order to assign 6 cpus to each replica
and avoid domain decomposition for my small system (as recommend in an
earlier correspondence).

Any help is appreciated,
Dan

-

GROMACS:  gmx mdrun, VERSION 5.1.4
Executable:   /home/dkozuch/programs/gromacs_514_gpu/bin/gmx_514_gpu
Data prefix:  /home/dkozuch/programs/gromacs_514_gpu
Command line:
  gmx_514_gpu mdrun -v -deffnm 1msi_eq -multidir 1 2 3 4 5 6 7 8 9 10 11 12
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 -pin on

GROMACS version:VERSION 5.1.4
Precision:  single
Memory model:   64 bit
MPI library:MPI
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 32)
GPU support:enabled
OpenCL support: disabled
invsqrt routine:gmx_software_invsqrt(x)
SIMD instructions:  AVX2_256
FFT library:fftw-3.3.4-sse2-avx
RDTSCP usage:   enabled
C++11 compilation:  disabled
TNG support:enabled
Tracing support:disabled
Built on:   Mon May 22 18:29:21 EDT 2017
Built by:   dkoz...@tigergpu.princeton.edu [CMAKE]
Build OS/arch:  Linux 3.10.0-514.16.1.el7.x86_64 x86_64
Build CPU vendor:   GenuineIntel
Build CPU brand:Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
Build CPU family:   6   Model: 79   Stepping: 1
Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt
lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd
rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
C compiler: /usr/bin/cc GNU 4.8.5
C compiler flags:-march=core-avx2-Wextra
-Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -Wall
-Wno-unused -Wunused-value -Wunused-parameter  -O3 -DNDEBUG
-funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
C++ compiler:   /usr/bin/c++ GNU 4.8.5
C++ compiler flags:  -march=core-avx2-Wextra
-Wno-missing-field-initializers -Wpointer-arith -Wall -Wno-unused-function
 -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
Boost version:  1.53.0 (external)
CUDA compiler:  /usr/local/cuda-8.0/bin/nvcc nvcc: NVIDIA (R) Cuda
compiler driver;Copyright (c) 2005-2016 NVIDIA Corporation;Built on
Sun_Sep__4_22:14:01_CDT_2016;Cuda compilation tools, release 8.0, V8.0.44
CUDA compiler
flags:-gencode;arch=compute_20,code=sm_20;-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_60,code=compute_60;-gencode;arch=compute_61,code=compute_61;-use_fast_math;;
;-march=core-avx2;-Wextra;-Wno-missing-field-initializers;-Wpointer-arith;-Wall;-Wno-unused-function;-O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast;-Wno-array-bounds;
CUDA driver:8.0
CUDA runtime:   8.0


Number of logical cores detected (28) does not match the number reported by
OpenMP (6).
Consider setting the launch configuration manually!

Running on 15 nodes with total 420 cores, 420 logical cores, 24 compatible
GPUs
  Cores per node:   28
  Logical cores per node:   28
  Compatible GPUs per node:  0 -  4
  Different nodes have different type(s) and/or order of GPUs
Hardware detected on host tiger-i20g2 (the node of MPI rank 4):
  CPU info:
Vendor: GenuineIntel
Brand:  Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
Family:  6  model: 79  stepping:  1
CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt
lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd
rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
SIMD instructions most likely to fit this hardware: AVX2_256
SIMD instructions selected at GROMACS compile time: AVX2_256
  GPU info:
Number of GPUs detected: 4
#0: NVIDIA Tesla P100-PCIE-16GB, compute cap.: 6.0, ECC: yes, stat:
compatible
#1: NVIDIA Tesla P100-PCIE-16GB, compute cap.: 6.0, ECC: yes, stat:
compatible
#2: NVIDIA Tesla P100-PCIE-16GB, compute cap.: 6.0, ECC: yes, stat:
compatible
#3: NVIDIA Tesla P100-PCIE-16GB, compute cap.: 6.0, ECC: yes, stat:
compatible

This is simulation 4 out of 60 running as a composite GROMACS
multi-simulation job. Setup for this simulation:

Using 1 MPI process
Using 6 OpenMP threads

4 compatible 

Re: [gmx-users] em and nvt problem

[Daniel Kozuch]

It would be helpful if you include the output of the em run and the log
file for the nvt run.

Best,
Dan

On Thu, May 25, 2017 at 7:12 AM, Kashif  wrote:

> Hi
> Whenever I tried to simulate one of my docked complex, the energy
> minimization step converged very fast and complete at 112 steps. And when I
> proceed with NVT equilibration I got split structures like 
> step4b_n6.pdb
> step4c_n6.pdb
> step5b_n6.pdb
> step5c_n6.pdb
>
> I have already docked the drug with my protein using different tool in
> order to get different docked pose of drug. So that I could get different
> topology file and coordinate file of the drug from PRODRG server.
> But the same problem encounters.
> Some time I got the error like .
>
> Fatal error:
> 1 particles communicated to PME node 5 are more than 2/3 times the cut-off
> out of the domain decomposition cell of their charge group in dimension y.
> This usually means that your system is not well equilibrated.
> 
>
> kindly help
>
> regards
> kashif
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/
> Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-requ...@gromacs.org.
>
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] Poor GPU Performance with GROMACS 5.1.4

[Daniel Kozuch]

Hi Marcelo,

That sounds reasonable depending on your time-step and other factors, but I
have not attempted to run with more than one job for GPU.
Maybe Mark can comment more.

Best,
Dan

On Thu, May 25, 2017 at 8:09 AM, Marcelo Depólo 
wrote:

> Hi,
>
>
> I had the same struggle benchmarking a similar system last week. Just for
> curiosity, could you tell us the performance you get when sharing your GPU
> with multiple jobs?
>
> In my case (6k atoms + Reaction field + 8 cores 2.2Ghz + TitanX Pascal),
> I've got ~440 ns/day. However, I get ~280 ns/day running 2 jobs and sharing
> the GPU.
>
> Cheers!
> --
> Marcelo
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/
> Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-requ...@gromacs.org.
>
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] Poor GPU Performance with GROMACS 5.1.4

[Daniel Kozuch]

I apologize for the confusion, but I found my error. I was failing to
request a certain number of cpus-per-task and the scheduler was having
issues assigning the threads because of this. Speed is now at ~400 ns/day
with a 3 fs timestep which seems reasonable.

Thanks for all the help,
Dan

On Wed, May 24, 2017 at 9:48 PM, Daniel Kozuch <dkoz...@princeton.edu>
wrote:

> Szilárd,
>
> I think I must be misunderstanding your advice. If I remove the domain
> decomposition and set pin on as suggested by Mark, using:
>
> gmx_gpu mdrun -deffnm my_tpr -dd 1 -pin on
>
> Then I get very poor performance and the following error:
>
> NOTE: Affinity setting for 6/6 threads failed. This can cause performance
> degradation.
>   If you think your settings are correct, ask on the gmx-users list.
>
> I am running only one rank and using 6 threads (I do not want to use all
> the available 28 cores on the node because I hope to run multiple of these
> jobs per node in the near future).
>
> Thanks for the help,
> Dan
>
>
> 
> -
> Log File:
>
> GROMACS version:VERSION 5.1.4
> Precision:  single
> Memory model:   64 bit
> MPI library:MPI
> OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 32)
> GPU support:enabled
> OpenCL support: disabled
> invsqrt routine:gmx_software_invsqrt(x)
> SIMD instructions:  AVX2_256
> FFT library:fftw-3.3.4-sse2-avx
> RDTSCP usage:   enabled
> C++11 compilation:  disabled
> TNG support:enabled
> Tracing support:disabled
> Built on:   Mon May 22 18:29:21 EDT 2017
> Built by:   dkoz...@tigergpu.princeton.edu [CMAKE]
> Build OS/arch:  Linux 3.10.0-514.16.1.el7.x86_64 x86_64
> Build CPU vendor:   GenuineIntel
> Build CPU brand:Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
> Build CPU family:   6   Model: 79   Stepping: 1
> Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt
> lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd
> rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
> C compiler: /usr/bin/cc GNU 4.8.5
> C compiler flags:-march=core-avx2-Wextra
> -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -Wall
> -Wno-unused -Wunused-value -Wunused-parameter  -O3 -DNDEBUG
> -funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
> C++ compiler:   /usr/bin/c++ GNU 4.8.5
> C++ compiler flags:  -march=core-avx2-Wextra
> -Wno-missing-field-initializers -Wpointer-arith -Wall
> -Wno-unused-function  -O3 -DNDEBUG -funroll-all-loops
> -fexcess-precision=fast  -Wno-array-bounds
> Boost version:  1.53.0 (external)
> CUDA compiler:  /usr/local/cuda-8.0/bin/nvcc nvcc: NVIDIA (R) Cuda
> compiler driver;Copyright (c) 2005-2016 NVIDIA Corporation;Built on
> Sun_Sep__4_22:14:01_CDT_2016;Cuda compilation tools, release 8.0, V8.0.44
> CUDA compiler flags:-gencode;arch=compute_20,code=sm_20;-gencode;arch=comp
> ute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-
> gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_
> 50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-
> gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_
> 61,code=sm_61;-gencode;arch=compute_60,code=compute_60;-
> gencode;arch=compute_61,code=compute_61;-use_fast_math;;
> ;-march=core-avx2;-Wextra;-Wno-missing-field-initializers;-
> Wpointer-arith;-Wall;-Wno-unused-function;-O3;-DNDEBUG;-
> funroll-all-loops;-fexcess-precision=fast;-Wno-array-bounds;
> CUDA driver:8.0
> CUDA runtime:   8.0
>
>
> Number of logical cores detected (28) does not match the number reported
> by OpenMP (1).
> Consider setting the launch configuration manually!
>
> Running on 1 node with total 28 logical cores, 1 compatible GPU
> Hardware detected on host tiger-i23g14 (the node of MPI rank 0):
>   CPU info:
> Vendor: GenuineIntel
> Brand:  Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
> Family:  6  model: 79  stepping:  1
> CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt
> lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd
> rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
> SIMD instructions most likely to fit this hardware: AVX2_256
> SIMD instructions selected at GROMACS compile time: AVX2_256
>   GPU info:
> Number of GPUs detected: 1
> #0: NVIDIA Tesla P100-PCIE-16GB, compute cap.: 6.0, ECC: yes, stat:
> compatible
>
> Using 1 MPI process
> Using 6 OpenMP threads
>
> 1 compatible GPU is present, with ID 0
> 1 GPU auto-selected for this run.
> Mapping of GPU ID to t

Re: [gmx-users] Poor GPU Performance with GROMACS 5.1.4

[Daniel Kozuch]

Szilárd,

I think I must be misunderstanding your advice. If I remove the domain
decomposition and set pin on as suggested by Mark, using:

gmx_gpu mdrun -deffnm my_tpr -dd 1 -pin on

Then I get very poor performance and the following error:

NOTE: Affinity setting for 6/6 threads failed. This can cause performance
degradation.
  If you think your settings are correct, ask on the gmx-users list.

I am running only one rank and using 6 threads (I do not want to use all
the available 28 cores on the node because I hope to run multiple of these
jobs per node in the near future).

Thanks for the help,
Dan


-
Log File:

GROMACS version:VERSION 5.1.4
Precision:  single
Memory model:   64 bit
MPI library:MPI
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 32)
GPU support:enabled
OpenCL support: disabled
invsqrt routine:gmx_software_invsqrt(x)
SIMD instructions:  AVX2_256
FFT library:fftw-3.3.4-sse2-avx
RDTSCP usage:   enabled
C++11 compilation:  disabled
TNG support:enabled
Tracing support:disabled
Built on:   Mon May 22 18:29:21 EDT 2017
Built by:   dkoz...@tigergpu.princeton.edu [CMAKE]
Build OS/arch:  Linux 3.10.0-514.16.1.el7.x86_64 x86_64
Build CPU vendor:   GenuineIntel
Build CPU brand:Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
Build CPU family:   6   Model: 79   Stepping: 1
Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt
lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd
rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
C compiler: /usr/bin/cc GNU 4.8.5
C compiler flags:-march=core-avx2-Wextra
-Wno-missing-field-initializers
-Wno-sign-compare -Wpointer-arith -Wall -Wno-unused -Wunused-value
-Wunused-parameter  -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast
 -Wno-array-bounds
C++ compiler:   /usr/bin/c++ GNU 4.8.5
C++ compiler flags:  -march=core-avx2-Wextra
-Wno-missing-field-initializers
-Wpointer-arith -Wall -Wno-unused-function  -O3 -DNDEBUG -funroll-all-loops
-fexcess-precision=fast  -Wno-array-bounds
Boost version:  1.53.0 (external)
CUDA compiler:  /usr/local/cuda-8.0/bin/nvcc nvcc: NVIDIA (R) Cuda
compiler driver;Copyright (c) 2005-2016 NVIDIA Corporation;Built on
Sun_Sep__4_22:14:01_CDT_2016;Cuda compilation tools, release 8.0, V8.0.44
CUDA compiler flags:-gencode;arch=compute_20,code=sm_20;-gencode;arch=
compute_30,code=sm_30;-gencode;arch=compute_35,code=
sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=
compute_50,code=sm_50;-gencode;arch=compute_52,code=
sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=
compute_61,code=sm_61;-gencode;arch=compute_60,code=
compute_60;-gencode;arch=compute_61,code=compute_61;-use_fast_math;;
;-march=core-avx2;-Wextra;-Wno-missing-field-initializers;-Wpointer-arith;-
Wall;-Wno-unused-function;-O3;-DNDEBUG;-funroll-all-loops;-
fexcess-precision=fast;-Wno-array-bounds;
CUDA driver:8.0
CUDA runtime:   8.0


Number of logical cores detected (28) does not match the number reported by
OpenMP (1).
Consider setting the launch configuration manually!

Running on 1 node with total 28 logical cores, 1 compatible GPU
Hardware detected on host tiger-i23g14 (the node of MPI rank 0):
  CPU info:
Vendor: GenuineIntel
Brand:  Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
Family:  6  model: 79  stepping:  1
CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt
lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd
rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
SIMD instructions most likely to fit this hardware: AVX2_256
SIMD instructions selected at GROMACS compile time: AVX2_256
  GPU info:
Number of GPUs detected: 1
#0: NVIDIA Tesla P100-PCIE-16GB, compute cap.: 6.0, ECC: yes, stat:
compatible

Using 1 MPI process
Using 6 OpenMP threads

1 compatible GPU is present, with ID 0
1 GPU auto-selected for this run.
Mapping of GPU ID to the 1 PP rank in this node: 0

NOTE: GROMACS was configured without NVML support hence it can not exploit
  application clocks of the detected Tesla P100-PCIE-16GB GPU to
improve performance.
  Recompile with the NVML library (compatible with the driver used) or
set application clocks manually.


Using GPU 8x8 non-bonded kernels

Removing pbc first time

Overriding thread affinity set outside gmx_514_gpu

Pinning threads with an auto-selected logical core stride of 4

NOTE: Affinity setting for 6/6 threads failed. This can cause performance
degradation.
  If you think your settings are correct, ask on the gmx-users list.

Initializing LINear Constraint Solver

 R E A L   C Y C L E   A N D   T I M E   A C C O U N T I N G

On 1 MPI rank, each using 6 OpenMP threads

 Computing:  Num   Num  CallWall time Giga-Cycles
 

Re: [gmx-users] Poor GPU Performance with GROMACS 5.1.4

[Daniel Kozuch]

Thanks so much for the quick reply. That seems to have fixed the wait time
issues. Unfortunately, I'm still only getting ~300 ns/day for the benchmark
system (villin vsites, http://www.gromacs.org/GPU_acceleration), while the
website claims over 1000 ns/day.

I'm running on a NVIDIA Tesla P100-PCIE-16GB with 8 Xeon(R) CPU E5-2680 v4
@ 2.40GHz. I can see that the cpus are now under performing (324% used).
Any suggestions?


_


 R E A L   C Y C L E   A N D   T I M E   A C C O U N T I N G

On 2 MPI ranks, each using 4 OpenMP threads

 Computing:  Num   Num  CallWall time Giga-Cycles
 Ranks Threads  Count  (s) total sum%

-
 Domain decomp. 24   4001   4.402 84.517   3.2
 DD comm. load  24   3983   0.021  0.402   0.0
 DD comm. bounds24   3982   0.014  0.267   0.0
 Vsite constr.  24 11   3.330 63.929   2.4
 Neighbor search24   4001   7.495143.911   5.5
 Launch GPU ops.24 22   4.820 92.537   3.5
 Comm. coord.   24  96000   2.212 42.468   1.6
 Force  24 11  12.465239.335   9.1
 Wait + Comm. F 24 11   2.572 49.381   1.9
 PME mesh   24 11  59.323   1139.002  43.3
 Wait GPU nonlocal  24 11   0.483  9.282   0.4
 Wait GPU local 24 11   0.292  5.607   0.2
 NB X/F buffer ops. 24 392002   5.703109.491   4.2
 Vsite spread   24 101002   2.762 53.030   2.0
 Write traj.24  1   0.007  0.130   0.0
 Update 24 11   4.372 83.942   3.2
 Constraints24 11  23.858458.072  17.4
 Comm. energies 24  20001   0.146  2.803   0.1
 Rest   2.739 52.595   2.0

-
 Total137.015   2630.701 100.0

-
 Breakdown of PME mesh computation

-
 PME redist. X/F24 22   6.021115.598   4.4
 PME spread/gather  24 22  36.204695.123  26.4
 PME 3D-FFT 24 22  13.127252.036   9.6
 PME 3D-FFT Comm.   24 22   2.007 38.538   1.5
 PME solve Elec 24 11   0.541 10.392   0.4

-

   Core t (s)   Wall t (s)(%)
   Time:  444.060  137.015  324.1
 (ns/day)(hour/ns)
Performance:  315.2960.076
Finished mdrun on rank 0 Wed May 24 15:48:59 2017

On Wed, May 24, 2017 at 3:25 PM, Smith, Micholas D. <smit...@ornl.gov>
wrote:

> Try just using your equivalent of:
>
> mpirun -n 2 -npernode 2 gmx_mpi mdrun (your run stuff here) -ntomp 4
> -gpu_id 00
>
> That may speed it up.
>
> ===
> Micholas Dean Smith, PhD.
> Post-doctoral Research Associate
> University of Tennessee/Oak Ridge National Laboratory
> Center for Molecular Biophysics
>
> 
> From: gromacs.org_gmx-users-boun...@maillist.sys.kth.se <
> gromacs.org_gmx-users-boun...@maillist.sys.kth.se> on behalf of Daniel
> Kozuch <dkoz...@princeton.edu>
> Sent: Wednesday, May 24, 2017 3:08 PM
> To: gromacs.org_gmx-users@maillist.sys.kth.se
> Subject: [gmx-users] Poor GPU Performance with GROMACS 5.1.4
>
> Hello,
>
> I'm using GROMACS 5.1.4 on 8 CPUs and 1 GPU for a system of ~8000 atoms in
> a dodecahedron box, and I'm having trouble getting good performance out of
> the GPU. Specifically it appears that there is significant performance loss
> to wait times ("Wait + Comm. F" and "Wait GPU nonlocal"). I have pasted the
> relevant parts of the log file below. I suspect that I have set up my
> ranks/threads badly, but I am unsure where the issue is. I have tried
> changing the local variable OMP_NUM_THREADS from 1 to 2 per the note
> generated by GROMACS, but this severely slows down the simulation to the
> point where it takes 10 minutes to get a few picoseconds.
>
> I have tried browsing through the mailing lists, but I haven't found a
> solution to this particular problem.
>
> Any hel

[gmx-users] Poor GPU Performance with GROMACS 5.1.4

[Daniel Kozuch]

Hello,

I'm using GROMACS 5.1.4 on 8 CPUs and 1 GPU for a system of ~8000 atoms in
a dodecahedron box, and I'm having trouble getting good performance out of
the GPU. Specifically it appears that there is significant performance loss
to wait times ("Wait + Comm. F" and "Wait GPU nonlocal"). I have pasted the
relevant parts of the log file below. I suspect that I have set up my
ranks/threads badly, but I am unsure where the issue is. I have tried
changing the local variable OMP_NUM_THREADS from 1 to 2 per the note
generated by GROMACS, but this severely slows down the simulation to the
point where it takes 10 minutes to get a few picoseconds.

I have tried browsing through the mailing lists, but I haven't found a
solution to this particular problem.

Any help is appreciated,
Dan






GROMACS:  gmx mdrun, VERSION 5.1.4
Executable:   /home/dkozuch/programs/gromacs_514_gpu/bin/gmx_514_gpu
Data prefix:  /home/dkozuch/programs/gromacs_514_gpu
Command line:
  gmx_514_gpu mdrun -deffnm 1ucs_npt -ntomp 1

GROMACS version:VERSION 5.1.4
Precision:  single
Memory model:   64 bit
MPI library:MPI
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 32)
GPU support:enabled
OpenCL support: disabled
invsqrt routine:gmx_software_invsqrt(x)
SIMD instructions:  AVX2_256
FFT library:fftw-3.3.4-sse2-avx
RDTSCP usage:   enabled
C++11 compilation:  disabled
TNG support:enabled
Tracing support:disabled
Built on:   Mon May 22 18:29:21 EDT 2017
Built by:   dkoz...@tigergpu.princeton.edu [CMAKE]
Build OS/arch:  Linux 3.10.0-514.16.1.el7.x86_64 x86_64
Build CPU vendor:   GenuineIntel
Build CPU brand:Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
Build CPU family:   6   Model: 79   Stepping: 1
Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt
lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd
rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
C compiler: /usr/bin/cc GNU 4.8.5
C compiler flags:-march=core-avx2-Wextra
-Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -Wall
-Wno-unused -Wunused-value -Wunused-parameter  -O3 -DNDEBUG
-funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
C++ compiler:   /usr/bin/c++ GNU 4.8.5
C++ compiler flags:  -march=core-avx2-Wextra
-Wno-missing-field-initializers -Wpointer-arith -Wall -Wno-unused-function
 -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
Boost version:  1.53.0 (external)
CUDA compiler:  /usr/local/cuda-8.0/bin/nvcc nvcc: NVIDIA (R) Cuda
compiler driver;Copyright (c) 2005-2016 NVIDIA Corporation;Built on
Sun_Sep__4_22:14:01_CDT_2016;Cuda compilation tools, release 8.0, V8.0.44
CUDA compiler flags:-gencode;arch=compute_20,code=sm_20;-gencode;arch=comp
ute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-
gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_
50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-
gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_
61,code=sm_61;-gencode;arch=compute_60,code=compute_60;-
gencode;arch=compute_61,code=compute_61;-use_fast_math;;
;-march=core-avx2;-Wextra;-Wno-missing-field-initializers;-
Wpointer-arith;-Wall;-Wno-unused-function;-O3;-DNDEBUG;-
funroll-all-loops;-fexcess-precision=fast;-Wno-array-bounds;
CUDA driver:8.0
CUDA runtime:   8.0


Number of logical cores detected (28) does not match the number reported by
OpenMP (1).
Consider setting the launch configuration manually!

Running on 1 node with total 28 logical cores, 1 compatible GPU
Hardware detected on host tiger-i23g10 (the node of MPI rank 0):
  CPU info:
Vendor: GenuineIntel
Brand:  Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
Family:  6  model: 79  stepping:  1
CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt
lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd
rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
SIMD instructions most likely to fit this hardware: AVX2_256
SIMD instructions selected at GROMACS compile time: AVX2_256
  GPU info:
Number of GPUs detected: 1
#0: NVIDIA Tesla P100-PCIE-16GB, compute cap.: 6.0, ECC: yes, stat:
compatible


For optimal performance with a GPU nstlist (now 10) should be larger.
The optimum depends on your CPU and GPU resources.
You might want to try several nstlist values.
Changing nstlist from 10 to 25, rlist from 1.014 to 1.098

Initializing Domain Decomposition on 8 ranks
Dynamic load balancing: auto
Will sort the charge groups at every domain (re)decomposition
Initial maximum inter charge-group distances:
two-body bonded interactions: 0.417 nm, LJ-14, atoms 514 517
  multi-body bonded interactions: 0.417 nm, Proper Dih., atoms 514 517
Minimum cell size due to bonded