Re: [COOT] Wild refinement of waters
After more off-list correspondence, I further investigated the water refinement issue. I now think that the problem isn't what I thought it was. A few months ago, I had noticed how frustratingly long it took for Coot to start moving the atoms when RSRing a fragment into a cryo-EM map. I increased the initial step size and the RSR was noticeably quicker. Good. I tested that the new step size made sense for refinement of protein in an x-ray map - and it was. Good. However, what I didn't check was the most minimal case, i.e. what we have been discussing here: the refinement of just one (water) atom. And hence the problems you've been seeing (and I haven't). So today I changed the initial step size back to how it was calculated last year and the water refinement stabilized - and that is without needing soft harmonic-approximation NBC restraints. As of now, the initial step size depends on the number of atoms, so, I think, we can satisfy both scenarios. I have pushed a commit to the repo (7b7a7e2db0c6760c4467c6de0c8d2ac5364624c9) and the fix will be in the next release. Paul. To unsubscribe from the COOT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [COOT] Wild refinement of waters
On Thu, 2021-06-03 at 09:02 -0700, Philip Kiser wrote: > Thank you for spotting the problem and introducing the new restraint. I have > to say that it was easy to get in the > habit of creating poorly placed water molecules when filling difference map > peaks as the prior refinement method for > isolated waters would just move them into the nearby 2Fo-Fc peak. You can do that - that's reasonable - but it makes more sense (to me) to allow the protein to move in response to a badly placed water - and that stabilizes the refinement (in my hands). > Any residual placement errors or clashes would get fixed during reciprocal > space refinement and all was good. Yes, that's the idea with Coot's RSR. > But I will take your advice and change my ways. Well, refine with sphere refine rather than just one water and you should be good. And if not, the soft restraints will do the trick. Paul. To unsubscribe from the COOT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [COOT] Wild refinement of waters
Hi Paul, Thank you for spotting the problem and introducing the new restraint. I have to say that it was easy to get in the habit of creating poorly placed water molecules when filling difference map peaks as the prior refinement method for isolated waters would just move them into the nearby 2Fo-Fc peak. Any residual placement errors or clashes would get fixed during reciprocal space refinement and all was good. But I will take your advice and change my ways. Much appreciated, Philip On Thu, Jun 3, 2021 at 3:12 AM Paul Emsley wrote: > On Wed, 2021-06-02 at 15:56 +0100, Paul Emsley wrote: > > On Tue, 2021-06-01 at 20:45 +0100, David M Dranow wrote: > > > Does anyone have a decent set of refinement parameters for Coot 0.9.5 > in linux? My colleagues and I find that using > > > the "r" shortcut causes most waters to fly off and center on carbons, > sometimes as far away as 45 Å! I've tried to > > > change the overall weight and Lennard-Jones epsilon, and > Geman-McClure alpha and I find no combination that keeps > > > all > > > waters within the density that they already are in. I'd be happy to > provide a model and map to demonstrate this > > > phenomenon. Our current work-around is to use R, but this is not > ideal. If anyone has any idea how to fix this, we > > > would greatly appreciate any advice. > > > > > > > This reminds me of the exchange about the new refinement on CCP4BB some > time ago. Briefly, people reported problems > > that > > I wasn't seeing. > > > > So I'd like to see what you see. Either make a screencast of your > session or conjure up a coordinates file and a list > > of coot operations that makes the issue apparent. Until then, I am in > the dark. > > > > > > Thanks to files provided by Philip Kiser I now see the problem. > > First, one had to create an unorthodox model - one that included a bad > non-bonded contact distance to a water > (this is not something I do) > Second, one had to refine just the water, and not the residues to which it > was hydrogen-bonded or other atoms in > the environment > (this is not something I do either) > > By its nature, refinement of this sort will give a considerable kick to a > bad water. > I have therefore introduced an optional "Soft Mode" for non-bonded-contact > restraints. > set_refinement_use_soft_mode_nbc_restraints(flag) > > This is available now in the repo and will be in the next release. > > But for me, the lessons are: Don't create models with poorly placed waters > in the first place. But if you do, don't do > isolated refinement of the waters one by one. > > Paul. > > > > To unsubscribe from the COOT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > > This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing > list hosted by www.jiscmail.ac.uk, terms & conditions are available at > https://www.jiscmail.ac.uk/policyandsecurity/ > To unsubscribe from the COOT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [COOT] Wild refinement of waters
On Wed, 2021-06-02 at 15:56 +0100, Paul Emsley wrote: > On Tue, 2021-06-01 at 20:45 +0100, David M Dranow wrote: > > Does anyone have a decent set of refinement parameters for Coot 0.9.5 in > > linux? My colleagues and I find that using > > the "r" shortcut causes most waters to fly off and center on carbons, > > sometimes as far away as 45 Å! I've tried to > > change the overall weight and Lennard-Jones epsilon, and Geman-McClure > > alpha and I find no combination that keeps > > all > > waters within the density that they already are in. I'd be happy to > > provide a model and map to demonstrate this > > phenomenon. Our current work-around is to use R, but this is not ideal. > > If anyone has any idea how to fix this, we > > would greatly appreciate any advice. > > > > This reminds me of the exchange about the new refinement on CCP4BB some time > ago. Briefly, people reported problems > that > I wasn't seeing. > > So I'd like to see what you see. Either make a screencast of your session or > conjure up a coordinates file and a list > of coot operations that makes the issue apparent. Until then, I am in the > dark. > > Thanks to files provided by Philip Kiser I now see the problem. First, one had to create an unorthodox model - one that included a bad non-bonded contact distance to a water (this is not something I do) Second, one had to refine just the water, and not the residues to which it was hydrogen-bonded or other atoms in the environment (this is not something I do either) By its nature, refinement of this sort will give a considerable kick to a bad water. I have therefore introduced an optional "Soft Mode" for non-bonded-contact restraints. set_refinement_use_soft_mode_nbc_restraints(flag) This is available now in the repo and will be in the next release. But for me, the lessons are: Don't create models with poorly placed waters in the first place. But if you do, don't do isolated refinement of the waters one by one. Paul. To unsubscribe from the COOT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [COOT] Wild refinement of waters
On Wed, 2021-06-02 at 09:03 -0700, Philip Kiser wrote: > I also have observed the unusual behavior with waters not staying in the > density after using the "r" key binding. Do you have an example? The "R" key (I mean not "Shift-R") should not move waters if the centred residue is not a water. > In many cases, there aren't any egregious clashes that could be forcing the > water out of the density. I wonder what "egregious" means here. Does Validate → Atom Overlaps (Coot) give you anything other than green (or nothing)? > I tried changing the Lennard-Jones setting in the refine/regularize panel but > it didn't help. Changing epsilon will change the refinement, but may well not make it behave as you want it to behave if indeed the interaction is egregious. > The issue frequently happens while placing waters into peaks identified by > difference map peak searching. Often, for > weakly bound waters, the center of the Fo-Fc peak does not overlap with the > 2Fo-Fc peak (at 1 RMSD) indeed... > and if one uses the "place water" key binding followed directly by the "r" > key binding, the water often ends up in > some distant 2Fo-Fc peak. That doesn't sound familiar. It sounds like it's getting kicked by some non-bonding interaction. > Switching to the Fo-Fc map for RSR fixed the problem but it is a little > inconvenient to be switching back and forth > between the maps. I imagine that you can create a custom toggle button in a few lines of python to do this switching. Paul. > On Wed, Jun 2, 2021 at 8:03 AM Paul Emsley wrote: > > On Wed, 2021-06-02 at 10:44 +0100, Eleanor Dodson wrote: > > > > > I think it is a sphere refine problem. Just use the old real space > > > refine.. > > > > > sphere refine hates clashes, > > > > > > > > It's not "sphere refine" that hates clashes, it's that the non-bonded > > contact interactions have a new model. > > > > > > > > > so before you try to fix something you need to delete any feature which > > > will create one.. > > > > > > > > ... during real space refinement. > > > > > > > > Yes. Or add the entity that causes the bumps to the RSR molten zone/moving > > atoms. > > > > > > > > > I argue with Paul about the philosophy but he has the expertise! > > > > > > > > I am open to being wrong about how RSR works, but until I see a problem, I > > can't fix it. > > > > > > > > Paul > > > > > > > > > > > > > > On Wed, 2 Jun 2021 at 09:43, F.Xavier Gomis-Rüth > > > wrote: > > > > > > I have experienced the same issue with 0.9.5 for Mac. > > > > > > Most strikingly, this seems to occur randomly: some waters are refined > > > > > > as usual but other are shifted dozens of Å to stick to a carbon of the > > > > model. > > > > > > A similar effect is observed when refining solvents with the "x" key. > > > > > > > > > > > > > > > > > > On 1/6/21 21:45, David M Dranow wrote: > > > > > > > Hi All, > > > > > > > > > > > > > > Does anyone have a decent set of refinement parameters for Coot 0.9.5 > > > > > in linux? My colleagues and I find that > > > > > > > using the "r" shortcut causes most waters to fly off and center on > > > > > carbons, sometimes as far away as 45 Å! > > I've > > > > > > > tried to change the overall weight and Lennard-Jones epsilon, and > > > > > Geman-McClure alpha and I find no > > combination > > > > > > > that keeps all waters within the density that they already are in. > > > > > I'd be happy to provide a model and map to > > > > > > > demonstrate this phenomenon. Our current work-around is to use R, > > > > > but this is not ideal. If anyone has any > > idea > > > > > > > how to fix this, we would greatly appreciate any advice. > > > > > > > > > > > > > > Thank you! > > > > > > > -David > > > > > > > > > > > > > > > > > > > > > > > > > > > > To unsubscribe from the COOT list, click the following link: > > > > > > > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > > > > > > > > > > > > > > This message was issued to members of www.jiscmail.ac.uk/COOT, a > > > > > mailing list hosted by www.jiscmail.ac.uk, > > terms > > > > > > > & conditions are available at > > > > > https://www.jiscmail.ac.uk/policyandsecurity/ > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > To unsubscribe from the COOT list, click the following link: > > > > > > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > > > > > > > > > > > > > > > > To unsubscribe from the COOT list, click the following link: > > > > > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > > > > > > > > > > > > > > > > To unsubscribe from the COOT list, click the following link: > > > > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > > > > > > > > This message was issued to members
Re: [COOT] Wild refinement of waters
Hi Everyone, I also have observed the unusual behavior with waters not staying in the density after using the "r" key binding. In many cases, there aren't any egregious clashes that could be forcing the water out of the density. I tried changing the Lennard-Jones setting in the refine/regularize panel but it didn't help. The issue frequently happens while placing waters into peaks identified by difference map peak searching. Often, for weakly bound waters, the center of the Fo-Fc peak does not overlap with the 2Fo-Fc peak (at 1 RMSD) and if one uses the "place water" key binding followed directly by the "r" key binding, the water often ends up in some distant 2Fo-Fc peak. Switching to the Fo-Fc map for RSR fixed the problem but it is a little inconvenient to be switching back and forth between the maps. Regards, Philip On Wed, Jun 2, 2021 at 8:03 AM Paul Emsley wrote: > On Wed, 2021-06-02 at 10:44 +0100, Eleanor Dodson wrote: > > I think it is a sphere refine problem. Just use the old real space > refine.. > > sphere refine hates clashes, > > It's not "sphere refine" that hates clashes, it's that the non-bonded > contact interactions have a new model. > > > so before you try to fix something you need to delete any feature which > will create one.. > > ... during real space refinement. > > Yes. Or add the entity that causes the bumps to the RSR molten zone/moving > atoms. > > > I argue with Paul about the philosophy but he has the expertise! > > I am open to being wrong about how RSR works, but until I see a problem, I > can't fix it. > > Paul > > > > > On Wed, 2 Jun 2021 at 09:43, F.Xavier Gomis-Rüth > wrote: > > > I have experienced the same issue with 0.9.5 for Mac. > > > Most strikingly, this seems to occur randomly: some waters are refined > > > as usual but other are shifted dozens of Å to stick to a carbon of the > model. > > > A similar effect is observed when refining solvents with the "x" key. > > > > > > > > > On 1/6/21 21:45, David M Dranow wrote: > > > > Hi All, > > > > > > > > Does anyone have a decent set of refinement parameters for Coot > 0.9.5 in linux? My colleagues and I find that > > > > using the "r" shortcut causes most waters to fly off and center on > carbons, sometimes as far away as 45 Å! I've > > > > tried to change the overall weight and Lennard-Jones epsilon, and > Geman-McClure alpha and I find no combination > > > > that keeps all waters within the density that they already are in. > I'd be happy to provide a model and map to > > > > demonstrate this phenomenon. Our current work-around is to use R, > but this is not ideal. If anyone has any idea > > > > how to fix this, we would greatly appreciate any advice. > > > > > > > > Thank you! > > > > -David > > > > > > > > > > > > > > > > > To unsubscribe from the COOT list, click the following link: > > > > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > > > > > > > > This message was issued to members of www.jiscmail.ac.uk/COOT, a > mailing list hosted by www.jiscmail.ac.uk, terms > > > > & conditions are available at > https://www.jiscmail.ac.uk/policyandsecurity/ > > > > > > -- > > > > > > > > > To unsubscribe from the COOT list, click the following link: > > > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > > > > > > > To unsubscribe from the COOT list, click the following link: > > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > > > > To unsubscribe from the COOT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > > This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing > list hosted by www.jiscmail.ac.uk, terms & conditions are available at > https://www.jiscmail.ac.uk/policyandsecurity/ > To unsubscribe from the COOT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [COOT] Wild refinement of waters
On Wed, 2021-06-02 at 10:44 +0100, Eleanor Dodson wrote: > I think it is a sphere refine problem. Just use the old real space refine.. > sphere refine hates clashes, It's not "sphere refine" that hates clashes, it's that the non-bonded contact interactions have a new model. > so before you try to fix something you need to delete any feature which will > create one.. ... during real space refinement. Yes. Or add the entity that causes the bumps to the RSR molten zone/moving atoms. > I argue with Paul about the philosophy but he has the expertise! I am open to being wrong about how RSR works, but until I see a problem, I can't fix it. Paul > > On Wed, 2 Jun 2021 at 09:43, F.Xavier Gomis-Rüth wrote: > > I have experienced the same issue with 0.9.5 for Mac. > > Most strikingly, this seems to occur randomly: some waters are refined > > as usual but other are shifted dozens of Å to stick to a carbon of the > > model. > > A similar effect is observed when refining solvents with the "x" key. > > > > > > On 1/6/21 21:45, David M Dranow wrote: > > > Hi All, > > > > > > Does anyone have a decent set of refinement parameters for Coot 0.9.5 in > > > linux? My colleagues and I find that > > > using the "r" shortcut causes most waters to fly off and center on > > > carbons, sometimes as far away as 45 Å! I've > > > tried to change the overall weight and Lennard-Jones epsilon, and > > > Geman-McClure alpha and I find no combination > > > that keeps all waters within the density that they already are in. I'd > > > be happy to provide a model and map to > > > demonstrate this phenomenon. Our current work-around is to use R, but > > > this is not ideal. If anyone has any idea > > > how to fix this, we would greatly appreciate any advice. > > > > > > Thank you! > > > -David > > > > > > > > > > > > To unsubscribe from the COOT list, click the following link: > > > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > > > > > > This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing > > > list hosted by www.jiscmail.ac.uk, terms > > > & conditions are available at > > > https://www.jiscmail.ac.uk/policyandsecurity/ > > > > -- > > > > > > To unsubscribe from the COOT list, click the following link: > > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > > > > To unsubscribe from the COOT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 To unsubscribe from the COOT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [COOT] Wild refinement of waters
On Tue, 2021-06-01 at 20:45 +0100, David M Dranow wrote: > > Does anyone have a decent set of refinement parameters for Coot 0.9.5 in > linux? My colleagues and I find that using > the "r" shortcut causes most waters to fly off and center on carbons, > sometimes as far away as 45 Å! I've tried to > change the overall weight and Lennard-Jones epsilon, and Geman-McClure alpha > and I find no combination that keeps all > waters within the density that they already are in. I'd be happy to provide > a model and map to demonstrate this > phenomenon. Our current work-around is to use R, but this is not ideal. If > anyone has any idea how to fix this, we > would greatly appreciate any advice. > This reminds me of the exchange about the new refinement on CCP4BB some time ago. Briefly, people reported problems that I wasn't seeing. So I'd like to see what you see. Either make a screencast of your session or conjure up a coordinates file and a list of coot operations that makes the issue apparent. Until then, I am in the dark. Regards, Paul. To unsubscribe from the COOT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [COOT] Wild refinement of waters
I think it is a sphere refine problem. Just use the old real space refine.. sphere refine hates clashes , so before you try to fix something you need to delete any feature which will create one..\ I argue with Paul about the philosophy but he has the expertise! Eleanor On Wed, 2 Jun 2021 at 09:43, F.Xavier Gomis-Rüth wrote: > I have experienced the same issue with 0.9.5 for Mac. > Most strikingly, this seems to occur randomly: some waters are refined > as usual but other are shifted dozens of Å to stick to a carbon of the > model. > A similar effect is observed when refining solvents with the "x" key. > > > On 1/6/21 21:45, David M Dranow wrote: > > Hi All, > > Does anyone have a decent set of refinement parameters for Coot 0.9.5 in > linux? My colleagues and I find that using the "r" shortcut causes most > waters to fly off and center on carbons, sometimes as far away as 45 Å! I've > tried to change the overall weight and Lennard-Jones epsilon, and > Geman-McClure alpha and I find no combination that keeps all waters within > the density that they already are in. I'd be happy to provide a model and > map to demonstrate this phenomenon. Our current work-around is to use R, but > this is not ideal. If anyone has any idea how to fix this, we would greatly > appreciate any advice. > > Thank you! > -David > > > > To unsubscribe from the COOT list, click the following > link:https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > > This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing list > hosted by www.jiscmail.ac.uk, terms & conditions are available at > https://www.jiscmail.ac.uk/policyandsecurity/ > > > -- > > -- > > To unsubscribe from the COOT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 > To unsubscribe from the COOT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
Re: [COOT] Wild refinement of waters
I have experienced the same issue with 0.9.5 for Mac. Most strikingly, this seems to occur randomly: some waters are refined as usual but other are shifted dozens of Å to stick to a carbon of the model. A similar effect is observed when refining solvents with the "x" key. On 1/6/21 21:45, David M Dranow wrote: Hi All, Does anyone have a decent set of refinement parameters for Coot 0.9.5 in linux? My colleagues and I find that using the "r" shortcut causes most waters to fly off and center on carbons, sometimes as far away as 45 Å! I've tried to change the overall weight and Lennard-Jones epsilon, and Geman-McClure alpha and I find no combination that keeps all waters within the density that they already are in. I'd be happy to provide a model and map to demonstrate this phenomenon. Our current work-around is to use R, but this is not ideal. If anyone has any idea how to fix this, we would greatly appreciate any advice. Thank you! -David To unsubscribe from the COOT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/ -- To unsubscribe from the COOT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=COOT=1 This message was issued to members of www.jiscmail.ac.uk/COOT, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
