I dont want to comment much on the science behind FOM estimates - as
James points out after exptl phasing alone many reflections are
effectively unphased - if the FH/ FA for a reflection are very small
then you have no phase estimate at all, and if the F+ F- or FP FPH
measurements are weak any estimate of the phase must be extremely
unreliable.
However there are good tools for evaluating the reliability of FOM once
you have a structure. programs like phasematch can plot the phase
difference against FOM , (and they usually show it is overestimated
after density modification..)
Eleanor Dodson
Soisson, Stephen M wrote:
This is an interesting thread, and perhaps I should not dive in on such
a heady topic, BUT, I do want to point out my own particular bias
regarding FOM that is not entirely consistent with James' point of view.
In my experience, the FOM obtained after density modification runs are
almost always extremely optimistic - I have seen relatively high
apparent FOM after density modification runs (0.7) that had nearly
uninterpretable maps. As such, I am much more interested in knowing
about the FOM from the phasing calculations themselves and NOT after
density modification.
That said, applying arbitrary cut-offs to what would be deemed an
acceptable FOM, after phasing calculations, to generate maps that are
"interpretable" is not really a good thing to do. For instance, I just
had a structure where the FOM was 0.35 after phasing (a rubbish
structure perhaps?), BUT, the data are highly redundant and the solvent
content in the high 70% range. The post density modified maps are
stunningly good. One could easily imagine many other scenarios (e.g.
NCS) where the modified maps and apparent FOM would be decoupled.
So, I do agree with James's suggestion that perhaps we should be
retrospectively calculating a "real" FOM between the final model and the
actual maps you built into (after whatever you did to get them). This
seems like a very good idea indeed.
More personal biases revealed: I actually look at, and use, the Cullis
R values on anomalous and isomorphous data to help determine how much
signal is in the data. Simplified, these numbers are your Average
estimated lack-of-closure divided by your average observed difference.
That's an important thing to know, and I find them quite useful.
Steve
-----Original Message-----
From: CCP4 bulletin board [mailto:[email protected]] On Behalf Of
James Holton
Sent: Tuesday, April 13, 2010 1:48 PM
To: [email protected]
Subject: Re: [ccp4bb] Phasing statistics
Probably the only phasing stat that I pay any attention to these days is
the Figure of Merit (FOM). This is because, the _definition_ of FOM is
that it is the cosine of the phase error (or at least your best estimate
of it). FOM=1 is perfect phases and FOM=0 is random phases, and a
reasonable cutoff value for FOM is 0.5 (see Lunin & Woolfson, Acta D,
1993). Yes, there are ways to get various programs to report very
inaccurate values for FOM (such as running DM for thousands of cycles),
and yes, there are often legitimate reasons to run these programs in
this way. But, there are also very wrong things one can do to get low
Rmerge, Rcryst, and especially Rfree. It is simply a matter of knowing
(and reporting) what you are doing.
If you are worried that your favorite estimate of FOM is inaccurate,
then you can always turn to your most accurate phases: those of your
final, refined model (the one that you have convinced yourself is
"right" and ready to publish). Taking these as the "true phases", the
"true FOM" can always be obtained by comparing the final-model phases to
those of your initial map (using PHISTATS or SFTOOLS). This is by no
means standard practice, but perhaps it should be?
Anyway, FOM is _supposed_ to be the cosine of the phase error, and is
therefore the most relevant statistic when it comes to how good your
phases were when you started building. This is why it is important as a
reviewer to know what it is. If I am faced with a structure that was
built into a MAD map with initial FOM = 0.8 to 2 A resolution, then I am
already convinced that the structure is "right" because I know they had
a very clear map to build into. It is hard to do something egregiously
wrong with such a map (such as tracing it backwards), so I would even
excuse a high R/Rfree in this case, especially if the map has large
absent (disordered) regions that the authors were honest enough to not
build.
On the other hand, if the initial solvent-flattened SAD map had FOM=0.3
to 2 A, you are really pushing it. It is possible to get a correct
structure from such a map, but extremely difficult. One might combine
some MR phases with the SAD phases to improve them somewhat, but how
does one evaluate such a result? I'd say that if FOM < 0.5, then the
phases don't make you right. You need to look at other statistics (like
R/Rfree).
The extreme case, of course, is MR, where the "starting" FOM=0. The
author then makes an assumption about the starting phases (based on
prior knowledge such as homology with PDB ID = xxxx), and that
assumption is then borne out by an "acceptable" R/Rfree (Kleywegt &
Brunger, 1996). The "true FOM" (comparing final refined phases to those
of the initial MR hit) in this case might still be interesting because
it tells you a lot about how much rebuilding had to be done.
To answer Frank's question about a 4 A structure with anisotropic
diffraction (which I assume means that 4 A is in the best direction, and
the other(s) are 5 A or so), I would first ask that the "true"
resolution limit be denoted by the point where the average I/sigma drops
to ~2 (this is _without_ an anisotropic resolution cutoff!). Then we
probably have a 4.5A structure. The "metrics by which we then judge the
results?" then depends on the bigger question: "Does the evidence
presented justify the conclusions drawn?". If the conclusion is that
bond lengths in the active site are "strained", then the answer is
obviously "no". Indeed, if the conclusions rely on the helicies in a
4.5 A map being traced in the right direction, then I would also answer
"no". This is because at 4.5 A the image of a backward-traced helix
looks a _lot_ like the correctly-traced one (see
http://bl831.als.lbl.gov/~jamesh/movies/index.html#reso). To put it
another way, the R-factors alone are not convincing evidence of a
correct trace at 4.5A, and corroborating evidence must be presented to
make the helix direction convincing. By "presented", I mean spelled out
in the text, and by "corroborating evidence" I mean something as simple
as a clear connectivity with enough big side chains placed to deduce the
register of the sequence. Barring that, something like "SeMet scanning"
can also clarify tracing ambiguities (for a relevant example, see Chen
et al. (2007) PNAS 104 p 18999). I am not saying that every 4.5 A
structure needs to do this, but I am saying that the number of
alternative explanations (models) for a given observation (map)
increases as the map gets blurrier, and if a plausible alternative model
could change the conclusions of the paper, then it must be eliminated
with controls. You know, basic science stuff.
It is a common misconception that MAD/SAD/MIR phasing depends on
resolution, but nowhere on the Harker diagram does one see the
"resolution" of the vectors. The accuracy of the phase depends entirely
on the magnitude of the signal (delta-F) and the magnitude of the noise
(sigma(F)). This is why you only get experimental phases for strong
spots, and never all the way out to your "resolution limit". True, this
is a "resolution dependence", but it is actually the signal-to-noise
ratio itself that is important. The only part of experimental phasing
that seems to be reproducibly resolution-dependent is the density
modification used to clean it up. This seems to be limited to pushing
your "good phases" out by ~ 1 A in most cases (i.e. from 4A to 3A or
from 3.5A to 2.5A, etc.), but I'm not sure why that is. Probably
something in histogram matching. Unfortunately, I am not aware of a
good comprehensive review of the resolution dependence of phase
extension, possibly because one cannot do such an analysis with the data
currently available in the PDB (initial phases are not deposited).
I would finally like to note that I am highly uncomfortable with the
idea of excusing the reporting of data processing statistics if the
structure is deemed "correct". Formally, no protein structure is
intrinsically "correct" if it does not explain the data (Fobs) to
withing experimental error (~5%). In the "small molecule world" models
with Rcryst > Rmerge are rejected out-of-hand (and for good reason).
The only reason protein structures are "excused" from this rule is
because they have a good "track record" of agreeing with
experimentally-phased maps.
-James Holton
MAD Scientist
Frank von Delft wrote:
I fully agree, for high quality data.
What though if the data are not impeccable and the structure
necessarily ropey? E.g. 4A phases and anisotropic diffraction. By
what metrics do we then judge the results?
(I don't know the answer, btw, but our membranous colleagues surely
spend quite a bit of time with that question...)
phx.
On 12/04/2010 12:10, Anastassis Perrakis wrote:
Hi -
A year or so ago, I have asked as a referee somebody to provide for a
paper the statistics for their heavy atom derivative dataset,
and for the phasing statistics. For some good reasons, they were
unable to do that, and they (politely) asked me
'what would it change if you knew these, isn't the structure we
present impeccable?'. Well, I think they were right.
Their structure was surely correct, surely high quality. After that
incident and giving it some thought,
I fail to see why should one report e.g. PP or Rcullis, or why will I
care what they were if the structure has a convincing Rfree and is
properly validated.
If someone wants to cheat at the end of the day, its easy to provide
two numbers, but its hard to provide a good validated model that
agrees with the data.
(and, yes, you can also make up the data, but we have been there,
haven't we?!?)
So, my question to that referee, likely being a ccp4bb aficionado
that is reading this email, or to anyone else really, is:
"What would it help to judge the quality of the structure or the
paper if you know PP, Rcullis and FOM?"
Best -
A.
PS Especially since you used SHELXE for phasing these statistics are
utterly irrelevant, and possibly you could advice the referee to read
a bit about how SHELXE works ... or go to one of the nice courses
that George teaches ...
On Apr 12, 2010, at 10:37, Eleanor Dodson wrote:
You can feed the SHELX sites into phaser_er or CRANK both of which
will
give this sort of information.
Or mlphare if you know how to set it up..
Eleanor
Harmer, Nicholas wrote:
Dear CCP4ers,
I've been asked by a referee to provide the phasing statistics for
a SAD dataset that I used to solve a recent structure. Whilst I
have been able to find a figure-of-merit for the data after
phasing, I can't work out how to get any other statistics (e.g.
phasing power or an equivalent or Rcullis). Does anyone know a good
route to obtaining useful statistics to put in the paper for SAD
data?
The structure solution was carried out using SHELX C/D/E and then
ARP/wARP.
Thanks in advance,
Nic Harmer
=====================
Dr. Nic Harmer
School of Biosciences
University of Exeter
tel: +44 1392 725179
*P** **please don't print this e-mail unless you really need to*
Anastassis (Tassos) Perrakis, Principal Investigator / Staff Member
Department of Biochemistry (B8)
Netherlands Cancer Institute,
Dept. B8, 1066 CX Amsterdam, The Netherlands
Tel: +31 20 512 1951 Fax: +31 20 512 1954 Mobile / SMS: +31 6 28
597791
Notice: This e-mail message, together with any attachments, contains information
of Merck & Co., Inc. (One Merck Drive, Whitehouse Station, New Jersey, USA
08889), and/or its affiliates Direct contact information for affiliates is
available at http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely for the
use of the individual or entity named on this message. If you are not the intended
recipient, and have received this message in error, please notify us immediately by
reply e-mail and then delete it from your system.
