Jeff,
thanks - I can see the same equation and cutoff applied in ctruncate
source. Here is the relevant part of the code
// Bayesian statistics tells us to modify I/sigma by
subtracting off sigma/S
// where S is the mean intensity in the resolution shell
h = I/sigma - sigma/S;
// reject as unphysical reflections for which I < -3.7 sigma,
or h < -4.0
if (I/sigma < -3.7 || h < -4.0 ) {
nrej++;
if (debug) printf("unphys: %f %f %f %f\n",I,sigma,S,h);
return(0);
}
This seems to be arbitrary cutoff choice, given that they are
hard-coded. At the very least, cutoffs should depend on the total
number of reflections to represent famyliwise error rates.
It is however the h-based rejection that seems most problematic to me.
In the dataset in question, up to 20% reflections are rejected in the
highest resolution shell (granted, I/sigI there is 0.33). I would
expect that reflections are rejected when they are deemed to be outliers
due to reasons other than statistical errors (e.g. streaks, secondary
lattice spots in the background, etc). I must say that this was done
with extremely good quality data, so I doubt that 1 out of 5 reflections
returns some physically impossible measurement.
What is happening is that <sigma>=3S in the highest resolution shell,
and for many reflection h<-4.0. This does not mean that reflections are
"unphysical" though, just that shell as a whole has mostly weak data (in
this case 89% with I/sigI<2 and 73% with I/sigI<1).
What is counterintuitive is why do I have to discard reflections that
are just plain weak, and not really outliers?
Cheers,
Ed.
On 06/17/2013 10:29 PM, Jeff Headd wrote:
Hi Ed,
I'm not directly familiar with the ctruncate implementation of French
and Wilson, but from the implementation that I put into Phenix (based
on the original F&W paper) I can tell you that any reflection where
(I/sigI) - (sigI/mean_intensity) is less than a defined cutoff (in our
case -4.0), then it is rejected. Depending on sigI and the mean
intensity for a given shell, this can result in positive intensities
that are also rejected. Typically this will effect very small positive
intensities as you've observed.
I don't recall the mathematical justification for this and don't have
a copy of F&W here at home, but I can have a look in the morning when
I get into the lab and let you know.
Jeff
On Mon, Jun 17, 2013 at 5:04 PM, Ed Pozharski <epozh...@umaryland.edu
<mailto:epozh...@umaryland.edu>> wrote:
I noticed something strange when processing a dataset with
imosflm. The
final output ctruncate_etc.mtz, contains IMEAN and F columns, which
should be the conversion according to French&Wilson. Problem is that
IMEAN has no missing values (100% complete) while F has about 1500
missing (~97% complete)!
About half of the reflections that go missing are negative, but
half are
positive. About 5x more negative intensities are successfully
converted. Most impacted are high resolution shells with weak signal,
so I am sure impact on "normal" refinement would be minimal.
However, I am just puzzled why would ctruncate reject positive
intensities (or negative for that matter - I don't see any cutoff
described in the manual and the lowest I/sigI for successfully
converted
reflection is -18).
Is this a bug or feature?
Cheers,
Ed.
--
I don't know why the sacrifice thing didn't work.
Science behind it seemed so solid.
Julian, King of Lemurs
--
Oh, suddenly throwing a giraffe into a volcano to make water is crazy?
Julian, King of Lemurs
