I think the best way to deal with issues like this can be found in
Strunk & White "The Elements of Style" (1918). Among other things,
these authors put forward a rather simple yet often overlooked rule to
writing in general, which I think applies equally well to computer programs:
"Be clear."
The sentence itself is an example of how brevity need not sacrifice
clarity. Yes, you need labels in the table itself to be short, but
there is space immediately below (and above) every table that (IMHO)
ought to contain the definitions of each and every variable/abbreviation
used in the table, spelled out no matter how obvious it may seem to the
author. I can tell you many long and painful stories about me trying to
figure out what some variable in some equation in some paper actually
meant! Context is everything. If you are tight for space, cite a
reference (such as the manual).
That, and scientists talking about such quantities in email, papers,
etc. (such as myself) should also heed Strunk and White and also not
just assume that everyone knows exactly what "structure factor" means as
opposed to a "structure amplitude", let alone I/sigma. Indeed, the word
"intensity" is an incredibly ill-defined unit all by itself, to the
point of being useless. It can have units of photons, photons/s,
photons/area/s, photons/area, energy/volume, and many many more. Often
even in the same equation!
I would strongly advise against changing the "variable names" printed
out in log files by SCALA and other programs, especially when a given
name has persisted for a decade or more. Adding an "inline definition"
is fine, but changing names not only breaks programs that were written
to read these logs (and sometimes even humans reading the log), but it
also confines the meaning of "I/SIGMA from SCALA" to a particular period
in history.
So, what statistic do we want to look at? That depends on what you are
trying to do with the data. There is no way for Phil to know this, so
it is good that he prints out lots of different statistics. That said,
when talking about the data quality requirements for structure solution
by MAD/SAD, I suggest looking at I/sigma(I) where:
I - merged intensity (proportional to photons) assigned to a
reciprocal lattice point (hkl index)
sigma(I) - the error assigned to I
Exactly what I/sigma(I) is required to solve a structure, or make some
conclusion about a solved structure is a topic for another day.
-James Holton
MAD Scientist
Phil Evans wrote:
“I/sigma” statistics seem to be contentious & confusing (see recent
discussions on CCP4BB), particularly in what the various measures
should be called (and how they should be labelled in a table, where
there is only room for a very short name). I thought it worth
commenting on this issue at a little more length.
There are several interacting issues:
1) Statistics can be calculated either for individual observations Ihl
or for intensities averaged over multiple (symmetry-related or
replicate) measurements Ih(avg): both are useful, but they need to be
distinguished
2) The statistic can be (a) the ratio of means <I>/<sigma> or (b) the
mean of ratios <I/sigma> . These are not the same.
3)The “sigma” used in 2(a) can be either (a) the estimated corrected
SD or (b) the RMS scatter of observations ie the RMS deviation (which
is itself generally used to estimate a “correction” to the SD). The
RMS scatter cannot be used for 2(b) of course, since that needs
individual sigmas for each reflection.
4) Values will depend on how many outliers have been rejected.
For what it’s worth, Scala outputs two such statistics:-
(i) “I/sigma”: this is calculated for individual observations Ihl and
is the (mean intensity <Ihl>)/(RMS scatter of Ihl). RMS scatter = RMS
[Ihl – Ih(avg)]. This is some measure of the average significance of
individual observations, but does not take into account multiplicity.
In my new program under development (a Scala replacement) I have
relabelled this column “I/RMS” but I don’t really know what best to
call it. This value is a ratio of means (see 2(a) above).
(ii) “Mn(I/sd)”: this is the mean value of (Ih(avg)/sd(Ih(avg))),
where Ih(avg) is the (weighted) average over all observations for
reflection h, and sd(Ih(avg)) is the estimated SD of this average,
after any “corrections” have been applied. This is, I think, the best
estimate of “signal-to-noise ratio”, but does depend on realistic
estimates of sd(Ih(avg)), which is not entirely straightforward (and
certainly doesn’t allow for systematic errors!). This value is a mean
of ratios (see 2(b) above).
The “corrected” sd(Ihl) is calculated in Scala for each observation as
sd(Ihl)corrected = SdFac * sqrt{sd(I)**2 + SdB*Ihl*LP +
(SdAdd*Ihl)**2}
with the parameters SdFac, SdB & SdAdd determined by trying to make
the RMS normalised deviation Delta(hl) = (Ihl -
Ih(avg))/sd(Ihl)corrected = 1.0 for all intensity ranges (different
parameters for each run). If the sd estimates are correct, then the
distribution of Delta(hl) should have SD = 1.0, and this “correction”
tries to enforce this. This is more or less equivalent to making the
RMS scatter == average SD. However the uncertainties in how best to
estimate the real error do then influence the reliability of the
Mn(I/sd) statistic (see (ii) above)
So what statistics do we want to look at? Probably the main reason for
looking at signal/noise statistics is to choose a “real resolution”
cutoff, from some sort of signal/noise ratio. It isn’t clear (to me)
what is the best way of doing this, and it is particularly difficult
if the data are significantly anisotropic. The multiplicity needs to
be taken into account, so the individual “I/sigma” (see (i) above)
isn’t the best guide. Personally, I generally cut data at around the
point where Mn(I/sd) =~ 2, but I would cut off at <2 for anisotropic
data. I also find a useful guide from the correlation coefficient
between Ih(avg) (Imean) pairs in half-datasets (plotted by Scala): the
CC should be >0.5 at least, I think.
Note that the overall value of any of these statistics over all
resolution ranges is not very useful and can be confusing, depending
on the distribution of intensities, since it mixes up strong low
resolution data (high signal/noise) with weak high resolution data
(low signal/noise).
That leaves the question of how to label these statistics in a
consistent, clear and concise way: suggestions?
Phil Evans
