Yes, the function implemented in CNS includes the sigma term. Best wishes,
Randy ----- Randy J. Read Department of Haematology, University of Cambridge Cambridge Institute for Medical Research Tel: +44 1223 336500 Wellcome Trust/MRC Building Fax: +44 1223 336827 Hills Road E-mail: [email protected] Cambridge CB2 0XY, U.K. www-structmed.cimr.cam.ac.uk On 20 Jun 2013, at 22:18, Douglas Theobald <[email protected]> wrote: > On Jun 20, 2013, at 4:36 PM, Randy Read <[email protected]> wrote: > >> Hi, >> >> The intensity-based likelihood refinement target was in our paper in 1996 >> (http://www-structmed.cimr.cam.ac.uk/Personal/randy/pubs/li0224r.pdf). It's >> perfectly happy with negative net intensities. Basically, the question >> you're asking with a negative net intensity is what is the probability that >> you could observe a particular negative net intensity (with its associated >> standard deviation) given the intensity calculated from the model. The >> theory takes account of the fact that the true intensity is both correlated >> to the calculated one *and* constrained to be positive. As a result, >> reflections with more strongly negative net intensities shouldn't be >> overweighted relative to ones with intensities much closer to zero, unlike >> the intensity-based least-squares target. > > I see, since you integrated out the true intensity over 0 to inf. That > ameliorates most of my concern with I-based ML refinement. So, it also looks > like each measured intensity is weighted by a function of its measured > sigma_j. Is that correct? And was that implemented in CNS? > >> The expression is indeed more complicated than the one for amplitudes, and >> is computed by integrating the terms of a series expansion. We contributed >> code for this to CNS, and I'm not aware of anyone having implemented it yet >> for any other program. In fact, I get the feeling that it's rarely used >> even in CNS, even though our limited tests suggested that it can give better >> results than the amplitude-based target. >> >> ----- >> Randy J. Read >> Department of Haematology, University of Cambridge >> Cambridge Institute for Medical Research Tel: +44 1223 336500 >> Wellcome Trust/MRC Building Fax: +44 1223 336827 >> Hills Road >> E-mail: [email protected] >> Cambridge CB2 0XY, U.K. >> www-structmed.cimr.cam.ac.uk >> >> On 20 Jun 2013, at 19:13, Kay Diederichs <[email protected]> >> wrote: >> >>> Douglas, >>> >>> the intensity is negative if the integrated spot has a lower intensity than >>> the estimate of the background under the spot. So yes, we are not >>> _measuring_ negative intensities, rather we are estimating intensities, and >>> that estimate may turn out to be negative. In a later step we try to >>> "correct" for this, because it is non-physical, as you say. At that point, >>> the "proper statistical model" comes into play. Essentially we use this as >>> a "prior". In the order of increasing information, we can have more or less >>> informative priors for weak reflections: >>> 1) I > 0 >>> 2) I has a distribution looking like the right half of a Gaussian, and we >>> estimate its width from the variance of the intensities in a resolution >>> shell >>> 3) I follows a Wilson distribution, and we estimate its parameters from the >>> data in a resolution shell >>> 4) I must be related to Fcalc^2 (i.e. once the structure is solved, we >>> re-integrate using the Fcalc as prior) >>> For a given experiment, the problem is chicken-and-egg in the sense that >>> only if you know the characteristics of the data can you choose the correct >>> prior. >>> I guess that using prior 4) would be heavily frowned upon because there is >>> a danger of model bias. You could say: A Bayesian analysis done properly >>> should not suffer from model bias. This is probably true, but the theory to >>> ensure the word "properly" is not available at the moment. >>> Crystallographers usually use prior 3) which, as I tried to point out, also >>> has its weak spots, namely if the data do not behave like those of an ideal >>> crystal - and today's projects often result in data that would have been >>> discarded ten years ago, so they are far from ideal. >>> Prior 2) is available as an option in XDSCONV >>> Prior 1) seems to be used, or is available, in ctruncate in certain cases >>> (I don't know the details) >>> >>> Using intensities instead of amplitudes in refinement would avoid having to >>> choose a prior, and refinement would therefore not be compromised in case >>> of data violating the assumptions underlying the prior. >>> >>> By the way, it is not (Iobs-Icalc)^2 that would be optimized in refinement >>> against intensities, but rather the corresponding maximum likelihood >>> formula (which I seem to remember is more complicated than the amplitude ML >>> formula, or is not an analytical formula at all, but maybe somebody knows >>> better). >>> >>> best, >>> >>> Kay >>> >>> >>> On Thu, 20 Jun 2013 13:14:28 -0400, Douglas Theobald >>> <[email protected]> wrote: >>> >>>> I still don't see how you get a negative intensity from that. It seems >>>> you are saying that in many cases of a low intensity reflection, the >>>> integrated spot will be lower than the background. That is not equivalent >>>> to having a negative measurement (as the measurement is actually positive, >>>> and sometimes things are randomly less positive than backgroiund). If you >>>> are using a proper statistical model, after background correction you will >>>> end up with a positive (or 0) value for the integrated intensity. >>>> >>>> >>>> On Jun 20, 2013, at 1:08 PM, Andrew Leslie <[email protected]> >>>> wrote: >>>> >>>>> >>>>> The integration programs report a negative intensity simply because that >>>>> is the observation. >>>>> >>>>> Because of noise in the Xray background, in a large sample of intensity >>>>> estimates for reflections whose true intensity is very very small one >>>>> will inevitably get some measurements that are negative. These must not >>>>> be rejected because this will lead to bias (because some of these >>>>> intensities for symmetry mates will be estimated too large rather than >>>>> too small). It is not unusual for the intensity to remain negative even >>>>> after averaging symmetry mates. >>>>> >>>>> Andrew >>>>> >>>>> >>>>> On 20 Jun 2013, at 11:49, Douglas Theobald <[email protected]> wrote: >>>>> >>>>>> Seems to me that the negative Is should be dealt with early on, in the >>>>>> integration step. Why exactly do integration programs report negative >>>>>> Is to begin with? >>>>>> >>>>>> >>>>>> On Jun 20, 2013, at 12:45 PM, Dom Bellini <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Wouldnt be possible to take advantage of negative Is to >>>>>>> extrapolate/estimate the decay of scattering background (kind of Wilson >>>>>>> plot of background scattering) to flat out the background and push all >>>>>>> the Is to positive values? >>>>>>> >>>>>>> More of a question rather than a suggestion ... >>>>>>> >>>>>>> D >>>>>>> >>>>>>> >>>>>>> >>>>>>> From: CCP4 bulletin board [mailto:[email protected]] On Behalf Of >>>>>>> Ian Tickle >>>>>>> Sent: 20 June 2013 17:34 >>>>>>> To: ccp4bb >>>>>>> Subject: Re: [ccp4bb] ctruncate bug? >>>>>>> >>>>>>> Yes higher R factors is the usual reason people don't like I-based >>>>>>> refinement! >>>>>>> >>>>>>> Anyway, refining against Is doesn't solve the problem, it only >>>>>>> postpones it: you still need the Fs for maps! (though errors in Fs may >>>>>>> be less critical then). >>>>>>> -- Ian >>>>>>> >>>>>>> On 20 June 2013 17:20, Dale Tronrud >>>>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>>>> If you are refining against F's you have to find some way to avoid >>>>>>> calculating the square root of a negative number. That is why people >>>>>>> have historically rejected negative I's and why Truncate and cTruncate >>>>>>> were invented. >>>>>>> >>>>>>> When refining against I, the calculation of (Iobs - Icalc)^2 couldn't >>>>>>> care less if Iobs happens to be negative. >>>>>>> >>>>>>> As for why people still refine against F... When I was distributing >>>>>>> a refinement package it could refine against I but no one wanted to do >>>>>>> that. The "R values" ended up higher, but they were looking at R >>>>>>> values calculated from F's. Of course the F based R values are lower >>>>>>> when you refine against F's, that means nothing. >>>>>>> >>>>>>> If we could get the PDB to report both the F and I based R values >>>>>>> for all models maybe we could get a start toward moving to intensity >>>>>>> refinement. >>>>>>> >>>>>>> Dale Tronrud >>>>>>> >>>>>>> >>>>>>> On 06/20/2013 09:06 AM, Douglas Theobald wrote: >>>>>>> Just trying to understand the basic issues here. How could refining >>>>>>> directly against intensities solve the fundamental problem of negative >>>>>>> intensity values? >>>>>>> >>>>>>> >>>>>>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp >>>>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>>>> As a maybe better alternative, we should (once again) consider to >>>>>>> refine against intensities (and I guess George Sheldrick would agree >>>>>>> here). >>>>>>> >>>>>>> I have a simple question - what exactly, short of some sort of historic >>>>>>> inertia (or memory lapse), is the reason NOT to refine against >>>>>>> intensities? >>>>>>> >>>>>>> Best, BR >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> This e-mail and any attachments may contain confidential, copyright and >>>>>>> or privileged material, and are for the use of the intended addressee >>>>>>> only. If you are not the intended addressee or an authorised recipient >>>>>>> of the addressee please notify us of receipt by returning the e-mail >>>>>>> and do not use, copy, retain, distribute or disclose the information in >>>>>>> or attached to the e-mail. >>>>>>> >>>>>>> Any opinions expressed within this e-mail are those of the individual >>>>>>> and not necessarily of Diamond Light Source Ltd. >>>>>>> >>>>>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any >>>>>>> attachments are free from viruses and we cannot accept liability for >>>>>>> any damage which you may sustain as a result of software viruses which >>>>>>> may be transmitted in or with the message. >>>>>>> >>>>>>> Diamond Light Source Limited (company no. 4375679). Registered in >>>>>>> England and Wales with its registered office at Diamond House, Harwell >>>>>>> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United >>>>>>> Kingdom >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> >
