Dear Pavel, Diffuse scattering is probably the most difficult topic I have worked on. Reading Peter Moore's new book and his insights give me renewed hope we could make much more of it, as I mentioned to Tim re 'structure and dynamics'. You describe more aspects below obviously. Greetings, John Prof John R Helliwell DSc
On 24 Jun 2013, at 17:12, Pavel Afonine <[email protected]> wrote: > Refinement against images is a nice old idea. > From refinement technical point of view it's going to be challenging. > Refining just two flat bulk solvent model ksol&Bsol simultaneously may be > tricky, or occupancy + individual B-factor + TLS, or ask multipolar > refinement folk about whole slew of magic they use to refine different > multipolar parameters at different stages of refinement proces and in > different order and applied to different atom types (H vs non-H) > ...etc...etc. Now if you convolute all this with the whole diffraction > experiment parameters through using images in refinement that will be big > fun, I'm sure. > Pavel > > > > On Sun, Jun 23, 2013 at 11:13 PM, Jrh <[email protected]> wrote: > Dear Tom, > I find this suggestion of using the full images an excellent and visionary > one. > So, how to implement it? > We are part way along the path with James Holton's reverse Mosflm. > The computer memory challenge could be ameliorated by simple pixel averaging > at least initially. > The diffuse scattering would be the ultimate gold at the end of the rainbow. > Peter Moore's new book, inter alia, carries many splendid insights into the > diffuse scattering in our diffraction patterns. > Fullprof analyses have become a firm trend in other fields, admittedly with > simpler computing overheads. > Greetings, > John > > Prof John R Helliwell DSc FInstP > > > > On 21 Jun 2013, at 23:16, "Terwilliger, Thomas C" <[email protected]> > wrote: > > > I hope I am not duplicating too much of this fascinating discussion with > > these comments: perhaps the main reason there is confusion about what to > > do is that neither F nor I is really the most suitable thing to use in > > refinement. As pointed out several times in different ways, we don't > > measure F or I, we only measure counts on a detector. As a convenience, we > > "process" our diffraction images to estimate I or F and their uncertainties > > and model these uncertainties as simple functions (e.g., a Gaussian). > > There is no need in principle to do that, and if we were to refine instead > > against the raw image data these issues about positivity would disappear > > and our structures might even be a little better. > > > > Our standard procedure is to estimate F or I from counts on the detector, > > then to use these estimates of F or I in refinement. This is not so easy > > to do right because F or I contain many terms coming from many pixels and > > it is hard to model their statistics in detail. Further, attempts we make > > to estimate either F or I as physically plausible values (e.g., using the > > fact that they are not negative) will generally be biased (the values after > > correction will generally be systematically low or systematically high, as > > is true for the French and Wilson correction and as would be true for the > > truncation of I at zero or above). > > > > Randy's method for intensity refinement is an improvement because the > > statistics are treated more fully than just using an estimate of F or I and > > assuming its uncertainty has a simple distribution. So why not avoid all > > the problems with modeling the statistics of processed data and instead > > refine against the raw data. From the structural model you calculate F, > > from F and a detailed model of the experiment (the same model that is > > currently used in data processing) you calculate the counts expected on > > each pixel. Then you calculate the likelihood of the data given your models > > of the structure and of the experiment. This would have lots of benefits > > because it would allow improved descriptions of the experiment (decay, > > absorption, detector sensitivity, diffuse scattering and other "background" > > on the images,....on and on) that could lead to more accurate structures in > > the end. Of course there are some minor issues about putting all this in > > computer memory for refinement.... > > > > -Tom T > > ________________________________________ > > From: CCP4 bulletin board [[email protected]] on behalf of Phil > > [[email protected]] > > Sent: Friday, June 21, 2013 2:50 PM > > To: [email protected] > > Subject: Re: [ccp4bb] ctruncate bug? > > > > However you decide to argue the point, you must consider _all_ the > > observations of a reflection (replicates and symmetry related) together > > when you infer Itrue or F etc, otherwise you will bias the result even > > more. Thus you cannot (easily) do it during integration > > > > Phil > > > > Sent from my iPad > > > > On 21 Jun 2013, at 20:30, Douglas Theobald <[email protected]> wrote: > > > >> On Jun 21, 2013, at 2:48 PM, Ed Pozharski <[email protected]> wrote: > >> > >>> Douglas, > >>>>> Observed intensities are the best estimates that we can come up with in > >>>>> an experiment. > >>>> I also agree with this, and this is the clincher. You are arguing that > >>>> Ispot-Iback=Iobs is the best estimate we can come up with. I claim that > >>>> is absurd. How are you quantifying "best"? Usually we have some sort > >>>> of discrepancy measure between true and estimate, like RMSD, mean > >>>> absolute distance, log distance, or somesuch. Here is the important > >>>> point --- by any measure of discrepancy you care to use, the person who > >>>> estimates Iobs as 0 when Iback>Ispot will *always*, in *every case*, > >>>> beat the person who estimates Iobs with a negative value. This is an > >>>> indisputable fact. > >>> > >>> First off, you may find it useful to avoid such words as absurd and > >>> indisputable fact. I know political correctness may be sometimes > >>> overrated, but if you actually plan to have meaningful discussion, let's > >>> assume that everyone responding to your posts is just trying to help > >>> figure this out. > >> > >> I apologize for offending and using the strong words --- my intention was > >> not to offend. This is just how I talk when brainstorming with my > >> colleagues around a blackboard, but of course then you can see that I > >> smile when I say it. > >> > >>> To address your point, you are right that J=0 is closer to "true > >>> intensity" then a negative value. The problem is that we are not after a > >>> single intensity, but rather all of them, as they all contribute to > >>> electron density reconstruction. If you replace negative Iobs with E(J), > >>> you would systematically inflate the averages, which may turn problematic > >>> in some cases. > >> > >> So, I get the point. But even then, using any reasonable criterion, the > >> whole estimated dataset will be closer to the true data if you set all > >> "negative" intensity estimates to 0. > >> > >>> It is probably better to stick with "raw intensities" and construct > >>> theoretical predictions properly to account for their properties. > >>> > >>> What I was trying to tell you is that observed intensities is what we get > >>> from experiment. > >> > >> But they are not what you get from the detector. The detector spits out a > >> positive value for what's inside the spot. It is we, as human agents, who > >> later manipulate and massage that data value by subtracting the background > >> estimate. A value that has been subjected to a crude background > >> subtraction is not the raw experimental value. It has been modified, and > >> there must be some logic to why we massage the data in that particular > >> manner. I agree, of course, that the background should be accounted for > >> somehow. But why just subtract it away? There are other ways to massage > >> the data --- see my other post to Ian. My argument is that however we > >> massage the experimentally observed value should be physically informed, > >> and allowing negative intensity estimates violates the basic physics. > >> > >> [snip] > >> > >>>>> These observed intensities can be negative because while their true > >>>>> underlying value is positive, random errorsmay result in Iback>Ispot. > >>>>> There is absolutely nothing unphysical here. > >>>> Yes there is. The only way you can get a negative estimate is to make > >>>> unphysical assumptions. Namely, the estimate Ispot-Iback=Iobs assumes > >>>> that both the true value of I and the background noise come from a > >>>> Gaussian distribution that is allowed to have negative values. Both of > >>>> those assumptions are unphysical. > >>> > >>> See, I have a problem with this. Both common sense and laws of physics > >>> dictate that number of photons hitting spot on a detector is a positive > >>> number. There is no law of physics that dictates that under no > >>> circumstances there could be Ispot<Iback. > >> > >> That's not what I'm saying. Sure, Ispot can be less than Iback randomly. > >> That does not mean we have to estimate the detected intensity as negative, > >> after accounting for background. > >> > >>> Yes, E(Ispot)>=E(Iback). Yes, E(Ispot-Iback)>=0. But > >>> P(Ispot-Iback=0)>0, and therefore experimental sampling of Ispot-Iback is > >>> bound to occasionally produce negative values. What law of physics is > >>> broken when for a given reflection total number of photons in spot pixels > >>> is less that total number of photons in equal number of pixels in the > >>> surrounding background mask? > >>> > >>> Cheers, > >>> > >>> Ed. > >>> > >>> -- > >>> Oh, suddenly throwing a giraffe into a volcano to make water is crazy? > >>> Julian, King of Lemurs >
