All, this was my reply to one of James' emails which I just noticed was to me only, and I ought to have CC'd it to the BB, since it has relevance to others' contributions to the discussion.
Cheers -- Ian On Mon, 18 Oct 2021 at 11:27, Ian Tickle <[email protected]> wrote: > > James, no I don't think so, what does it have to do with the number of > pixels? All that matters is the photon flux and the area of measurement. > The only purpose of the detector is to make the measurement: it cannot > possibly change the measurement unless of course it's less than ideal. > Assuming we are using ideal quantum detectors with DQE = 1 you can take > away the detector and replace it with a different ideal detector with > different-sized pixels. The result must be the same for any ideal detector > assuming a fixed photon flux and measurement box. Remember that for the > Poisson distribution (and the true count is Poisson-distributed), the > expectation equals the variance. If you're saying that the variance is 100 > then so is the expectation: does that sound sensible given that no counts > were observed? > > I would take heed of Gergely's wise counsel: "It is easy to fall into the > trap of frequentist thinking and reduce data one step at a time.". Your > argument is step 1: estimate expectation and variance for each pixel; step > 2: add up the expectations and variances for all the pixels. It doesn't > work like that! > > I stick with my original suggestion that for no observed counts in some > area the best estimate of expectation and variance is 1 in that area (I'm > probably assuming an uninformative prior but as I said I haven't been > through the algebra in detail). > > Cheers > > -- Ian > > > On Sat, 16 Oct 2021 at 16:04, James Holton <[email protected]> wrote: > >> Sorry to be unclear. I am actually referring to the background. Assume >> the spot is a systematic absence, but we still need a variance. The >> "units" (if you will pemit) will be photons/pixel. Not photons/spot. >> >> I think in that case my 10x10 patch of independent pixels, all with zero >> observed counts, would have a variance of 100. The sum of two patches of >> 50 would also have a variance of 100. >> >> Right? >> >> >> On 10/16/2021 5:35 AM, Ian Tickle wrote: >> >> >> PS Note also that the prior is for the integrated spot intensity, not the >> individual pixel counts, so we should integrate before applying the +1 >> correction. >> >> I. >> >> >> On Sat, 16 Oct 2021 at 10:16, Ian Tickle <[email protected]> wrote: >> >>> >>> James, you're now talking about additivity of the observed or true >>> counts for different spots whereas I assumed your question still concerned >>> estimation of the expectation and variance of the true count of a given >>> spot, assuming some prior distribution of the true count. As we've already >>> seen, the latter does not behave in an intuitive way. >>> >>> We can use argument by reductio ad absurdum (reduction to absurdity) to >>> demonstrate this, i.e. assume the contrary (i.e. that the expected counts >>> and variances are additive), and show that it leads inevitably to a logical >>> contradiction. First it should be clear that the result must be >>> independent of the pixellation of the detector surface, i.e. the pixel size >>> (it needn't correspond to the hardware pixel detectors), provided it's not >>> smaller than the hardware pixel and not greater than the spot size. >>> >>> That means that we can subdivide the 10x10 area any way we like and we >>> should always get the same answer for the total expected value and >>> variance, so 100 1x1 pixels, or 50 2x1, or 25 2x2, or 4 5x5, or 1 10x10 >>> etc. Given that we accept the estimate of 1 for the expectation and >>> variance of the true count for zero observed count and assume additivity of >>> the expected values and variances, these choices give 100, 50, 25, 4 and 1 >>> as the answer! We can accept possible alternative solutions to a problem >>> where each solution is correct in its own universe, but not different >>> solutions that are simultaneously correct in the same universe: that's a >>> logical contradiction. So we are forced to abandon additivity for the >>> expected value and variance of the true count for a single spot. That of >>> course has nothing to do with additivity of those values for multiple >>> different spots: that is still valid. >>> >>> BTW I found this paper on Bayesian priors for the Poisson distribution: >>> "Inferring the intensity of Poisson processes at the limit of the detector >>> sensitivity": https://arxiv.org/pdf/hep-ex/9909047.pdf . >>> >>> Cheers >>> >>> -- Ian >>> >>> >>> On Sat, 16 Oct 2021 at 00:25, James Holton <[email protected]> wrote: >>> >>>> I don't follow. How is it that variances are not additive? We do this >>>> all the time when we merge data together. >>>> >>>> >>>> On 10/15/2021 4:22 PM, Ian Tickle wrote: >>>> >>>> James, also as the question is posed none of the answers is correct >>>> because a photon count must be an integer (there's no such thing as a >>>> fractional photon). A value of 1.0 implies a count and variance somewhere >>>> between around 0.95 and 1.05 which makes no sense and conflicts with the >>>> actual value of 1. >>>> >>>> -- Ian >>>> >>>> >>>> On Fri, 15 Oct 2021, 23:53 Ian Tickle, <[email protected]> wrote: >>>> >>>>> James, in that case as already discussed the expectation = variance = >>>>> 1. Clearly 0 is not an option since some true counts > 0 are highly >>>>> likely. If your thinking is that the best estimate for a single pixel >>>>> containing no counts is also 1, so that for the 100 pixel patch it must be >>>>> 100, that's not correct because the expectations and variances are not >>>>> additive. You have to consider the zero count for the entire 100-pixel >>>>> patch when estimating the best value for that patch. >>>>> >>>>> Cheers >>>>> >>>>> -- Ian >>>>> >>>>> >>>>> On Fri, 15 Oct 2021, 17:07 James Holton, <[email protected]> wrote: >>>>> >>>>>> Well I'll be... >>>>>> >>>>>> Kay Diederichs pointed out to me off-list that the k+1 expectation >>>>>> and variance from observing k photons is in "Bayesian Reasoning in Data >>>>>> Analysis: A Critical Introduction" by Giulio D. Agostini. Granted, that >>>>>> is >>>>>> with a uniform prior, which I take as the Bayesean equivalent of "I have >>>>>> no >>>>>> idea". >>>>>> >>>>>> So, if I'm looking to integrate a 10 x 10 patch of pixels on a weak >>>>>> detector image, and I find that area has zero counts, what variance >>>>>> shall I >>>>>> put on that observation? Is it: >>>>>> >>>>>> a) zero >>>>>> b) 1.0 >>>>>> c) 100 >>>>>> >>>>>> Wish I could say there are no wrong answers, but I think at least two >>>>>> of those are incorrect, >>>>>> >>>>>> -James Holton >>>>>> MAD Scientist >>>>>> >>>>>> On 10/13/2021 2:34 PM, Filipe Maia wrote: >>>>>> >>>>>> I forgot to add probably the most important. James is correct, the >>>>>> expected value of u, the true mean, given a single observation k is >>>>>> indeed >>>>>> k+1 and k+1 is also the mean square error of using k+1 as the estimator >>>>>> of >>>>>> the true mean. >>>>>> >>>>>> Cheers, >>>>>> Filipe >>>>>> >>>>>> On Wed, 13 Oct 2021 at 23:17, Filipe Maia <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> The maximum likelihood estimator for a Poisson distributed variable >>>>>>> is equal to the mean of the observations. In the case of a single >>>>>>> observation, it will be equal to that observation. As Graeme suggested, >>>>>>> you >>>>>>> can calculate the probability mass function for a given observation with >>>>>>> different Poisson parameters (i.e. true means) and see that function >>>>>>> peaks >>>>>>> when the parameter matches the observation. >>>>>>> >>>>>>> The root mean squared error of the estimation of the true mean from >>>>>>> a single observation k seems to be sqrt(k+2). Or to put it in another >>>>>>> way, >>>>>>> mean squared error, that is the expected value of (k-u)**2, for an >>>>>>> observation k and a true mean u, is equal to k+2. >>>>>>> >>>>>>> You can see some example calculations at >>>>>>> https://colab.research.google.com/drive/1eoaNrDqaPnP-4FTGiNZxMllP7SFHkQuS?usp=sharing >>>>>>> >>>>>>> Cheers, >>>>>>> Filipe >>>>>>> >>>>>>> On Wed, 13 Oct 2021 at 17:14, Winter, Graeme (DLSLtd,RAL,LSCI) < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> This rang a bell to me last night, and I think you can derive this >>>>>>>> from first principles >>>>>>>> >>>>>>>> If you assume an observation of N counts, you can calculate the >>>>>>>> probability of such an observation for a given Poisson rate constant >>>>>>>> X. If >>>>>>>> you then integrate over all possible value of X to work out the central >>>>>>>> value of the rate constant which is most likely to result in an >>>>>>>> observation >>>>>>>> of N I think you get X = N+1 >>>>>>>> >>>>>>>> I think it is the kind of calculation you can perform on a napkin, >>>>>>>> if memory serves >>>>>>>> >>>>>>>> All the best Graeme >>>>>>>> >>>>>>>> On 13 Oct 2021, at 16:10, Andrew Leslie - MRC LMB < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>> Hi Ian, James, >>>>>>>> >>>>>>>> I have a strong feeling that I have seen this >>>>>>>> result before, and it was due to Andy Hammersley at ESRF. I’ve done a >>>>>>>> literature search and there is a paper relating to errors in analysis >>>>>>>> of >>>>>>>> counting statistics (se below), but I had a quick look at this and >>>>>>>> could >>>>>>>> not find the (N+1) correction, so it must have been somewhere else. I >>>>>>>> Have >>>>>>>> cc’d Andy on this Email (hoping that this Email address from 2016 still >>>>>>>> works) and maybe he can throw more light on this. What I remember at >>>>>>>> the >>>>>>>> time I saw this was the simplicity of the correction. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> >>>>>>>> Andrew >>>>>>>> >>>>>>>> Reducing bias in the analysis of counting statistics data >>>>>>>> Hammersley, AP >>>>>>>> <https://www.webofscience.com/wos/author/record/2665675> (Hammersley, >>>>>>>> AP) Antoniadis, A >>>>>>>> <https://www.webofscience.com/wos/author/record/13070551> (Antoniadis, >>>>>>>> A) >>>>>>>> NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION >>>>>>>> A-ACCELERATORS SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT >>>>>>>> Volume 394 >>>>>>>> Issue 1-2 >>>>>>>> Page 219-224 >>>>>>>> DOI 10.1016/S0168-9002(97)00668-2 >>>>>>>> Published JUL 11 1997 >>>>>>>> >>>>>>>> On 12 Oct 2021, at 18:55, Ian Tickle <[email protected]> wrote: >>>>>>>> >>>>>>>> >>>>>>>> Hi James >>>>>>>> >>>>>>>> What the Poisson distribution tells you is that if the true count >>>>>>>> is N then the expectation and variance are also N. That's not the same >>>>>>>> thing as saying that for an observed count N the expectation and >>>>>>>> variance >>>>>>>> are N. Consider all those cases where the observed count is exactly >>>>>>>> zero. >>>>>>>> That can arise from any number of true counts, though as you noted >>>>>>>> larger >>>>>>>> values become increasingly unlikely. However those true counts are >>>>>>>> all >= >>>>>>>> 0 which means that the mean and variance of those true counts must be >>>>>>>> positive and non-zero. From your results they are both 1 though I >>>>>>>> haven't >>>>>>>> been through the algebra to prove it. >>>>>>>> >>>>>>>> So what you are saying seems correct: for N observed counts we >>>>>>>> should be taking the best estimate of the true value and variance as >>>>>>>> N+1. >>>>>>>> For reasonably large N the difference is small but if you are concerned >>>>>>>> with weak images it might start to become significant. >>>>>>>> >>>>>>>> Cheers >>>>>>>> >>>>>>>> -- Ian >>>>>>>> >>>>>>>> >>>>>>>> On Tue, 12 Oct 2021 at 17:56, James Holton <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> All my life I have believed that if you're counting photons then >>>>>>>>> the >>>>>>>>> error of observing N counts is sqrt(N). However, a calculation I >>>>>>>>> just >>>>>>>>> performed suggests its actually sqrt(N+1). >>>>>>>>> >>>>>>>>> My purpose here is to understand the weak-image limit of data >>>>>>>>> processing. Question is: for a given pixel, if one photon is all >>>>>>>>> you >>>>>>>>> got, what do you "know"? >>>>>>>>> >>>>>>>>> I simulated millions of 1-second experiments. For each I used a >>>>>>>>> "true" >>>>>>>>> beam intensity (Itrue) between 0.001 and 20 photons/s. That is, >>>>>>>>> for >>>>>>>>> Itrue= 0.001 the average over a very long exposure would be 1 >>>>>>>>> photon >>>>>>>>> every 1000 seconds or so. For a 1-second exposure the observed >>>>>>>>> count (N) >>>>>>>>> is almost always zero. About 1 in 1000 of them will see one >>>>>>>>> photon, and >>>>>>>>> roughly 1 in a million will get N=2. I do 10,000 such experiments >>>>>>>>> and >>>>>>>>> put the results into a pile. I then repeat with Itrue=0.002, >>>>>>>>> Itrue=0.003, etc. All the way up to Itrue = 20. At Itrue > 20 I >>>>>>>>> never >>>>>>>>> see N=1, not even in 1e7 experiments. With Itrue=0, I also see no >>>>>>>>> N=1 >>>>>>>>> events. >>>>>>>>> Now I go through my pile of results and extract those with N=1, >>>>>>>>> and >>>>>>>>> count up the number of times a given Itrue produced such an event. >>>>>>>>> The >>>>>>>>> histogram of Itrue values in this subset is itself Poisson, but >>>>>>>>> with >>>>>>>>> mean = 2 ! If I similarly count up events where 2 and only 2 >>>>>>>>> photons >>>>>>>>> were seen, the mean Itrue is 3. And if I look at only zero-count >>>>>>>>> events >>>>>>>>> the mean and standard deviation is unity. >>>>>>>>> >>>>>>>>> Does that mean the error of observing N counts is really sqrt(N+1) >>>>>>>>> ? >>>>>>>>> >>>>>>>>> I admit that this little exercise assumes that the distribution of >>>>>>>>> Itrue >>>>>>>>> is uniform between 0.001 and 20, but given that one photon has >>>>>>>>> been >>>>>>>>> observed Itrue values outside this range are highly unlikely. The >>>>>>>>> Itrue=0.001 and N=1 events are only a tiny fraction of the whole. >>>>>>>>> So, I >>>>>>>>> wold say that even if the prior distribution is not uniform, it is >>>>>>>>> certainly bracketed. Now, Itrue=0 is possible if the shutter >>>>>>>>> didn't >>>>>>>>> open, but if the rest of the detector pixels have N=~1, doesn't >>>>>>>>> this >>>>>>>>> affect the prior distribution of Itrue on our pixel of interest? >>>>>>>>> >>>>>>>>> Of course, two or more photons are better than one, but these days >>>>>>>>> with >>>>>>>>> small crystals and big detectors N=1 is no longer a trivial >>>>>>>>> situation. >>>>>>>>> I look forward to hearing your take on this. And no, this is not >>>>>>>>> a trick. >>>>>>>>> >>>>>>>>> -James Holton >>>>>>>>> MAD Scientist >>>>>>>>> >>>>>>>>> >>>>>>>>> ######################################################################## >>>>>>>>> >>>>>>>>> To unsubscribe from the CCP4BB list, click the following link: >>>>>>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>>>>>>>> >>>>>>>>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, >>>>>>>>> a mailing list hosted by www.jiscmail.ac.uk, terms & conditions >>>>>>>>> are available at https://www.jiscmail.ac.uk/policyandsecurity/ >>>>>>>>> >>>>>>>> >>>>>>>> ------------------------------ >>>>>>>> >>>>>>>> To unsubscribe from the CCP4BB list, click the following link: >>>>>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ------------------------------ >>>>>>>> >>>>>>>> To unsubscribe from the CCP4BB list, click the following link: >>>>>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> This e-mail and any attachments may contain confidential, copyright >>>>>>>> and or privileged material, and are for the use of the intended >>>>>>>> addressee >>>>>>>> only. If you are not the intended addressee or an authorised recipient >>>>>>>> of >>>>>>>> the addressee please notify us of receipt by returning the e-mail and >>>>>>>> do >>>>>>>> not use, copy, retain, distribute or disclose the information in or >>>>>>>> attached to the e-mail. >>>>>>>> Any opinions expressed within this e-mail are those of the >>>>>>>> individual and not necessarily of Diamond Light Source Ltd. >>>>>>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any >>>>>>>> attachments are free from viruses and we cannot accept liability for >>>>>>>> any >>>>>>>> damage which you may sustain as a result of software viruses which may >>>>>>>> be >>>>>>>> transmitted in or with the message. >>>>>>>> Diamond Light Source Limited (company no. 4375679). Registered in >>>>>>>> England and Wales with its registered office at Diamond House, Harwell >>>>>>>> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United >>>>>>>> Kingdom >>>>>>>> >>>>>>>> >>>>>>>> ------------------------------ >>>>>>>> >>>>>>>> To unsubscribe from the CCP4BB list, click the following link: >>>>>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>>>>>>> >>>>>>> >>>>>> ------------------------------ >>>>>> >>>>>> To unsubscribe from the CCP4BB list, click the following link: >>>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> >>>>>> To unsubscribe from the CCP4BB list, click the following link: >>>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>>>>> >>>>> >>>> >> ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
