I forgot to add probably the most important. James is correct, the expected value of u, the true mean, given a single observation k is indeed k+1 and k+1 is also the mean square error of using k+1 as the estimator of the true mean.
Cheers, Filipe On Wed, 13 Oct 2021 at 23:17, Filipe Maia <[email protected]> wrote: > Hi, > > The maximum likelihood estimator for a Poisson distributed variable is > equal to the mean of the observations. In the case of a single observation, > it will be equal to that observation. As Graeme suggested, you can > calculate the probability mass function for a given observation with > different Poisson parameters (i.e. true means) and see that function peaks > when the parameter matches the observation. > > The root mean squared error of the estimation of the true mean from a > single observation k seems to be sqrt(k+2). Or to put it in another way, > mean squared error, that is the expected value of (k-u)**2, for an > observation k and a true mean u, is equal to k+2. > > You can see some example calculations at > https://colab.research.google.com/drive/1eoaNrDqaPnP-4FTGiNZxMllP7SFHkQuS?usp=sharing > > Cheers, > Filipe > > On Wed, 13 Oct 2021 at 17:14, Winter, Graeme (DLSLtd,RAL,LSCI) < > [email protected]> wrote: > >> This rang a bell to me last night, and I think you can derive this from >> first principles >> >> If you assume an observation of N counts, you can calculate the >> probability of such an observation for a given Poisson rate constant X. If >> you then integrate over all possible value of X to work out the central >> value of the rate constant which is most likely to result in an observation >> of N I think you get X = N+1 >> >> I think it is the kind of calculation you can perform on a napkin, if >> memory serves >> >> All the best Graeme >> >> On 13 Oct 2021, at 16:10, Andrew Leslie - MRC LMB < >> [email protected]> wrote: >> >> Hi Ian, James, >> >> I have a strong feeling that I have seen this >> result before, and it was due to Andy Hammersley at ESRF. I’ve done a >> literature search and there is a paper relating to errors in analysis of >> counting statistics (se below), but I had a quick look at this and could >> not find the (N+1) correction, so it must have been somewhere else. I Have >> cc’d Andy on this Email (hoping that this Email address from 2016 still >> works) and maybe he can throw more light on this. What I remember at the >> time I saw this was the simplicity of the correction. >> >> Cheers, >> >> Andrew >> >> Reducing bias in the analysis of counting statistics data >> Hammersley, AP <https://www.webofscience.com/wos/author/record/2665675> >> (Hammersley, >> AP) Antoniadis, A >> <https://www.webofscience.com/wos/author/record/13070551> (Antoniadis, A) >> NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION A-ACCELERATORS >> SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT Volume 394 >> Issue 1-2 >> Page 219-224 >> DOI 10.1016/S0168-9002(97)00668-2 >> Published JUL 11 1997 >> >> On 12 Oct 2021, at 18:55, Ian Tickle <[email protected]> wrote: >> >> >> Hi James >> >> What the Poisson distribution tells you is that if the true count is N >> then the expectation and variance are also N. That's not the same thing as >> saying that for an observed count N the expectation and variance are N. >> Consider all those cases where the observed count is exactly zero. That >> can arise from any number of true counts, though as you noted larger values >> become increasingly unlikely. However those true counts are all >= 0 which >> means that the mean and variance of those true counts must be positive and >> non-zero. From your results they are both 1 though I haven't been through >> the algebra to prove it. >> >> So what you are saying seems correct: for N observed counts we should be >> taking the best estimate of the true value and variance as N+1. For >> reasonably large N the difference is small but if you are concerned with >> weak images it might start to become significant. >> >> Cheers >> >> -- Ian >> >> >> On Tue, 12 Oct 2021 at 17:56, James Holton <[email protected]> wrote: >> >>> All my life I have believed that if you're counting photons then the >>> error of observing N counts is sqrt(N). However, a calculation I just >>> performed suggests its actually sqrt(N+1). >>> >>> My purpose here is to understand the weak-image limit of data >>> processing. Question is: for a given pixel, if one photon is all you >>> got, what do you "know"? >>> >>> I simulated millions of 1-second experiments. For each I used a "true" >>> beam intensity (Itrue) between 0.001 and 20 photons/s. That is, for >>> Itrue= 0.001 the average over a very long exposure would be 1 photon >>> every 1000 seconds or so. For a 1-second exposure the observed count (N) >>> is almost always zero. About 1 in 1000 of them will see one photon, and >>> roughly 1 in a million will get N=2. I do 10,000 such experiments and >>> put the results into a pile. I then repeat with Itrue=0.002, >>> Itrue=0.003, etc. All the way up to Itrue = 20. At Itrue > 20 I never >>> see N=1, not even in 1e7 experiments. With Itrue=0, I also see no N=1 >>> events. >>> Now I go through my pile of results and extract those with N=1, and >>> count up the number of times a given Itrue produced such an event. The >>> histogram of Itrue values in this subset is itself Poisson, but with >>> mean = 2 ! If I similarly count up events where 2 and only 2 photons >>> were seen, the mean Itrue is 3. And if I look at only zero-count events >>> the mean and standard deviation is unity. >>> >>> Does that mean the error of observing N counts is really sqrt(N+1) ? >>> >>> I admit that this little exercise assumes that the distribution of Itrue >>> is uniform between 0.001 and 20, but given that one photon has been >>> observed Itrue values outside this range are highly unlikely. The >>> Itrue=0.001 and N=1 events are only a tiny fraction of the whole. So, I >>> wold say that even if the prior distribution is not uniform, it is >>> certainly bracketed. Now, Itrue=0 is possible if the shutter didn't >>> open, but if the rest of the detector pixels have N=~1, doesn't this >>> affect the prior distribution of Itrue on our pixel of interest? >>> >>> Of course, two or more photons are better than one, but these days with >>> small crystals and big detectors N=1 is no longer a trivial situation. >>> I look forward to hearing your take on this. And no, this is not a >>> trick. >>> >>> -James Holton >>> MAD Scientist >>> >>> ######################################################################## >>> >>> To unsubscribe from the CCP4BB list, click the following link: >>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>> >>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a >>> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are >>> available at https://www.jiscmail.ac.uk/policyandsecurity/ >>> >> >> ------------------------------ >> >> To unsubscribe from the CCP4BB list, click the following link: >> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >> >> >> >> ------------------------------ >> >> To unsubscribe from the CCP4BB list, click the following link: >> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >> >> >> >> >> -- >> >> This e-mail and any attachments may contain confidential, copyright and >> or privileged material, and are for the use of the intended addressee only. >> If you are not the intended addressee or an authorised recipient of the >> addressee please notify us of receipt by returning the e-mail and do not >> use, copy, retain, distribute or disclose the information in or attached to >> the e-mail. >> Any opinions expressed within this e-mail are those of the individual and >> not necessarily of Diamond Light Source Ltd. >> Diamond Light Source Ltd. cannot guarantee that this e-mail or any >> attachments are free from viruses and we cannot accept liability for any >> damage which you may sustain as a result of software viruses which may be >> transmitted in or with the message. >> Diamond Light Source Limited (company no. 4375679). Registered in England >> and Wales with its registered office at Diamond House, Harwell Science and >> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >> >> >> ------------------------------ >> >> To unsubscribe from the CCP4BB list, click the following link: >> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >> > ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
