Hi Ian, James,
I have a strong feeling that I have seen this result
before, and it was due to Andy Hammersley at ESRF. I’ve done a literature
search and there is a paper relating to errors in analysis of counting
statistics (se below), but I had a quick look at this and could not find the
(N+1) correction, so it must have been somewhere else. I Have cc’d Andy on this
Email (hoping that this Email address from 2016 still works) and maybe he can
throw more light on this. What I remember at the time I saw this was the
simplicity of the correction.
Cheers,
Andrew
Reducing bias in the analysis of counting statistics data
Hammersley, AP <https://www.webofscience.com/wos/author/record/2665675>
(Hammersley, AP) Antoniadis, A
<https://www.webofscience.com/wos/author/record/13070551> (Antoniadis, A)
NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION A-ACCELERATORS
SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT
Volume
394
Issue
1-2
Page
219-224
DOI
10.1016/S0168-9002(97)00668-2
Published
JUL 11 1997
> On 12 Oct 2021, at 18:55, Ian Tickle <[email protected]> wrote:
>
>
> Hi James
>
> What the Poisson distribution tells you is that if the true count is N then
> the expectation and variance are also N. That's not the same thing as saying
> that for an observed count N the expectation and variance are N. Consider
> all those cases where the observed count is exactly zero. That can arise
> from any number of true counts, though as you noted larger values become
> increasingly unlikely. However those true counts are all >= 0 which means
> that the mean and variance of those true counts must be positive and
> non-zero. From your results they are both 1 though I haven't been through
> the algebra to prove it.
>
> So what you are saying seems correct: for N observed counts we should be
> taking the best estimate of the true value and variance as N+1. For
> reasonably large N the difference is small but if you are concerned with weak
> images it might start to become significant.
>
> Cheers
>
> -- Ian
>
>
> On Tue, 12 Oct 2021 at 17:56, James Holton <[email protected]
> <mailto:[email protected]>> wrote:
> All my life I have believed that if you're counting photons then the
> error of observing N counts is sqrt(N). However, a calculation I just
> performed suggests its actually sqrt(N+1).
>
> My purpose here is to understand the weak-image limit of data
> processing. Question is: for a given pixel, if one photon is all you
> got, what do you "know"?
>
> I simulated millions of 1-second experiments. For each I used a "true"
> beam intensity (Itrue) between 0.001 and 20 photons/s. That is, for
> Itrue= 0.001 the average over a very long exposure would be 1 photon
> every 1000 seconds or so. For a 1-second exposure the observed count (N)
> is almost always zero. About 1 in 1000 of them will see one photon, and
> roughly 1 in a million will get N=2. I do 10,000 such experiments and
> put the results into a pile. I then repeat with Itrue=0.002,
> Itrue=0.003, etc. All the way up to Itrue = 20. At Itrue > 20 I never
> see N=1, not even in 1e7 experiments. With Itrue=0, I also see no N=1
> events.
> Now I go through my pile of results and extract those with N=1, and
> count up the number of times a given Itrue produced such an event. The
> histogram of Itrue values in this subset is itself Poisson, but with
> mean = 2 ! If I similarly count up events where 2 and only 2 photons
> were seen, the mean Itrue is 3. And if I look at only zero-count events
> the mean and standard deviation is unity.
>
> Does that mean the error of observing N counts is really sqrt(N+1) ?
>
> I admit that this little exercise assumes that the distribution of Itrue
> is uniform between 0.001 and 20, but given that one photon has been
> observed Itrue values outside this range are highly unlikely. The
> Itrue=0.001 and N=1 events are only a tiny fraction of the whole. So, I
> wold say that even if the prior distribution is not uniform, it is
> certainly bracketed. Now, Itrue=0 is possible if the shutter didn't
> open, but if the rest of the detector pixels have N=~1, doesn't this
> affect the prior distribution of Itrue on our pixel of interest?
>
> Of course, two or more photons are better than one, but these days with
> small crystals and big detectors N=1 is no longer a trivial situation.
> I look forward to hearing your take on this. And no, this is not a trick.
>
> -James Holton
> MAD Scientist
>
> ########################################################################
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> <https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
>
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB
> <http://www.jiscmail.ac.uk/CCP4BB>, a mailing list hosted by
> www.jiscmail.ac.uk <http://www.jiscmail.ac.uk/>, terms & conditions are
> available at https://www.jiscmail.ac.uk/policyandsecurity/
> <https://www.jiscmail.ac.uk/policyandsecurity/>
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> <https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
