Well I'll be...
Kay Diederichs pointed out to me off-list that the k+1 expectation and
variance from observing k photons is in "Bayesian Reasoning in Data
Analysis: A Critical Introduction" by Giulio D. Agostini. Granted, that
is with a uniform prior, which I take as the Bayesean equivalent of "I
have no idea".
So, if I'm looking to integrate a 10 x 10 patch of pixels on a weak
detector image, and I find that area has zero counts, what variance
shall I put on that observation? Is it:
a) zero
b) 1.0
c) 100
Wish I could say there are no wrong answers, but I think at least two of
those are incorrect,
-James Holton
MAD Scientist
On 10/13/2021 2:34 PM, Filipe Maia wrote:
I forgot to add probably the most important. James is correct, the
expected value of u, the true mean, given a single observation k is
indeed k+1 and k+1 is also the mean square error of using k+1 as the
estimator of the true mean.
Cheers,
Filipe
On Wed, 13 Oct 2021 at 23:17, Filipe Maia <fil...@xray.bmc.uu.se
<mailto:fil...@xray.bmc.uu.se>> wrote:
Hi,
The maximum likelihood estimator for a Poisson distributed
variable is equal to the mean of the observations. In the case of
a single observation, it will be equal to that observation. As
Graeme suggested, you can calculate the probability mass function
for a given observation with different Poisson parameters (i.e.
true means) and see that function peaks when the parameter matches
the observation.
The root mean squared error of the estimation of the true mean
from a single observation k seems to be sqrt(k+2). Or to put it in
another way, mean squared error, that is the expected value of
(k-u)**2, for an observation k and a true mean u, is equal to k+2.
You can see some example calculations at
https://colab.research.google.com/drive/1eoaNrDqaPnP-4FTGiNZxMllP7SFHkQuS?usp=sharing
<https://colab.research.google.com/drive/1eoaNrDqaPnP-4FTGiNZxMllP7SFHkQuS?usp=sharing>
Cheers,
Filipe
On Wed, 13 Oct 2021 at 17:14, Winter, Graeme (DLSLtd,RAL,LSCI)
<00006a19cead4548-dmarc-requ...@jiscmail.ac.uk
<mailto:00006a19cead4548-dmarc-requ...@jiscmail.ac.uk>> wrote:
This rang a bell to me last night, and I think you can derive
this from first principles
If you assume an observation of N counts, you can calculate
the probability of such an observation for a given Poisson
rate constant X. If you then integrate over all possible value
of X to work out the central value of the rate constant which
is most likely to result in an observation of N I think you
get X = N+1
I think it is the kind of calculation you can perform on a
napkin, if memory serves
All the best Graeme
On 13 Oct 2021, at 16:10, Andrew Leslie - MRC LMB
<and...@mrc-lmb.cam.ac.uk <mailto:and...@mrc-lmb.cam.ac.uk>>
wrote:
Hi Ian, James,
I have a strong feeling that I have
seen this result before, and it was due to Andy Hammersley at
ESRF. I’ve done a literature search and there is a paper
relating to errors in analysis of counting statistics (se
below), but I had a quick look at this and could not find the
(N+1) correction, so it must have been somewhere else. I Have
cc’d Andy on this Email (hoping that this Email address from
2016 still works) and maybe he can throw more light on this.
What I remember at the time I saw this was the simplicity of
the correction.
Cheers,
Andrew
Reducing bias in the analysis of counting statistics data
Hammersley, AP
<https://www.webofscience.com/wos/author/record/2665675>(Hammersley,
AP) Antoniadis, A
<https://www.webofscience.com/wos/author/record/13070551>(Antoniadis,
A)
NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION
A-ACCELERATORS SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT
Volume
394
Issue
1-2
Page
219-224
DOI
10.1016/S0168-9002(97)00668-2
Published
JUL 11 1997
On 12 Oct 2021, at 18:55, Ian Tickle <ianj...@gmail.com
<mailto:ianj...@gmail.com>> wrote:
Hi James
What the Poisson distribution tells you is that if the true
count is N then the expectation and variance are also N.
That's not the same thing as saying that for an observed
count N the expectation and variance are N. Consider all
those cases where the observed count is exactly zero. That
can arise from any number of true counts, though as you
noted larger values become increasingly unlikely. However
those true counts are all >= 0 which means that the mean and
variance of those true counts must be positive and
non-zero. From your results they are both 1 though I
haven't been through the algebra to prove it.
So what you are saying seems correct: for N observed counts
we should be taking the best estimate of the true value and
variance as N+1. For reasonably large N the difference is
small but if you are concerned with weak images it might
start to become significant.
Cheers
-- Ian
On Tue, 12 Oct 2021 at 17:56, James Holton <jmhol...@lbl.gov
<mailto:jmhol...@lbl.gov>> wrote:
All my life I have believed that if you're counting
photons then the
error of observing N counts is sqrt(N). However, a
calculation I just
performed suggests its actually sqrt(N+1).
My purpose here is to understand the weak-image limit of
data
processing. Question is: for a given pixel, if one
photon is all you
got, what do you "know"?
I simulated millions of 1-second experiments. For each I
used a "true"
beam intensity (Itrue) between 0.001 and 20 photons/s.
That is, for
Itrue= 0.001 the average over a very long exposure would
be 1 photon
every 1000 seconds or so. For a 1-second exposure the
observed count (N)
is almost always zero. About 1 in 1000 of them will see
one photon, and
roughly 1 in a million will get N=2. I do 10,000 such
experiments and
put the results into a pile. I then repeat with
Itrue=0.002,
Itrue=0.003, etc. All the way up to Itrue = 20. At Itrue
> 20 I never
see N=1, not even in 1e7 experiments. With Itrue=0, I
also see no N=1
events.
Now I go through my pile of results and extract those
with N=1, and
count up the number of times a given Itrue produced such
an event. The
histogram of Itrue values in this subset is itself
Poisson, but with
mean = 2 ! If I similarly count up events where 2 and
only 2 photons
were seen, the mean Itrue is 3. And if I look at only
zero-count events
the mean and standard deviation is unity.
Does that mean the error of observing N counts is really
sqrt(N+1) ?
I admit that this little exercise assumes that the
distribution of Itrue
is uniform between 0.001 and 20, but given that one
photon has been
observed Itrue values outside this range are highly
unlikely. The
Itrue=0.001 and N=1 events are only a tiny fraction of
the whole. So, I
wold say that even if the prior distribution is not
uniform, it is
certainly bracketed. Now, Itrue=0 is possible if the
shutter didn't
open, but if the rest of the detector pixels have N=~1,
doesn't this
affect the prior distribution of Itrue on our pixel of
interest?
Of course, two or more photons are better than one, but
these days with
small crystals and big detectors N=1 is no longer a
trivial situation.
I look forward to hearing your take on this. And no,
this is not a trick.
-James Holton
MAD Scientist
########################################################################
To unsubscribe from the CCP4BB list, click the following
link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
This message was issued to members of
www.jiscmail.ac.uk/CCP4BB
<http://www.jiscmail.ac.uk/CCP4BB>, a mailing list
hosted by www.jiscmail.ac.uk
<http://www.jiscmail.ac.uk/>, terms & conditions are
available at
https://www.jiscmail.ac.uk/policyandsecurity/
<https://www.jiscmail.ac.uk/policyandsecurity/>
------------------------------------------------------------------------
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
------------------------------------------------------------------------
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
--
This e-mail and any attachments may contain confidential,
copyright and or privileged material, and are for the use of
the intended addressee only. If you are not the intended
addressee or an authorised recipient of the addressee please
notify us of receipt by returning the e-mail and do not use,
copy, retain, distribute or disclose the information in or
attached to the e-mail.
Any opinions expressed within this e-mail are those of the
individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or
any attachments are free from viruses and we cannot accept
liability for any damage which you may sustain as a result of
software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered
in England and Wales with its registered office at Diamond
House, Harwell Science and Innovation Campus, Didcot,
Oxfordshire, OX11 0DE, United Kingdom
------------------------------------------------------------------------
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
------------------------------------------------------------------------
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
