Hi Kay Can I just confirm that your result Iobs=0.01 sigIobs=0.01 is the estimate of the true average intensity *per pixel* for a patch of 100 pixels? So then the total count for all 100 pixels is 1 with variance also 1, or in general for k observed counts in the patch, expectation = variance = k+1 for the total count, irrespective of the number of pixels? If so then that agrees with my own conclusion. It makes sense because Iobs=0.01 sigIobs=0.01 cannot come from a Poisson process (which obviously requires expectation = variance = an integer), whereas the total count does come from a Poisson process.
The difference from my approach is that you seem to have come at it via the individual pixel counts whereas I came straight from the Agostini result applied to the whole patch. The number of pixels seems to me to be irrelevant for the whole patch since the design of the detector, assuming it's an ideal detector with DQE = 1 surely cannot change the photon flux coming from the source: all ideal detectors whatever their pixel layout must give the same result. The number of pixels is then only relevant if one needs to know the average intensity per pixel, i.e. the total and s.d. divided by the number of pixels. Note the pixels here need not even correspond to the hardware pixels, they can be any arbitrary subdivision of the detector surface. Best wishes -- Ian On Tue, 19 Oct 2021 at 12:39, Kay Diederichs <[email protected]> wrote: > James, > > I am saying that my answer to "what is the expectation and variance if I > observe a 10x10 patch of pixels with zero > counts?" is Iobs=0.01 sigIobs=0.01 (and Iobs=sigIobs=1 if there is only > one pixel) IF the uniform prior applies. I agree with Gergely and others > that this prior (with its high expectation value and variance) appears > unrealistic. > > In your posting of Sat, 16 Oct 2021 12:00:30 -0700 you make a calculation > of Ppix that appears like a more suitable expectation value of a prior to > me. A suitable prior might then be 1/Ppix * e^(-l/Ppix) (Agostini §7.7.1). > The Bayesian argument is IIUC that the prior plays a minor role if you do > repeated measurements of the same value, because you use the posterior of > the first measurement as the prior for the second, and so on. What this > means is that your Ppix must play the role of a scale factor if you > consider the 100-pixel experiment. > However, for the 1-pixel experiment, having a more suitable prior should > be more important. > > best, > Kay > > > > > On Mon, 18 Oct 2021 12:40:45 -0700, James Holton <[email protected]> wrote: > > >Thank you very much for this Kay! > > > >So, to summarize, you are saying the answer to my question "what is the > >expectation and variance if I observe a 10x10 patch of pixels with zero > >counts?" is: > >Iobs = 0.01 > >sigIobs = 0.01 (defining sigIobs = sqrt(variance(Iobs))) > > > >And for the one-pixel case: > >Iobs = 1 > >sigIobs = 1 > > > >but in both cases the distribution is NOT Gaussian, but rather > >exponential. And that means adding variances may not be the way to > >propagate error. > > > >Is that right? > > > >-James Holton > >MAD Scientist > > > > > > > >On 10/18/2021 7:00 AM, Kay Diederichs wrote: > >> Hi James, > >> > >> I'm a bit behind ... > >> > >> My answer about the basic question ("a patch of 100 pixels each with > zero counts - what is the variance?") you ask is the following: > >> > >> 1) we all know the Poisson PDF (Probability Distribution Function) > P(k|l) = l^k*e^(-l)/k! (where k stands for for an integer >=0 and l is > lambda) which tells us the probability of observing k counts if we know l. > The PDF is normalized: SUM_over_k (P(k|l)) is 1 when k=0...infinity is 1. > >> 2) you don't know before the experiment what l is, and you assume it is > some number x with 0<=x<=xmax (the xmax limit can be calculated by looking > at the physics of the experiment; it is finite and less than the overload > value of the pixel, otherwise you should do a different experiment). Since > you don't know that number, all the x values are equally likely - you use a > uniform prior. > >> 3) what is the PDF P(l|k) of l if we observe k counts? That can be > found with Bayes theorem, and it turns out that (due to the uniform prior) > the right hand side of the formula looks the same as in 1) : P(l|k) = > l^k*e^(-l)/k! (again, the ! stands for the factorial, it is not a semantic > exclamation mark). This is eqs. 7.42 and 7.43 in Agostini "Bayesian > Reasoning in Data Analysis". > >> 3a) side note: if we calculate the expectation value for l, by > multiplying with l and integrating over l from 0 to infinity, we obtain > E(P(l|k))=k+1, and similarly for the variance (Agostini eqs 7.45 and 7.46) > >> 4) for k=0 (zero counts observed in a single pixel), this reduces to > P(l|0)=e^(-l) for a single observation (pixel). (this is basic math; see > also §7.4.1 of Agostini. > >> 5) since we have 100 independent pixels, we must multiply the > individual PDFs to get the overall PDF f, and also normalize to make the > integral over that PDF to be 1: the result is f(l|all 100 pixels are > 0)=n*e^(-n*l). (basic math). A more Bayesian procedure would be to realize > that the posterior PDF P(l|0)=e^(-l) of the first pixel should be used as > the prior for the second pixel, and so forth until the 100th pixel. This > has the same result f(l|all 100 pixels are 0)=n*e^(-n*l) (Agostini § 7.7.2)! > >> 6) the expectation value INTEGRAL_0_to_infinity over l*n*e^(-n*l) dl is > 1/n . This is 1 if n=1 as we know from 3a), and 1/100 for 100 pixels with > 0 counts. > >> 7) the variance is then INTEGRAL_0_to_infinity over > (l-1/n)^2*n*e^(-n*l) dl . This is 1/n^2 > >> > >> I find these results quite satisfactory. Please note that they deviate > from the MLE result: expectation value=0, variance=0 . The problem appears > to be that a Maximum Likelihood Estimator may give wrong results for small > n; something that I've read a couple of times but which appears not to be > universally known/taught. Clearly, the result in 6) and 7) for large n > converges towards 0, as it should be. > >> What this also means is that one should really work out the PDF instead > of just adding expectation values and variances (and arriving at 100 if all > 100 pixels have zero counts) because it is contradictory to use a uniform > prior for all the pixels if OTOH these agree perfectly in being 0! > >> > >> What this means for zero-dose extrapolation I have not thought about. > At least it prevents infinite weights! > >> > >> Best, > >> Kay > >> > >> > >> > >> > > > >######################################################################## > > > >To unsubscribe from the CCP4BB list, click the following link: > >https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > > >This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a > mailing list hosted by www.jiscmail.ac.uk, terms & conditions are > available at https://www.jiscmail.ac.uk/policyandsecurity/ > > ######################################################################## > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a > mailing list hosted by www.jiscmail.ac.uk, terms & conditions are > available at https://www.jiscmail.ac.uk/policyandsecurity/ > ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
