Re: [ccp4bb] am I doing this right?

[Marin van Heel]

Dear All,
In the context of this discussion, I suggest the readers / discussion
participants to have a look at new ideas on: Information Content of images,
SNR, DQEs, A priori Probabilities, Shannon's Channel Information
capacity,etc., expressed in:  Van Heel & Schatz, arXiv 2020. (There is also
an easy-to-find  Youtube lecture on the issue by "marin van heel").
Cheers,
Marin

On Tue, Oct 26, 2021 at 3:58 AM Jacob Keller  wrote:

> Hi All,
>
> haven't been following CCP4BB for a while, then I come back to this juicy
> Holtonian thread!
>
> Sorry for being more practical, but could you use a windowed approach:
> integrate values of the same pixel/relp combo (roxel?) over time (however
> that works with frame slicing) to estimate the error over many frames, then
> shrink the window incrementally, see whether the  successive
> plotted shrinking-window values show a trend? Most likely flat for
> background? This could be used for the spots as well as background. Would
> this lose the precious temporal info?
>
> Jacob
>
> On Fri, Oct 22, 2021 at 3:25 AM Gergely Katona 
> wrote:
>
>> Hi,
>>
>>
>>
>> I have more estimates to the same problem using a multinomial data
>> distribution. I should have realized that for prediction, I do not have to
>> deal with infinite likelihood of 0 trials when observing only 0s on an
>> image. Whenever 0 photons generated by the latent process, the image is
>> automatically empty. With this simplification, I still have to hide behind
>> mathematical convenience and use Gamma prior for the latent Poisson
>> process, but observing 0 counts just increments the beta parameter by 1
>> compared to the prior belief. With equal photon capture probabilities, the
>> mean counts are about 0.01 and the std is about 0.1 with
>> rate≈Gamma(alpha=1, beta=0.1) prior . With a symmetric Dirichlet prior to
>> the capture probabilities, the means appear unchanged, but the predicted
>> stds starts high at very low concentration parameter and level off at high
>> concentration parameter. This becomes more apparent at high photon counts
>> (high alpha of Gamma distribution). The answer is different if we look at
>> the std across the detector plane or across time of a single pixel.
>>
>> Details of the calculation below:
>>
>>
>>
>>
>> https://colab.research.google.com/drive/1NK43_3r1rH5lBTDS2rzIFDFNWqFfekrZ?usp=sharing
>>
>>
>>
>> Best wishes,
>>
>>
>>
>> Gergely
>>
>>
>>
>> Gergely Katona, Professor, Chairman of the Chemistry Program Council
>>
>> Department of Chemistry and Molecular Biology, University of Gothenburg
>>
>> Box 462, 40530 Göteborg, Sweden
>>
>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>>
>> Web: http://katonalab.eu, Email: gergely.kat...@gu.se
>>
>>
>>
>> *From:* CCP4 bulletin board  *On Behalf Of *Nave,
>> Colin (DLSLtd,RAL,LSCI)
>> *Sent:* 21 October, 2021 19:21
>> *To:* CCP4BB@JISCMAIL.AC.UK
>> *Subject:* Re: [ccp4bb] am I doing this right?
>>
>>
>>
>> Congratulations to James for starting this interesting discussion.
>>
>>
>>
>> For those who are like me, nowhere near a black belt in statistics, the
>> thread has included a number of distributions.  I have had to look up where
>> these apply and investigate their properties.
>>
>> As an example,
>>
>> “The Poisson distribution is used to model the # of events in the
>> future, Exponential distribution is used to predict the wait time until the
>> very first event, and Gamma distribution is used to predict the wait time
>> until the k-th event.”
>>
>> A useful calculator for distributions can be found at
>>
>> https://keisan.casio.com/menu/system/0540
>>
>> a specific example is at
>>
>> https://keisan.casio.com/exec/system/1180573179
>>
>> where cumulative probabilities for a Poisson distribution can be found
>> given values for x and lambda.
>>
>>
>>
>> The most appropriate prior is another issue which has come up e.g. is a
>> flat prior appropriate? I can see that a different prior would be
>> appropriate for different areas of the detector (e.g. 1 pixel instead of
>> 100 pixels) but the most appropriate prior seems a bit arbitrary to me. One
>> of James’ examples was 10^5 background photons distributed among  10^6
>> pixels – what is the most appropriate prior for this case? I presume it is
>> OK to update the prior after each observation but I understand that it can
>> create di

Re: [ccp4bb] am I doing this right?

[Jacob Keller]

Hi All,

haven't been following CCP4BB for a while, then I come back to this juicy
Holtonian thread!

Sorry for being more practical, but could you use a windowed approach:
integrate values of the same pixel/relp combo (roxel?) over time (however
that works with frame slicing) to estimate the error over many frames, then
shrink the window incrementally, see whether the  successive
plotted shrinking-window values show a trend? Most likely flat for
background? This could be used for the spots as well as background. Would
this lose the precious temporal info?

Jacob

On Fri, Oct 22, 2021 at 3:25 AM Gergely Katona  wrote:

> Hi,
>
>
>
> I have more estimates to the same problem using a multinomial data
> distribution. I should have realized that for prediction, I do not have to
> deal with infinite likelihood of 0 trials when observing only 0s on an
> image. Whenever 0 photons generated by the latent process, the image is
> automatically empty. With this simplification, I still have to hide behind
> mathematical convenience and use Gamma prior for the latent Poisson
> process, but observing 0 counts just increments the beta parameter by 1
> compared to the prior belief. With equal photon capture probabilities, the
> mean counts are about 0.01 and the std is about 0.1 with
> rate≈Gamma(alpha=1, beta=0.1) prior . With a symmetric Dirichlet prior to
> the capture probabilities, the means appear unchanged, but the predicted
> stds starts high at very low concentration parameter and level off at high
> concentration parameter. This becomes more apparent at high photon counts
> (high alpha of Gamma distribution). The answer is different if we look at
> the std across the detector plane or across time of a single pixel.
>
> Details of the calculation below:
>
>
>
>
> https://colab.research.google.com/drive/1NK43_3r1rH5lBTDS2rzIFDFNWqFfekrZ?usp=sharing
>
>
>
> Best wishes,
>
>
>
> Gergely
>
>
>
> Gergely Katona, Professor, Chairman of the Chemistry Program Council
>
> Department of Chemistry and Molecular Biology, University of Gothenburg
>
> Box 462, 40530 Göteborg, Sweden
>
> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>
> Web: http://katonalab.eu, Email: gergely.kat...@gu.se
>
>
>
> *From:* CCP4 bulletin board  *On Behalf Of *Nave,
> Colin (DLSLtd,RAL,LSCI)
> *Sent:* 21 October, 2021 19:21
> *To:* CCP4BB@JISCMAIL.AC.UK
> *Subject:* Re: [ccp4bb] am I doing this right?
>
>
>
> Congratulations to James for starting this interesting discussion.
>
>
>
> For those who are like me, nowhere near a black belt in statistics, the
> thread has included a number of distributions.  I have had to look up where
> these apply and investigate their properties.
>
> As an example,
>
> “The Poisson distribution is used to model the # of events in the future,
> Exponential distribution is used to predict the wait time until the very
> first event, and Gamma distribution is used to predict the wait time until
> the k-th event.”
>
> A useful calculator for distributions can be found at
>
> https://keisan.casio.com/menu/system/0540
>
> a specific example is at
>
> https://keisan.casio.com/exec/system/1180573179
>
> where cumulative probabilities for a Poisson distribution can be found
> given values for x and lambda.
>
>
>
> The most appropriate prior is another issue which has come up e.g. is a
> flat prior appropriate? I can see that a different prior would be
> appropriate for different areas of the detector (e.g. 1 pixel instead of
> 100 pixels) but the most appropriate prior seems a bit arbitrary to me. One
> of James’ examples was 10^5 background photons distributed among  10^6
> pixels – what is the most appropriate prior for this case? I presume it is
> OK to update the prior after each observation but I understand that it can
> create difficulties if not done properly.
>
>
>
> Being able to select the prior is sometimes seen as a strength of Bayesian
> methods. However, as a strong advocate of Bayesian methods once put it,
> this is a bit like Achilles boasting about his heel!
>
>
>
> I hope for some agreement among the black belts. It would be good to end
> up with some clarity about the most appropriate probability distributions
> and priors. Also, have we got clarity about the question being asked?
>
>
>
> Thanks to all for the interesting points.
>
>
>
> Colin
>
> *From:* CCP4 bulletin board  *On Behalf Of *Randy
> John Read
> *Sent:* 21 October 2021 13:23
> *To:* CCP4BB@JISCMAIL.AC.UK
> *Subject:* Re: [ccp4bb] am I doing this right?
>
>
>
> Hi Kay,
>
>
>
> No, I still think the answer should come out the same if you have 

Re: [ccp4bb] am I doing this right?

[Gergely Katona]

Hi,

I have more estimates to the same problem using a multinomial data 
distribution. I should have realized that for prediction, I do not have to deal 
with infinite likelihood of 0 trials when observing only 0s on an image. 
Whenever 0 photons generated by the latent process, the image is automatically 
empty. With this simplification, I still have to hide behind mathematical 
convenience and use Gamma prior for the latent Poisson process, but observing 0 
counts just increments the beta parameter by 1 compared to the prior belief. 
With equal photon capture probabilities, the mean counts are about 0.01 and the 
std is about 0.1 with rate≈Gamma(alpha=1, beta=0.1) prior . With a symmetric 
Dirichlet prior to the capture probabilities, the means appear unchanged, but 
the predicted stds starts high at very low concentration parameter and level 
off at high concentration parameter. This becomes more apparent at high photon 
counts (high alpha of Gamma distribution). The answer is different if we look 
at the std across the detector plane or across time of a single pixel.
Details of the calculation below:

https://colab.research.google.com/drive/1NK43_3r1rH5lBTDS2rzIFDFNWqFfekrZ?usp=sharing

Best wishes,

Gergely

Gergely Katona, Professor, Chairman of the Chemistry Program Council
Department of Chemistry and Molecular Biology, University of Gothenburg
Box 462, 40530 Göteborg, Sweden
Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
Web: http://katonalab.eu, Email: gergely.kat...@gu.se

From: CCP4 bulletin board  On Behalf Of Nave, Colin 
(DLSLtd,RAL,LSCI)
Sent: 21 October, 2021 19:21
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Congratulations to James for starting this interesting discussion.

For those who are like me, nowhere near a black belt in statistics, the thread 
has included a number of distributions.  I have had to look up where these 
apply and investigate their properties.
As an example,
“The Poisson distribution is used to model the # of events in the future, 
Exponential distribution is used to predict the wait time until the very first 
event, and Gamma distribution is used to predict the wait time until the k-th 
event.”
A useful calculator for distributions can be found at
https://keisan.casio.com/menu/system/0540
a specific example is at
https://keisan.casio.com/exec/system/1180573179
where cumulative probabilities for a Poisson distribution can be found given 
values for x and lambda.

The most appropriate prior is another issue which has come up e.g. is a flat 
prior appropriate? I can see that a different prior would be appropriate for 
different areas of the detector (e.g. 1 pixel instead of 100 pixels) but the 
most appropriate prior seems a bit arbitrary to me. One of James’ examples was 
10^5 background photons distributed among  10^6 pixels – what is the most 
appropriate prior for this case? I presume it is OK to update the prior after 
each observation but I understand that it can create difficulties if not done 
properly.

Being able to select the prior is sometimes seen as a strength of Bayesian 
methods. However, as a strong advocate of Bayesian methods once put it, this is 
a bit like Achilles boasting about his heel!

I hope for some agreement among the black belts. It would be good to end up 
with some clarity about the most appropriate probability distributions and 
priors. Also, have we got clarity about the question being asked?

Thanks to all for the interesting points.

Colin
From: CCP4 bulletin board mailto:CCP4BB@JISCMAIL.AC.UK>> 
On Behalf Of Randy John Read
Sent: 21 October 2021 13:23
To: CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>
Subject: Re: [ccp4bb] am I doing this right?

Hi Kay,

No, I still think the answer should come out the same if you have good reason 
to believe that all the 100 pixels are equally likely to receive a photon (for 
instance because your knowledge of the geometry of the source and the detector 
says the difference in their positions is insignificant, i.e. part of your 
prior expectation). Unless the exact position of the spot where you detect the 
photon is relevant, detecting 1 photon on a big pixel and detecting the same 
photon on 1 of 100 smaller pixels covering the same area are equivalent events. 
What should be different in the analysis, if you're thinking about individual 
pixels, is that the expected value for a photon landing on any of the pixels 
will be 100 times lower for each of the smaller pixels than the single big 
pixel, so that the expected value of their sum is the same. You won't get to 
that conclusion without having a different prior probability for the two cases 
that reflects the 100-fold lower flux through the smaller area, regardless of 
the total power of the source.

Best wishes,
Randy

On 21 Oct 2021, at 13:03, Kay Diederichs 
mailto:kay.diederi...@uni-konstanz.de>> wrote:

Randy,

I must admit that I am not certain about my 

Re: [ccp4bb] am I doing this right?

[Gergely Katona]

Dear Colin,

I think you are right: the question is quite open to interpretation.

Here is a Bayesian model, which shows that finding 100 pixels with 0 has almost 
exactly 42% chance (which should come to no surprise to the fans of 
Hitchhiker’s guide to Galaxy).

https://colab.research.google.com/drive/1nvYxA8cG17hmE3uFPBUZIfHTGoFeXhAU?usp=sharing

I am sorry about the trolling here, but I hope this illustrates that the 
question about the True model might be misplaced for a Bayesian.

Here are some other answers that relates to James’ idea about time dependent 
rates. This shows that the expectation varies from time to time and at time 0 
it is exactly 0! It is all due to the constraint of the model not to the priors.

https://colab.research.google.com/drive/1J6K3-Wgz_4PMVlJgItdQNzqZmb0SfsKx?usp=sharing

Here is another example where even the intercept is free to vary. Pymc3 start 
to struggle a bit and the priors get more and more important.

https://colab.research.google.com/drive/1ECnbThFBaa4HoT25kFSSm8pxfXnzY6Ty?usp=sharing

As for big question what is THE expectation of 100 pixels with 0 counts, 
someone else has to answer that. If I had a black belt in statistics I would 
construct a modified Multinomial likelihood function that includes 0 trials, 
because I am that stubborn.

Best wishes,

Gergely



Gergely Katona, Professor, Chairman of the Chemistry Program Council
Department of Chemistry and Molecular Biology, University of Gothenburg
Box 462, 40530 Göteborg, Sweden
Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
Web: http://katonalab.eu, Email: gergely.kat...@gu.se

From: CCP4 bulletin board  On Behalf Of Nave, Colin 
(DLSLtd,RAL,LSCI)
Sent: 21 October, 2021 19:21
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Congratulations to James for starting this interesting discussion.

For those who are like me, nowhere near a black belt in statistics, the thread 
has included a number of distributions.  I have had to look up where these 
apply and investigate their properties.
As an example,
“The Poisson distribution is used to model the # of events in the future, 
Exponential distribution is used to predict the wait time until the very first 
event, and Gamma distribution is used to predict the wait time until the k-th 
event.”
A useful calculator for distributions can be found at
https://keisan.casio.com/menu/system/0540
a specific example is at
https://keisan.casio.com/exec/system/1180573179
where cumulative probabilities for a Poisson distribution can be found given 
values for x and lambda.

The most appropriate prior is another issue which has come up e.g. is a flat 
prior appropriate? I can see that a different prior would be appropriate for 
different areas of the detector (e.g. 1 pixel instead of 100 pixels) but the 
most appropriate prior seems a bit arbitrary to me. One of James’ examples was 
10^5 background photons distributed among  10^6 pixels – what is the most 
appropriate prior for this case? I presume it is OK to update the prior after 
each observation but I understand that it can create difficulties if not done 
properly.

Being able to select the prior is sometimes seen as a strength of Bayesian 
methods. However, as a strong advocate of Bayesian methods once put it, this is 
a bit like Achilles boasting about his heel!

I hope for some agreement among the black belts. It would be good to end up 
with some clarity about the most appropriate probability distributions and 
priors. Also, have we got clarity about the question being asked?

Thanks to all for the interesting points.

Colin
From: CCP4 bulletin board mailto:CCP4BB@JISCMAIL.AC.UK>> 
On Behalf Of Randy John Read
Sent: 21 October 2021 13:23
To: CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>
Subject: Re: [ccp4bb] am I doing this right?

Hi Kay,

No, I still think the answer should come out the same if you have good reason 
to believe that all the 100 pixels are equally likely to receive a photon (for 
instance because your knowledge of the geometry of the source and the detector 
says the difference in their positions is insignificant, i.e. part of your 
prior expectation). Unless the exact position of the spot where you detect the 
photon is relevant, detecting 1 photon on a big pixel and detecting the same 
photon on 1 of 100 smaller pixels covering the same area are equivalent events. 
What should be different in the analysis, if you're thinking about individual 
pixels, is that the expected value for a photon landing on any of the pixels 
will be 100 times lower for each of the smaller pixels than the single big 
pixel, so that the expected value of their sum is the same. You won't get to 
that conclusion without having a different prior probability for the two cases 
that reflects the 100-fold lower flux through the smaller area, regardless of 
the total power of the source.

Best wishes,
Randy

On 21 Oct 2021, at 13:03, Kay Diederichs 
mailto:kay.di

Re: [ccp4bb] am I doing this right?

[Nave, Colin (DLSLtd,RAL,LSCI)]

Congratulations to James for starting this interesting discussion.

For those who are like me, nowhere near a black belt in statistics, the thread 
has included a number of distributions.  I have had to look up where these 
apply and investigate their properties.
As an example,
“The Poisson distribution is used to model the # of events in the future, 
Exponential distribution is used to predict the wait time until the very first 
event, and Gamma distribution is used to predict the wait time until the k-th 
event.”
A useful calculator for distributions can be found at
https://keisan.casio.com/menu/system/0540
a specific example is at
https://keisan.casio.com/exec/system/1180573179
where cumulative probabilities for a Poisson distribution can be found given 
values for x and lambda.

The most appropriate prior is another issue which has come up e.g. is a flat 
prior appropriate? I can see that a different prior would be appropriate for 
different areas of the detector (e.g. 1 pixel instead of 100 pixels) but the 
most appropriate prior seems a bit arbitrary to me. One of James’ examples was 
10^5 background photons distributed among  10^6 pixels – what is the most 
appropriate prior for this case? I presume it is OK to update the prior after 
each observation but I understand that it can create difficulties if not done 
properly.

Being able to select the prior is sometimes seen as a strength of Bayesian 
methods. However, as a strong advocate of Bayesian methods once put it, this is 
a bit like Achilles boasting about his heel!

I hope for some agreement among the black belts. It would be good to end up 
with some clarity about the most appropriate probability distributions and 
priors. Also, have we got clarity about the question being asked?

Thanks to all for the interesting points.

Colin
From: CCP4 bulletin board  On Behalf Of Randy John Read
Sent: 21 October 2021 13:23
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Hi Kay,

No, I still think the answer should come out the same if you have good reason 
to believe that all the 100 pixels are equally likely to receive a photon (for 
instance because your knowledge of the geometry of the source and the detector 
says the difference in their positions is insignificant, i.e. part of your 
prior expectation). Unless the exact position of the spot where you detect the 
photon is relevant, detecting 1 photon on a big pixel and detecting the same 
photon on 1 of 100 smaller pixels covering the same area are equivalent events. 
What should be different in the analysis, if you're thinking about individual 
pixels, is that the expected value for a photon landing on any of the pixels 
will be 100 times lower for each of the smaller pixels than the single big 
pixel, so that the expected value of their sum is the same. You won't get to 
that conclusion without having a different prior probability for the two cases 
that reflects the 100-fold lower flux through the smaller area, regardless of 
the total power of the source.

Best wishes,
Randy


On 21 Oct 2021, at 13:03, Kay Diederichs 
mailto:kay.diederi...@uni-konstanz.de>> wrote:

Randy,

I must admit that I am not certain about my answer, but I lean toward thinking 
that the result (of the two thought experiments that you describe) is not the 
same. I do agree that it makes sense that the expectation value is the same, 
and the math that I sketched in 
https://www.jiscmail.ac.uk/cgi-bin/wa-jisc.exe?A2=CCP4BB;bdd31b04.2110 actually 
shows this. But the variance? To me, a 100-pixel patch with all zeroes is no 
different from sequentially observing 100 pixels, one after the other. For the 
first of these pixels, I have no idea what the count is, until I observe it. 
For the second, I am less surprised that it is 0 because I observed 0 for the 
first. And so on, until the 100th. For the last one, my belief that I will 
observe a zero before I read out the pixel is much higher than for the first 
pixel. The variance is just the inverse of the amount of error (squared) that 
we assign to our belief in the expectation value. And that amount of belief is 
very different. I find it satisfactory that the sigma goes down with the sqrt() 
of the number of pixels.

Also, I don't find an error in the math of my posting of Mon, 18 Oct 2021 
15:00:42 +0100 . I do think that a uniform prior is not realistic, but this 
does not seem to make much difference for the 100-pixel thought experiment.

We could change the thought experiment in the following way - you observe 99 
pixels with zero counts, and 1 with 1 count. Would you still say that both the 
big-pixel-single-observation and the 100-pixel experiment should give 
expectation value of 2 and variance of 2? I wouldn't.

Best wishes,
Kay

On Thu, 21 Oct 2021 09:00:23 +, Randy John Read 
mailto:rj...@cam.ac.uk>> wrote:


Just to be a bit clearer, I mean that the calculation of the expected value and 
its variance should give the

Re: [ccp4bb] am I doing this right?

[Gergely Katona]

I am sorry, this was a dead-end idea multinomial distribution with 0 trials is 
not defined.

Gergely

From: CCP4 bulletin board  On Behalf Of Gergely Katona
Sent: 21 October, 2021 15:29
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Dear Randy and Kay,

My solution would involve a multinomial distribution for assigning the counts 
to pixels.
Something like this:

rate  ≈  Gamma(1,1)
total_counts  ≈ Poisson(rate)
probs ≈ Dirichlet (alpha=1, for all pixels)
pixel_counts  ≈ Multinomial (total_counts, probs of the different pixels) # 
fixed observations

Here I assume that some process generates a random number of total photons 
through a Poisson process. These are distributed across the different pixels 
through a multinomial process. I do not lock them to have equal probability to 
capture the photon, but the prior have the same concentration parameter for all 
pixels. If I am very confident that the capture probabilities are the same, I 
can always increase alphas of the Dirichlet distribution uniformly or just 
simply replace probs with 1/(total_number of pixels) for all the pixels. 
Unfortunately, I do not have a black belt in statistics so I am not sure what 
the closed form of the posterior probabilities would be (if such exist).


Best wishes,

Gergely



From: CCP4 bulletin board mailto:CCP4BB@JISCMAIL.AC.UK>> 
On Behalf Of Randy John Read
Sent: 21 October, 2021 14:23
To: CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>
Subject: Re: [ccp4bb] am I doing this right?

Hi Kay,

No, I still think the answer should come out the same if you have good reason 
to believe that all the 100 pixels are equally likely to receive a photon (for 
instance because your knowledge of the geometry of the source and the detector 
says the difference in their positions is insignificant, i.e. part of your 
prior expectation). Unless the exact position of the spot where you detect the 
photon is relevant, detecting 1 photon on a big pixel and detecting the same 
photon on 1 of 100 smaller pixels covering the same area are equivalent events. 
What should be different in the analysis, if you're thinking about individual 
pixels, is that the expected value for a photon landing on any of the pixels 
will be 100 times lower for each of the smaller pixels than the single big 
pixel, so that the expected value of their sum is the same. You won't get to 
that conclusion without having a different prior probability for the two cases 
that reflects the 100-fold lower flux through the smaller area, regardless of 
the total power of the source.

Best wishes,
Randy

On 21 Oct 2021, at 13:03, Kay Diederichs 
mailto:kay.diederi...@uni-konstanz.de>> wrote:

Randy,

I must admit that I am not certain about my answer, but I lean toward thinking 
that the result (of the two thought experiments that you describe) is not the 
same. I do agree that it makes sense that the expectation value is the same, 
and the math that I sketched in 
https://www.jiscmail.ac.uk/cgi-bin/wa-jisc.exe?A2=CCP4BB;bdd31b04.2110 actually 
shows this. But the variance? To me, a 100-pixel patch with all zeroes is no 
different from sequentially observing 100 pixels, one after the other. For the 
first of these pixels, I have no idea what the count is, until I observe it. 
For the second, I am less surprised that it is 0 because I observed 0 for the 
first. And so on, until the 100th. For the last one, my belief that I will 
observe a zero before I read out the pixel is much higher than for the first 
pixel. The variance is just the inverse of the amount of error (squared) that 
we assign to our belief in the expectation value. And that amount of belief is 
very different. I find it satisfactory that the sigma goes down with the sqrt() 
of the number of pixels.

Also, I don't find an error in the math of my posting of Mon, 18 Oct 2021 
15:00:42 +0100 . I do think that a uniform prior is not realistic, but this 
does not seem to make much difference for the 100-pixel thought experiment.

We could change the thought experiment in the following way - you observe 99 
pixels with zero counts, and 1 with 1 count. Would you still say that both the 
big-pixel-single-observation and the 100-pixel experiment should give 
expectation value of 2 and variance of 2? I wouldn't.

Best wishes,
Kay

On Thu, 21 Oct 2021 09:00:23 +, Randy John Read 
mailto:rj...@cam.ac.uk>> wrote:

Just to be a bit clearer, I mean that the calculation of the expected value and 
its variance should give the same answer if you're comparing one pixel for a 
particular length of exposure with the sum obtained from either a larger number 
of smaller pixels covering the same area for the same length of exposure, or 
the sum from the same pixel measured for smaller time slices adding up to the 
same total exposure.

On 21 Oct 2021, at 09:54, Randy John Read 
mailto:rj...@cam.ac.uk><mailto:rj...@cam.ac.uk>> wrote:

I would think that if this p

Re: [ccp4bb] am I doing this right?

[Gergely Katona]

Dear Randy and Kay,

My solution would involve a multinomial distribution for assigning the counts 
to pixels.
Something like this:

rate  ≈  Gamma(1,1)
total_counts  ≈ Poisson(rate)
probs ≈ Dirichlet (alpha=1, for all pixels)
pixel_counts  ≈ Multinomial (total_counts, probs of the different pixels) # 
fixed observations

Here I assume that some process generates a random number of total photons 
through a Poisson process. These are distributed across the different pixels 
through a multinomial process. I do not lock them to have equal probability to 
capture the photon, but the prior have the same concentration parameter for all 
pixels. If I am very confident that the capture probabilities are the same, I 
can always increase alphas of the Dirichlet distribution uniformly or just 
simply replace probs with 1/(total_number of pixels) for all the pixels. 
Unfortunately, I do not have a black belt in statistics so I am not sure what 
the closed form of the posterior probabilities would be (if such exist).


Best wishes,

Gergely



From: CCP4 bulletin board  On Behalf Of Randy John Read
Sent: 21 October, 2021 14:23
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Hi Kay,

No, I still think the answer should come out the same if you have good reason 
to believe that all the 100 pixels are equally likely to receive a photon (for 
instance because your knowledge of the geometry of the source and the detector 
says the difference in their positions is insignificant, i.e. part of your 
prior expectation). Unless the exact position of the spot where you detect the 
photon is relevant, detecting 1 photon on a big pixel and detecting the same 
photon on 1 of 100 smaller pixels covering the same area are equivalent events. 
What should be different in the analysis, if you're thinking about individual 
pixels, is that the expected value for a photon landing on any of the pixels 
will be 100 times lower for each of the smaller pixels than the single big 
pixel, so that the expected value of their sum is the same. You won't get to 
that conclusion without having a different prior probability for the two cases 
that reflects the 100-fold lower flux through the smaller area, regardless of 
the total power of the source.

Best wishes,
Randy


On 21 Oct 2021, at 13:03, Kay Diederichs 
mailto:kay.diederi...@uni-konstanz.de>> wrote:

Randy,

I must admit that I am not certain about my answer, but I lean toward thinking 
that the result (of the two thought experiments that you describe) is not the 
same. I do agree that it makes sense that the expectation value is the same, 
and the math that I sketched in 
https://www.jiscmail.ac.uk/cgi-bin/wa-jisc.exe?A2=CCP4BB;bdd31b04.2110 actually 
shows this. But the variance? To me, a 100-pixel patch with all zeroes is no 
different from sequentially observing 100 pixels, one after the other. For the 
first of these pixels, I have no idea what the count is, until I observe it. 
For the second, I am less surprised that it is 0 because I observed 0 for the 
first. And so on, until the 100th. For the last one, my belief that I will 
observe a zero before I read out the pixel is much higher than for the first 
pixel. The variance is just the inverse of the amount of error (squared) that 
we assign to our belief in the expectation value. And that amount of belief is 
very different. I find it satisfactory that the sigma goes down with the sqrt() 
of the number of pixels.

Also, I don't find an error in the math of my posting of Mon, 18 Oct 2021 
15:00:42 +0100 . I do think that a uniform prior is not realistic, but this 
does not seem to make much difference for the 100-pixel thought experiment.

We could change the thought experiment in the following way - you observe 99 
pixels with zero counts, and 1 with 1 count. Would you still say that both the 
big-pixel-single-observation and the 100-pixel experiment should give 
expectation value of 2 and variance of 2? I wouldn't.

Best wishes,
Kay

On Thu, 21 Oct 2021 09:00:23 +, Randy John Read 
mailto:rj...@cam.ac.uk>> wrote:


Just to be a bit clearer, I mean that the calculation of the expected value and 
its variance should give the same answer if you're comparing one pixel for a 
particular length of exposure with the sum obtained from either a larger number 
of smaller pixels covering the same area for the same length of exposure, or 
the sum from the same pixel measured for smaller time slices adding up to the 
same total exposure.

On 21 Oct 2021, at 09:54, Randy John Read 
mailto:rj...@cam.ac.uk><mailto:rj...@cam.ac.uk>> wrote:

I would think that if this problem is being approached correctly, with the 
right prior, it shouldn't matter whether you collect the same signal 
distributed over 100 smaller pixels or the same pixel measured for the same 
length of exposure but with 100 time slices; you should get the same answer. So 
I would want to formulate the problem in a way where this 

Re: [ccp4bb] am I doing this right?

[Randy John Read]

Hi Kay,

No, I still think the answer should come out the same if you have good reason 
to believe that all the 100 pixels are equally likely to receive a photon (for 
instance because your knowledge of the geometry of the source and the detector 
says the difference in their positions is insignificant, i.e. part of your 
prior expectation). Unless the exact position of the spot where you detect the 
photon is relevant, detecting 1 photon on a big pixel and detecting the same 
photon on 1 of 100 smaller pixels covering the same area are equivalent events. 
What should be different in the analysis, if you're thinking about individual 
pixels, is that the expected value for a photon landing on any of the pixels 
will be 100 times lower for each of the smaller pixels than the single big 
pixel, so that the expected value of their sum is the same. You won't get to 
that conclusion without having a different prior probability for the two cases 
that reflects the 100-fold lower flux through the smaller area, regardless of 
the total power of the source.

Best wishes,
Randy

On 21 Oct 2021, at 13:03, Kay Diederichs 
mailto:kay.diederi...@uni-konstanz.de>> wrote:

Randy,

I must admit that I am not certain about my answer, but I lean toward thinking 
that the result (of the two thought experiments that you describe) is not the 
same. I do agree that it makes sense that the expectation value is the same, 
and the math that I sketched in 
https://www.jiscmail.ac.uk/cgi-bin/wa-jisc.exe?A2=CCP4BB;bdd31b04.2110 actually 
shows this. But the variance? To me, a 100-pixel patch with all zeroes is no 
different from sequentially observing 100 pixels, one after the other. For the 
first of these pixels, I have no idea what the count is, until I observe it. 
For the second, I am less surprised that it is 0 because I observed 0 for the 
first. And so on, until the 100th. For the last one, my belief that I will 
observe a zero before I read out the pixel is much higher than for the first 
pixel. The variance is just the inverse of the amount of error (squared) that 
we assign to our belief in the expectation value. And that amount of belief is 
very different. I find it satisfactory that the sigma goes down with the sqrt() 
of the number of pixels.

Also, I don't find an error in the math of my posting of Mon, 18 Oct 2021 
15:00:42 +0100 . I do think that a uniform prior is not realistic, but this 
does not seem to make much difference for the 100-pixel thought experiment.

We could change the thought experiment in the following way - you observe 99 
pixels with zero counts, and 1 with 1 count. Would you still say that both the 
big-pixel-single-observation and the 100-pixel experiment should give 
expectation value of 2 and variance of 2? I wouldn't.

Best wishes,
Kay

On Thu, 21 Oct 2021 09:00:23 +, Randy John Read 
mailto:rj...@cam.ac.uk>> wrote:

Just to be a bit clearer, I mean that the calculation of the expected value and 
its variance should give the same answer if you're comparing one pixel for a 
particular length of exposure with the sum obtained from either a larger number 
of smaller pixels covering the same area for the same length of exposure, or 
the sum from the same pixel measured for smaller time slices adding up to the 
same total exposure.

On 21 Oct 2021, at 09:54, Randy John Read 
mailto:rj...@cam.ac.uk>> wrote:

I would think that if this problem is being approached correctly, with the 
right prior, it shouldn't matter whether you collect the same signal 
distributed over 100 smaller pixels or the same pixel measured for the same 
length of exposure but with 100 time slices; you should get the same answer. So 
I would want to formulate the problem in a way where this invariance is 
satisfied. I thought it was, from some of the earlier descriptions of the 
problem, but this sounds worrying.

I think you're trying to say the same thing here, Kay. Is that right?

Best wishes,

Randy

On 21 Oct 2021, at 08:51, Kay Diederichs 
mailto:kay.diederi...@uni-konstanz.de>>
 wrote:

Hi Ian,

it is Iobs=0.01 and sigIobs=0.01 for one pixel, but adding 100 pixels each with 
variance=sigIobs^2=0.0001 gives  0.01 , yielding a 100-pixel-sigIobs of 0.1 - 
different from the 1 you get. As if repeatedly observing the same count of 0 
lowers the estimated error by sqrt(n), where n is the number of observations 
(100 in this case).

best wishes,
Kay

On Wed, 20 Oct 2021 13:08:33 +0100, Ian Tickle 
mailto:ianj...@gmail.com>> wrote:

Hi Kay

Can I just confirm that your result Iobs=0.01 sigIobs=0.01 is the estimate
of the true average intensity *per pixel* for a patch of 100 pixels?  So
then the total count for all 100 pixels is 1 with variance also 1, or in
general for k observed counts in the patch, expectation = variance = k+1
for the total count, irrespective of the number of pixels?  If so then that
agrees with my own conclusion.  

Re: [ccp4bb] am I doing this right?

[Kay Diederichs]

Randy,

I must admit that I am not certain about my answer, but I lean toward thinking 
that the result (of the two thought experiments that you describe) is not the 
same. I do agree that it makes sense that the expectation value is the same, 
and the math that I sketched in 
https://www.jiscmail.ac.uk/cgi-bin/wa-jisc.exe?A2=CCP4BB;bdd31b04.2110 actually 
shows this. But the variance? To me, a 100-pixel patch with all zeroes is no 
different from sequentially observing 100 pixels, one after the other. For the 
first of these pixels, I have no idea what the count is, until I observe it. 
For the second, I am less surprised that it is 0 because I observed 0 for the 
first. And so on, until the 100th. For the last one, my belief that I will 
observe a zero before I read out the pixel is much higher than for the first 
pixel. The variance is just the inverse of the amount of error (squared) that 
we assign to our belief in the expectation value. And that amount of belief is 
very different. I find it satisfactory that the sigma goes down with the sqrt() 
of the number of pixels.

Also, I don't find an error in the math of my posting of Mon, 18 Oct 2021 
15:00:42 +0100 . I do think that a uniform prior is not realistic, but this 
does not seem to make much difference for the 100-pixel thought experiment.

We could change the thought experiment in the following way - you observe 99 
pixels with zero counts, and 1 with 1 count. Would you still say that both the 
big-pixel-single-observation and the 100-pixel experiment should give 
expectation value of 2 and variance of 2? I wouldn't.

Best wishes,
Kay

On Thu, 21 Oct 2021 09:00:23 +, Randy John Read  wrote:

>Just to be a bit clearer, I mean that the calculation of the expected value 
>and its variance should give the same answer if you're comparing one pixel for 
>a particular length of exposure with the sum obtained from either a larger 
>number of smaller pixels covering the same area for the same length of 
>exposure, or the sum from the same pixel measured for smaller time slices 
>adding up to the same total exposure.
>
>On 21 Oct 2021, at 09:54, Randy John Read 
>mailto:rj...@cam.ac.uk>> wrote:
>
>I would think that if this problem is being approached correctly, with the 
>right prior, it shouldn't matter whether you collect the same signal 
>distributed over 100 smaller pixels or the same pixel measured for the same 
>length of exposure but with 100 time slices; you should get the same answer. 
>So I would want to formulate the problem in a way where this invariance is 
>satisfied. I thought it was, from some of the earlier descriptions of the 
>problem, but this sounds worrying.
>
>I think you're trying to say the same thing here, Kay. Is that right?
>
>Best wishes,
>
>Randy
>
>On 21 Oct 2021, at 08:51, Kay Diederichs 
>mailto:kay.diederi...@uni-konstanz.de>> wrote:
>
>Hi Ian,
>
>it is Iobs=0.01 and sigIobs=0.01 for one pixel, but adding 100 pixels each 
>with variance=sigIobs^2=0.0001 gives  0.01 , yielding a 100-pixel-sigIobs of 
>0.1 - different from the 1 you get. As if repeatedly observing the same count 
>of 0 lowers the estimated error by sqrt(n), where n is the number of 
>observations (100 in this case).
>
>best wishes,
>Kay
>
>On Wed, 20 Oct 2021 13:08:33 +0100, Ian Tickle 
>mailto:ianj...@gmail.com>> wrote:
>
>Hi Kay
>
>Can I just confirm that your result Iobs=0.01 sigIobs=0.01 is the estimate
>of the true average intensity *per pixel* for a patch of 100 pixels?  So
>then the total count for all 100 pixels is 1 with variance also 1, or in
>general for k observed counts in the patch, expectation = variance = k+1
>for the total count, irrespective of the number of pixels?  If so then that
>agrees with my own conclusion.  It makes sense because Iobs=0.01
>sigIobs=0.01 cannot come from a Poisson process (which obviously requires
>expectation = variance = an integer), whereas the total count does come
>from a Poisson process.
>
>The difference from my approach is that you seem to have come at it via the
>individual pixel counts whereas I came straight from the Agostini result
>applied to the whole patch.  The number of pixels seems to me to be
>irrelevant for the whole patch since the design of the detector, assuming
>it's an ideal detector with DQE = 1 surely cannot change the photon flux
>coming from the source: all ideal detectors whatever their pixel layout
>must give the same result.  The number of pixels is then only relevant if
>one needs to know the average intensity per pixel, i.e. the total and s.d.
>divided by the number of pixels.  Note the pixels here need not even
>correspond to the hardware pixels, they can be any arbitrary subdivision of
>the detector surface.
>
>Best wishes
>
>-- Ian
>
>
>On Tue, 19 Oct 2021 at 12:39, Kay Diederichs 
>mailto:kay.diederi...@uni-konstanz.de>>
>wrote:
>
>James,
>
>I am saying that my answer to "what is the expectation and variance if I
>observe a 10x10 patch of pixels with zero
>counts?" is 

Re: [ccp4bb] am I doing this right?

[Randy John Read]

Just to be a bit clearer, I mean that the calculation of the expected value and 
its variance should give the same answer if you're comparing one pixel for a 
particular length of exposure with the sum obtained from either a larger number 
of smaller pixels covering the same area for the same length of exposure, or 
the sum from the same pixel measured for smaller time slices adding up to the 
same total exposure.

On 21 Oct 2021, at 09:54, Randy John Read 
mailto:rj...@cam.ac.uk>> wrote:

I would think that if this problem is being approached correctly, with the 
right prior, it shouldn't matter whether you collect the same signal 
distributed over 100 smaller pixels or the same pixel measured for the same 
length of exposure but with 100 time slices; you should get the same answer. So 
I would want to formulate the problem in a way where this invariance is 
satisfied. I thought it was, from some of the earlier descriptions of the 
problem, but this sounds worrying.

I think you're trying to say the same thing here, Kay. Is that right?

Best wishes,

Randy

On 21 Oct 2021, at 08:51, Kay Diederichs 
mailto:kay.diederi...@uni-konstanz.de>> wrote:

Hi Ian,

it is Iobs=0.01 and sigIobs=0.01 for one pixel, but adding 100 pixels each with 
variance=sigIobs^2=0.0001 gives  0.01 , yielding a 100-pixel-sigIobs of 0.1 - 
different from the 1 you get. As if repeatedly observing the same count of 0 
lowers the estimated error by sqrt(n), where n is the number of observations 
(100 in this case).

best wishes,
Kay

On Wed, 20 Oct 2021 13:08:33 +0100, Ian Tickle 
mailto:ianj...@gmail.com>> wrote:

Hi Kay

Can I just confirm that your result Iobs=0.01 sigIobs=0.01 is the estimate
of the true average intensity *per pixel* for a patch of 100 pixels?  So
then the total count for all 100 pixels is 1 with variance also 1, or in
general for k observed counts in the patch, expectation = variance = k+1
for the total count, irrespective of the number of pixels?  If so then that
agrees with my own conclusion.  It makes sense because Iobs=0.01
sigIobs=0.01 cannot come from a Poisson process (which obviously requires
expectation = variance = an integer), whereas the total count does come
from a Poisson process.

The difference from my approach is that you seem to have come at it via the
individual pixel counts whereas I came straight from the Agostini result
applied to the whole patch.  The number of pixels seems to me to be
irrelevant for the whole patch since the design of the detector, assuming
it's an ideal detector with DQE = 1 surely cannot change the photon flux
coming from the source: all ideal detectors whatever their pixel layout
must give the same result.  The number of pixels is then only relevant if
one needs to know the average intensity per pixel, i.e. the total and s.d.
divided by the number of pixels.  Note the pixels here need not even
correspond to the hardware pixels, they can be any arbitrary subdivision of
the detector surface.

Best wishes

-- Ian


On Tue, 19 Oct 2021 at 12:39, Kay Diederichs 
mailto:kay.diederi...@uni-konstanz.de>>
wrote:

James,

I am saying that my answer to "what is the expectation and variance if I
observe a 10x10 patch of pixels with zero
counts?" is Iobs=0.01 sigIobs=0.01 (and Iobs=sigIobs=1 if there is only
one pixel) IF the uniform prior applies. I agree with Gergely and others
that this prior (with its high expectation value and variance) appears
unrealistic.

In your posting of Sat, 16 Oct 2021 12:00:30 -0700 you make a calculation
of Ppix that appears like a more suitable expectation value of a prior to
me. A suitable prior might then be 1/Ppix * e^(-l/Ppix) (Agostini §7.7.1).
The Bayesian argument is IIUC that the prior plays a minor role if you do
repeated measurements of the same value, because you use the posterior of
the first measurement as the prior for the second, and so on. What this
means is that your Ppix must play the role of a scale factor if you
consider the 100-pixel experiment.
However, for the 1-pixel experiment, having a more suitable prior should
be more important.

best,
Kay




On Mon, 18 Oct 2021 12:40:45 -0700, James Holton 
mailto:jmhol...@lbl.gov>> wrote:

Thank you very much for this Kay!

So, to summarize, you are saying the answer to my question "what is the
expectation and variance if I observe a 10x10 patch of pixels with zero
counts?" is:
Iobs = 0.01
sigIobs = 0.01 (defining sigIobs = sqrt(variance(Iobs)))

And for the one-pixel case:
Iobs = 1
sigIobs = 1

but in both cases the distribution is NOT Gaussian, but rather
exponential. And that means adding variances may not be the way to
propagate error.

Is that right?

-James Holton
MAD Scientist



On 10/18/2021 7:00 AM, Kay Diederichs wrote:
Hi James,

I'm a bit behind ...

My answer about the basic question ("a patch of 100 pixels each with
zero counts - what is the variance?") you ask is the following:

1) we all know the Poisson PDF (Probability Distribution Function)
P(k|l) = l^k*e^(-l)/k!  

Re: [ccp4bb] am I doing this right?

[Randy John Read]

I would think that if this problem is being approached correctly, with the 
right prior, it shouldn't matter whether you collect the same signal 
distributed over 100 smaller pixels or the same pixel measured for the same 
length of exposure but with 100 time slices; you should get the same answer. So 
I would want to formulate the problem in a way where this invariance is 
satisfied. I thought it was, from some of the earlier descriptions of the 
problem, but this sounds worrying.

I think you're trying to say the same thing here, Kay. Is that right?

Best wishes,

Randy

On 21 Oct 2021, at 08:51, Kay Diederichs 
mailto:kay.diederi...@uni-konstanz.de>> wrote:

Hi Ian,

it is Iobs=0.01 and sigIobs=0.01 for one pixel, but adding 100 pixels each with 
variance=sigIobs^2=0.0001 gives  0.01 , yielding a 100-pixel-sigIobs of 0.1 - 
different from the 1 you get. As if repeatedly observing the same count of 0 
lowers the estimated error by sqrt(n), where n is the number of observations 
(100 in this case).

best wishes,
Kay

On Wed, 20 Oct 2021 13:08:33 +0100, Ian Tickle 
mailto:ianj...@gmail.com>> wrote:

Hi Kay

Can I just confirm that your result Iobs=0.01 sigIobs=0.01 is the estimate
of the true average intensity *per pixel* for a patch of 100 pixels?  So
then the total count for all 100 pixels is 1 with variance also 1, or in
general for k observed counts in the patch, expectation = variance = k+1
for the total count, irrespective of the number of pixels?  If so then that
agrees with my own conclusion.  It makes sense because Iobs=0.01
sigIobs=0.01 cannot come from a Poisson process (which obviously requires
expectation = variance = an integer), whereas the total count does come
from a Poisson process.

The difference from my approach is that you seem to have come at it via the
individual pixel counts whereas I came straight from the Agostini result
applied to the whole patch.  The number of pixels seems to me to be
irrelevant for the whole patch since the design of the detector, assuming
it's an ideal detector with DQE = 1 surely cannot change the photon flux
coming from the source: all ideal detectors whatever their pixel layout
must give the same result.  The number of pixels is then only relevant if
one needs to know the average intensity per pixel, i.e. the total and s.d.
divided by the number of pixels.  Note the pixels here need not even
correspond to the hardware pixels, they can be any arbitrary subdivision of
the detector surface.

Best wishes

-- Ian


On Tue, 19 Oct 2021 at 12:39, Kay Diederichs 
mailto:kay.diederi...@uni-konstanz.de>>
wrote:

James,

I am saying that my answer to "what is the expectation and variance if I
observe a 10x10 patch of pixels with zero
counts?" is Iobs=0.01 sigIobs=0.01 (and Iobs=sigIobs=1 if there is only
one pixel) IF the uniform prior applies. I agree with Gergely and others
that this prior (with its high expectation value and variance) appears
unrealistic.

In your posting of Sat, 16 Oct 2021 12:00:30 -0700 you make a calculation
of Ppix that appears like a more suitable expectation value of a prior to
me. A suitable prior might then be 1/Ppix * e^(-l/Ppix) (Agostini §7.7.1).
The Bayesian argument is IIUC that the prior plays a minor role if you do
repeated measurements of the same value, because you use the posterior of
the first measurement as the prior for the second, and so on. What this
means is that your Ppix must play the role of a scale factor if you
consider the 100-pixel experiment.
However, for the 1-pixel experiment, having a more suitable prior should
be more important.

best,
Kay




On Mon, 18 Oct 2021 12:40:45 -0700, James Holton 
mailto:jmhol...@lbl.gov>> wrote:

Thank you very much for this Kay!

So, to summarize, you are saying the answer to my question "what is the
expectation and variance if I observe a 10x10 patch of pixels with zero
counts?" is:
Iobs = 0.01
sigIobs = 0.01 (defining sigIobs = sqrt(variance(Iobs)))

And for the one-pixel case:
Iobs = 1
sigIobs = 1

but in both cases the distribution is NOT Gaussian, but rather
exponential. And that means adding variances may not be the way to
propagate error.

Is that right?

-James Holton
MAD Scientist



On 10/18/2021 7:00 AM, Kay Diederichs wrote:
Hi James,

I'm a bit behind ...

My answer about the basic question ("a patch of 100 pixels each with
zero counts - what is the variance?") you ask is the following:

1) we all know the Poisson PDF (Probability Distribution Function)
P(k|l) = l^k*e^(-l)/k!  (where k stands for for an integer >=0 and l is
lambda) which tells us the probability of observing k counts if we know l.
The PDF is normalized: SUM_over_k (P(k|l)) is 1 when k=0...infinity is 1.
2) you don't know before the experiment what l is, and you assume it is
some number x with 0<=x<=xmax (the xmax limit can be calculated by looking
at the physics of the experiment; it is finite and less than the overload
value of the pixel, otherwise you should do a different experiment). Since

Re: [ccp4bb] am I doing this right?

[Kay Diederichs]

Hi Ian,

it is Iobs=0.01 and sigIobs=0.01 for one pixel, but adding 100 pixels each with 
variance=sigIobs^2=0.0001 gives  0.01 , yielding a 100-pixel-sigIobs of 0.1 - 
different from the 1 you get. As if repeatedly observing the same count of 0 
lowers the estimated error by sqrt(n), where n is the number of observations 
(100 in this case).

best wishes,
Kay

On Wed, 20 Oct 2021 13:08:33 +0100, Ian Tickle  wrote:

>Hi Kay
>
>Can I just confirm that your result Iobs=0.01 sigIobs=0.01 is the estimate
>of the true average intensity *per pixel* for a patch of 100 pixels?  So
>then the total count for all 100 pixels is 1 with variance also 1, or in
>general for k observed counts in the patch, expectation = variance = k+1
>for the total count, irrespective of the number of pixels?  If so then that
>agrees with my own conclusion.  It makes sense because Iobs=0.01
>sigIobs=0.01 cannot come from a Poisson process (which obviously requires
>expectation = variance = an integer), whereas the total count does come
>from a Poisson process.
>
>The difference from my approach is that you seem to have come at it via the
>individual pixel counts whereas I came straight from the Agostini result
>applied to the whole patch.  The number of pixels seems to me to be
>irrelevant for the whole patch since the design of the detector, assuming
>it's an ideal detector with DQE = 1 surely cannot change the photon flux
>coming from the source: all ideal detectors whatever their pixel layout
>must give the same result.  The number of pixels is then only relevant if
>one needs to know the average intensity per pixel, i.e. the total and s.d.
>divided by the number of pixels.  Note the pixels here need not even
>correspond to the hardware pixels, they can be any arbitrary subdivision of
>the detector surface.
>
>Best wishes
>
>-- Ian
>
>
>On Tue, 19 Oct 2021 at 12:39, Kay Diederichs 
>wrote:
>
>> James,
>>
>> I am saying that my answer to "what is the expectation and variance if I
>> observe a 10x10 patch of pixels with zero
>> counts?" is Iobs=0.01 sigIobs=0.01 (and Iobs=sigIobs=1 if there is only
>> one pixel) IF the uniform prior applies. I agree with Gergely and others
>> that this prior (with its high expectation value and variance) appears
>> unrealistic.
>>
>> In your posting of Sat, 16 Oct 2021 12:00:30 -0700 you make a calculation
>> of Ppix that appears like a more suitable expectation value of a prior to
>> me. A suitable prior might then be 1/Ppix * e^(-l/Ppix) (Agostini §7.7.1).
>> The Bayesian argument is IIUC that the prior plays a minor role if you do
>> repeated measurements of the same value, because you use the posterior of
>> the first measurement as the prior for the second, and so on. What this
>> means is that your Ppix must play the role of a scale factor if you
>> consider the 100-pixel experiment.
>> However, for the 1-pixel experiment, having a more suitable prior should
>> be more important.
>>
>> best,
>> Kay
>>
>>
>>
>>
>> On Mon, 18 Oct 2021 12:40:45 -0700, James Holton  wrote:
>>
>> >Thank you very much for this Kay!
>> >
>> >So, to summarize, you are saying the answer to my question "what is the
>> >expectation and variance if I observe a 10x10 patch of pixels with zero
>> >counts?" is:
>> >Iobs = 0.01
>> >sigIobs = 0.01 (defining sigIobs = sqrt(variance(Iobs)))
>> >
>> >And for the one-pixel case:
>> >Iobs = 1
>> >sigIobs = 1
>> >
>> >but in both cases the distribution is NOT Gaussian, but rather
>> >exponential. And that means adding variances may not be the way to
>> >propagate error.
>> >
>> >Is that right?
>> >
>> >-James Holton
>> >MAD Scientist
>> >
>> >
>> >
>> >On 10/18/2021 7:00 AM, Kay Diederichs wrote:
>> >> Hi James,
>> >>
>> >> I'm a bit behind ...
>> >>
>> >> My answer about the basic question ("a patch of 100 pixels each with
>> zero counts - what is the variance?") you ask is the following:
>> >>
>> >> 1) we all know the Poisson PDF (Probability Distribution Function)
>> P(k|l) = l^k*e^(-l)/k!  (where k stands for for an integer >=0 and l is
>> lambda) which tells us the probability of observing k counts if we know l.
>> The PDF is normalized: SUM_over_k (P(k|l)) is 1 when k=0...infinity is 1.
>> >> 2) you don't know before the experiment what l is, and you assume it is
>> some number x with 0<=x<=xmax (the xmax limit can be calculated by looking
>> at the physics of the experiment; it is finite and less than the overload
>> value of the pixel, otherwise you should do a different experiment). Since
>> you don't know that number, all the x values are equally likely - you use a
>> uniform prior.
>> >> 3) what is the PDF P(l|k) of l if we observe k counts?  That can be
>> found with Bayes theorem, and it turns out that (due to the uniform prior)
>> the right hand side of the formula looks the same as in 1) : P(l|k) =
>> l^k*e^(-l)/k! (again, the ! stands for the factorial, it is not a semantic
>> exclamation mark). This is eqs. 7.42 and 7.43 in Agostini "Bayesian
>> Reasoning in Data 

Re: [ccp4bb] am I doing this right?

[Ian Tickle]

Hi Kay

Can I just confirm that your result Iobs=0.01 sigIobs=0.01 is the estimate
of the true average intensity *per pixel* for a patch of 100 pixels?  So
then the total count for all 100 pixels is 1 with variance also 1, or in
general for k observed counts in the patch, expectation = variance = k+1
for the total count, irrespective of the number of pixels?  If so then that
agrees with my own conclusion.  It makes sense because Iobs=0.01
sigIobs=0.01 cannot come from a Poisson process (which obviously requires
expectation = variance = an integer), whereas the total count does come
from a Poisson process.

The difference from my approach is that you seem to have come at it via the
individual pixel counts whereas I came straight from the Agostini result
applied to the whole patch.  The number of pixels seems to me to be
irrelevant for the whole patch since the design of the detector, assuming
it's an ideal detector with DQE = 1 surely cannot change the photon flux
coming from the source: all ideal detectors whatever their pixel layout
must give the same result.  The number of pixels is then only relevant if
one needs to know the average intensity per pixel, i.e. the total and s.d.
divided by the number of pixels.  Note the pixels here need not even
correspond to the hardware pixels, they can be any arbitrary subdivision of
the detector surface.

Best wishes

-- Ian


On Tue, 19 Oct 2021 at 12:39, Kay Diederichs 
wrote:

> James,
>
> I am saying that my answer to "what is the expectation and variance if I
> observe a 10x10 patch of pixels with zero
> counts?" is Iobs=0.01 sigIobs=0.01 (and Iobs=sigIobs=1 if there is only
> one pixel) IF the uniform prior applies. I agree with Gergely and others
> that this prior (with its high expectation value and variance) appears
> unrealistic.
>
> In your posting of Sat, 16 Oct 2021 12:00:30 -0700 you make a calculation
> of Ppix that appears like a more suitable expectation value of a prior to
> me. A suitable prior might then be 1/Ppix * e^(-l/Ppix) (Agostini §7.7.1).
> The Bayesian argument is IIUC that the prior plays a minor role if you do
> repeated measurements of the same value, because you use the posterior of
> the first measurement as the prior for the second, and so on. What this
> means is that your Ppix must play the role of a scale factor if you
> consider the 100-pixel experiment.
> However, for the 1-pixel experiment, having a more suitable prior should
> be more important.
>
> best,
> Kay
>
>
>
>
> On Mon, 18 Oct 2021 12:40:45 -0700, James Holton  wrote:
>
> >Thank you very much for this Kay!
> >
> >So, to summarize, you are saying the answer to my question "what is the
> >expectation and variance if I observe a 10x10 patch of pixels with zero
> >counts?" is:
> >Iobs = 0.01
> >sigIobs = 0.01 (defining sigIobs = sqrt(variance(Iobs)))
> >
> >And for the one-pixel case:
> >Iobs = 1
> >sigIobs = 1
> >
> >but in both cases the distribution is NOT Gaussian, but rather
> >exponential. And that means adding variances may not be the way to
> >propagate error.
> >
> >Is that right?
> >
> >-James Holton
> >MAD Scientist
> >
> >
> >
> >On 10/18/2021 7:00 AM, Kay Diederichs wrote:
> >> Hi James,
> >>
> >> I'm a bit behind ...
> >>
> >> My answer about the basic question ("a patch of 100 pixels each with
> zero counts - what is the variance?") you ask is the following:
> >>
> >> 1) we all know the Poisson PDF (Probability Distribution Function)
> P(k|l) = l^k*e^(-l)/k!  (where k stands for for an integer >=0 and l is
> lambda) which tells us the probability of observing k counts if we know l.
> The PDF is normalized: SUM_over_k (P(k|l)) is 1 when k=0...infinity is 1.
> >> 2) you don't know before the experiment what l is, and you assume it is
> some number x with 0<=x<=xmax (the xmax limit can be calculated by looking
> at the physics of the experiment; it is finite and less than the overload
> value of the pixel, otherwise you should do a different experiment). Since
> you don't know that number, all the x values are equally likely - you use a
> uniform prior.
> >> 3) what is the PDF P(l|k) of l if we observe k counts?  That can be
> found with Bayes theorem, and it turns out that (due to the uniform prior)
> the right hand side of the formula looks the same as in 1) : P(l|k) =
> l^k*e^(-l)/k! (again, the ! stands for the factorial, it is not a semantic
> exclamation mark). This is eqs. 7.42 and 7.43 in Agostini "Bayesian
> Reasoning in Data Analysis".
> >> 3a) side note: if we calculate the expectation value for l, by
> multiplying with l and integrating over l from 0 to infinity, we obtain
> E(P(l|k))=k+1, and similarly for the variance (Agostini eqs 7.45 and 7.46)
> >> 4) for k=0 (zero counts observed in a single pixel), this reduces to
> P(l|0)=e^(-l) for a single observation (pixel). (this is basic math; see
> also §7.4.1 of Agostini.
> >> 5) since we have 100 independent pixels, we must multiply the
> individual PDFs to get the overall PDF f, and also 

Re: [ccp4bb] am I doing this right?

[James Holton]

Thank you Gergely,

Oh, don't worry, I am not concerned about belief. Neither the model nor 
the data care what I believe.


What I am really asking is: what is the proper way to combine weak 
observations?


Right now, in pretty much all structural sciences we are not used to 
doing this, but we are entering an era where we will have to.


I was trying to ask a simple question with the 10x10 pixel patch because 
(as Graeme, Ian and others pointed out) it highlights how the solution 
must also apply to two patches of 50 pixels.  In reality, unfortunately, 
those two patches might not be next to each other and will have 
different Lorentz factors, polarizaiton factors, absorption factors, and 
probably different partiality as well. These values are knowable, but 
they are not integers. The way we currently deal with all this is to 
first convert patches of pixels into an expectation and variance, then 
apply all the corrections, and finally "merge" everything with error 
propagation into simple list of h,k,l,Iobs,sigIobs that we can compare 
to a PDB file.


You are absolutely right that the best thing to do would be fitting a 
model of the whole diffractometer and crystal, structure factors 
included, directly and pixel-by-pixel to the image data.  Some 
colleagues and I managed to do this recently 
(https://doi.org/10.1107/s2052252520013007). It is rather 
computationally expensive, but seems to be working.


I hope this will be a useful tool, but I don't think such an approach 
will ever completely supplant data reduction, as there are many 
advantages to the latter.  But only if you do the statistics right!  
This is why I asked the community so that folks cleverer and more 
experienced than I in such matters (such as yourself) can correct me if 
I'm getting something wrong.  And the community benefits from the 
discussion.


Thank you for your thoughtful and thought-provoking insights!

-James Holton
MAD Scientist


On 10/19/2021 2:05 AM, Gergely Katona wrote:

Dear James,

I am sorry to nitpick, but this is the answer to "what is my belief of expectation 
and variance if I observe a 10x10 patch of pixels with zero counts?" This will 
heavily depend on my model.
When I make predictions like this, my intention is not to replace the data with a 
"new and improved" data that is closer to the Truth and deposit in some 
database from the position of authority.

I would simply use it to validate my model. Well, my model expects the Iobs to 
be 0.01, but in fact it is 0. This may make me slightly worried, but then I 
look at the posterior distribution and I see 0 with highest posterior 
probability so I relax a bit that I do not have to throw out my model outright. 
Still, a better model may be out there.
For a Bayesian the data is fixed and holy, the model may change. And the question rarely manifests 
like that one does not have to spend a lot of time pondering about if a uniform distribution of the 
rate is compatible with my belief in some quantum process. Bayesian folks are pragmatic. Your 
question about "what is my belief about the slope and intercept of a line that is the basis of 
some time-dependent random process given my observations" is more relevant. It is 
straightforward to implement as a Bayesian network to answer this question and it will give you 
predictions that looks deceptively like the data. Here, you only care about your prior belief about 
the magnitude of slope and intercept, the belief about what the rate may be independent of time is 
quite irrelevant and so are the predictions they may make. And I guess you would not intend to 
deposit images that were generated by the predictions of these posterior models and the "new 
and improved data".

Best wishes,

Gergely


Gergely Katona, Professor, Chairman of the Chemistry Program Council
Department of Chemistry and Molecular Biology, University of Gothenburg
Box 462, 40530 Göteborg, Sweden
Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
Web: http://katonalab.eu, Email: gergely.kat...@gu.se

-Original Message-
From: CCP4 bulletin board  On Behalf Of James Holton
Sent: 18 October, 2021 21:41
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Thank you very much for this Kay!

So, to summarize, you are saying the answer to my question "what is the expectation 
and variance if I observe a 10x10 patch of pixels with zero counts?" is:
Iobs = 0.01
sigIobs = 0.01 (defining sigIobs = sqrt(variance(Iobs)))

And for the one-pixel case:
Iobs = 1
sigIobs = 1

but in both cases the distribution is NOT Gaussian, but rather exponential. And 
that means adding variances may not be the way to propagate error.

Is that right?

-James Holton
MAD Scientist



On 10/18/2021 7:00 AM, Kay Diederichs wrote:

Hi James,

I'm a bit behind ...

My answer about the basic question ("a patch of 100 pixels each with zero counts - 
what is the vari

Re: [ccp4bb] am I doing this right?

[Kay Diederichs]

James, 

I am saying that my answer to "what is the expectation and variance if I 
observe a 10x10 patch of pixels with zero
counts?" is Iobs=0.01 sigIobs=0.01 (and Iobs=sigIobs=1 if there is only one 
pixel) IF the uniform prior applies. I agree with Gergely and others that this 
prior (with its high expectation value and variance) appears unrealistic.

In your posting of Sat, 16 Oct 2021 12:00:30 -0700 you make a calculation of 
Ppix that appears like a more suitable expectation value of a prior to me. A 
suitable prior might then be 1/Ppix * e^(-l/Ppix) (Agostini §7.7.1). The 
Bayesian argument is IIUC that the prior plays a minor role if you do repeated 
measurements of the same value, because you use the posterior of the first 
measurement as the prior for the second, and so on. What this means is that 
your Ppix must play the role of a scale factor if you consider the 100-pixel 
experiment.
However, for the 1-pixel experiment, having a more suitable prior should be 
more important.

best,
Kay




On Mon, 18 Oct 2021 12:40:45 -0700, James Holton  wrote:

>Thank you very much for this Kay!
>
>So, to summarize, you are saying the answer to my question "what is the
>expectation and variance if I observe a 10x10 patch of pixels with zero
>counts?" is:
>Iobs = 0.01
>sigIobs = 0.01 (defining sigIobs = sqrt(variance(Iobs)))
>
>And for the one-pixel case:
>Iobs = 1
>sigIobs = 1
>
>but in both cases the distribution is NOT Gaussian, but rather
>exponential. And that means adding variances may not be the way to
>propagate error.
>
>Is that right?
>
>-James Holton
>MAD Scientist
>
>
>
>On 10/18/2021 7:00 AM, Kay Diederichs wrote:
>> Hi James,
>>
>> I'm a bit behind ...
>>
>> My answer about the basic question ("a patch of 100 pixels each with zero 
>> counts - what is the variance?") you ask is the following:
>>
>> 1) we all know the Poisson PDF (Probability Distribution Function)  P(k|l) = 
>> l^k*e^(-l)/k!  (where k stands for for an integer >=0 and l is lambda) which 
>> tells us the probability of observing k counts if we know l. The PDF is 
>> normalized: SUM_over_k (P(k|l)) is 1 when k=0...infinity is 1.
>> 2) you don't know before the experiment what l is, and you assume it is some 
>> number x with 0<=x<=xmax (the xmax limit can be calculated by looking at the 
>> physics of the experiment; it is finite and less than the overload value of 
>> the pixel, otherwise you should do a different experiment). Since you don't 
>> know that number, all the x values are equally likely - you use a uniform 
>> prior.
>> 3) what is the PDF P(l|k) of l if we observe k counts?  That can be found 
>> with Bayes theorem, and it turns out that (due to the uniform prior) the 
>> right hand side of the formula looks the same as in 1) : P(l|k) = 
>> l^k*e^(-l)/k! (again, the ! stands for the factorial, it is not a semantic 
>> exclamation mark). This is eqs. 7.42 and 7.43 in Agostini "Bayesian 
>> Reasoning in Data Analysis".
>> 3a) side note: if we calculate the expectation value for l, by multiplying 
>> with l and integrating over l from 0 to infinity, we obtain E(P(l|k))=k+1, 
>> and similarly for the variance (Agostini eqs 7.45 and 7.46)
>> 4) for k=0 (zero counts observed in a single pixel), this reduces to 
>> P(l|0)=e^(-l) for a single observation (pixel). (this is basic math; see 
>> also §7.4.1 of Agostini.
>> 5) since we have 100 independent pixels, we must multiply the individual 
>> PDFs to get the overall PDF f, and also normalize to make the integral over 
>> that PDF to be 1: the result is f(l|all 100 pixels are 0)=n*e^(-n*l). (basic 
>> math). A more Bayesian procedure would be to realize that the posterior PDF 
>> P(l|0)=e^(-l) of the first pixel should be used as the prior for the second 
>> pixel, and so forth until the 100th pixel. This has the same result f(l|all 
>> 100 pixels are 0)=n*e^(-n*l) (Agostini § 7.7.2)!
>> 6) the expectation value INTEGRAL_0_to_infinity over l*n*e^(-n*l) dl is 1/n 
>> .  This is 1 if n=1 as we know from 3a), and 1/100 for 100 pixels with 0 
>> counts.
>> 7) the variance is then INTEGRAL_0_to_infinity over (l-1/n)^2*n*e^(-n*l) dl 
>> . This is 1/n^2
>>
>> I find these results quite satisfactory. Please note that they deviate from 
>> the MLE result: expectation value=0, variance=0 . The problem appears to be 
>> that a Maximum Likelihood Estimator may give wrong results for small n; 
>> something that I've read a couple of times but which appears not to be 
>> universally known/taught. Clearly, the result in 6) and 7) for large n 
>> converges towards 0, as it should be.
>> What this also means is that one should really work out the PDF instead of 
>> just adding expectation values and variances (and arriving at 100 if all 100 
>> pixels have zero counts) because it is contradictory to use a uniform prior 
>> for all the pixels if OTOH these agree perfectly in being 0!
>>
>> What this means for zero-dose extrapolation I have not thought about. At 
>> least it prevents 

Re: [ccp4bb] am I doing this right?

[Gergely Katona]

Dear James,

I am sorry to nitpick, but this is the answer to "what is my belief of 
expectation and variance if I observe a 10x10 patch of pixels with zero 
counts?" This will heavily depend on my model.
When I make predictions like this, my intention is not to replace the data with 
a "new and improved" data that is closer to the Truth and deposit in some 
database from the position of authority. 

I would simply use it to validate my model. Well, my model expects the Iobs to 
be 0.01, but in fact it is 0. This may make me slightly worried, but then I 
look at the posterior distribution and I see 0 with highest posterior 
probability so I relax a bit that I do not have to throw out my model outright. 
Still, a better model may be out there.
For a Bayesian the data is fixed and holy, the model may change. And the 
question rarely manifests like that one does not have to spend a lot of time 
pondering about if a uniform distribution of the rate is compatible with my 
belief in some quantum process. Bayesian folks are pragmatic. Your question 
about "what is my belief about the slope and intercept of a line that is the 
basis of some time-dependent random process given my observations" is more 
relevant. It is straightforward to implement as a Bayesian network to answer 
this question and it will give you predictions that looks deceptively like the 
data. Here, you only care about your prior belief about the magnitude of slope 
and intercept, the belief about what the rate may be independent of time is 
quite irrelevant and so are the predictions they may make. And I guess you 
would not intend to deposit images that were generated by the predictions of 
these posterior models and the "new and improved data".

Best wishes,

Gergely


Gergely Katona, Professor, Chairman of the Chemistry Program Council
Department of Chemistry and Molecular Biology, University of Gothenburg
Box 462, 40530 Göteborg, Sweden
Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
Web: http://katonalab.eu, Email: gergely.kat...@gu.se

-Original Message-
From: CCP4 bulletin board  On Behalf Of James Holton
Sent: 18 October, 2021 21:41
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Thank you very much for this Kay!

So, to summarize, you are saying the answer to my question "what is the 
expectation and variance if I observe a 10x10 patch of pixels with zero 
counts?" is:
Iobs = 0.01
sigIobs = 0.01 (defining sigIobs = sqrt(variance(Iobs)))

And for the one-pixel case:
Iobs = 1
sigIobs = 1

but in both cases the distribution is NOT Gaussian, but rather exponential. And 
that means adding variances may not be the way to propagate error.

Is that right?

-James Holton
MAD Scientist



On 10/18/2021 7:00 AM, Kay Diederichs wrote:
> Hi James,
>
> I'm a bit behind ...
>
> My answer about the basic question ("a patch of 100 pixels each with zero 
> counts - what is the variance?") you ask is the following:
>
> 1) we all know the Poisson PDF (Probability Distribution Function)  P(k|l) = 
> l^k*e^(-l)/k!  (where k stands for for an integer >=0 and l is lambda) which 
> tells us the probability of observing k counts if we know l. The PDF is 
> normalized: SUM_over_k (P(k|l)) is 1 when k=0...infinity is 1.
> 2) you don't know before the experiment what l is, and you assume it is some 
> number x with 0<=x<=xmax (the xmax limit can be calculated by looking at the 
> physics of the experiment; it is finite and less than the overload value of 
> the pixel, otherwise you should do a different experiment). Since you don't 
> know that number, all the x values are equally likely - you use a uniform 
> prior.
> 3) what is the PDF P(l|k) of l if we observe k counts?  That can be found 
> with Bayes theorem, and it turns out that (due to the uniform prior) the 
> right hand side of the formula looks the same as in 1) : P(l|k) = 
> l^k*e^(-l)/k! (again, the ! stands for the factorial, it is not a semantic 
> exclamation mark). This is eqs. 7.42 and 7.43 in Agostini "Bayesian Reasoning 
> in Data Analysis".
> 3a) side note: if we calculate the expectation value for l, by 
> multiplying with l and integrating over l from 0 to infinity, we 
> obtain E(P(l|k))=k+1, and similarly for the variance (Agostini eqs 
> 7.45 and 7.46)
> 4) for k=0 (zero counts observed in a single pixel), this reduces to 
> P(l|0)=e^(-l) for a single observation (pixel). (this is basic math; see also 
> §7.4.1 of Agostini.
> 5) since we have 100 independent pixels, we must multiply the individual PDFs 
> to get the overall PDF f, and also normalize to make the integral over that 
> PDF to be 1: the result is f(l|all 100 pixels are 0)=n*e^(-n*l). (basic 
> math). A more Bayesian procedure would be to realize that the posterior PDF 
> P(l|0)=e^(-l) of the

Re: [ccp4bb] am I doing this right?

[Rasmus Fogh]

do some kind of 
smoothing thing, but then you are losing information again. Perhaps 
also making ill-founded assumptions. You need error bars of some 
kind, and, better yet, the shape of the distribution implied by those 
error bars.


And all this makes me think somebody must have already done this. I'm 
willing to bet probably some time in the late 1700s to early 1800s. 
All we're really talking about here is augmenting maximum-likelihood 
estimation of an average value to maximum-likelihood estimation of a 
straight line. That is, slope and intercept, with sigmas on both. I 
suspect the proper approach is to first bring everything down to the 
exact information content of a single photon (or lack of a photon), 
and build up from there.  If you are lucky enough to have a large 
number of photons then linear regression will work, and you are back 
to Diederichs (2003). But when you're photon-starved the statistics 
of single photons become more and more important.  This led me to: is 
it k? or k+1 ?  When k=0 getting this wrong could introduce a factor 
of infinity.


So, perhaps the big "consequence of getting it wrong" is embarrassing 
myself by re-making a 200-year old mistake I am not currently aware 
of. I am confident a solution exists, but only recently started 
working on this.  So, I figured ... ask the world?


-James Holton
MAD Scientist


On 10/17/2021 1:51 AM, Frank Von Delft wrote:
James, I've been watching the thread with fascination, but also the 
confusion of wild ignorance. I've finally realised why.


What I've missed is: what exactly makes the question so important?  
I've understood what brought it up, if course, but not the 
consequence of getting it wrong.


Frank

Sent from tiny silly touch screen

*From:* James Holton 
*Sent:* Saturday, 16 October 2021 20:01
*To:* CCP4BB@JISCMAIL.AC.UK
*Subject:* Re: [ccp4bb] am I doing this right?

Thank you everyone for your thoughtful and thought-provoking responses!

But, I am starting to think I was not as clear as I could have been
about my question.  I am actually concerning myself with background, 
not

necessarily Bragg peaks.  With Bragg photons you want the sum, but for
background you want the average.

What I'm getting at is: how does one properly weight a zero-photon
observation when it comes time to combine it with others?  Hopefully
they are not all zero.  If they are, check your shutter.

So, ignoring Bragg photons for the moment (let us suppose it is a
systematic absence) what I am asking is: what is the variance, or,
better yet,what is the WEIGHT one should assign to the observation of
zero photons in a patch of 10x10 pixels?

In the absence of any prior knowledge this is a difficult question, but
a question we kind of need to answer if we want to properly measure 
data

from weak images.  So, what do we do?

Well, with the "I have no idea" uniform prior, it would seem that
expectation (Epix) and variance (Vpix) would be k+1 = 1 for each pixel,
and therefore the sum of Epix and Vpix over the 100 independent 
pixels is:


Epatch=Vpatch=100 photons

I know that seems weird to assume 100 photons should have hit when we
actually saw none, but consider what that zero-photon count, all by
itself, is really telling you:
a) Epix > 20 ? No way. That is "right out". Given we know its Poisson
distributed, and that background is flat, it is VERY unlikely you 
have E

that big when you saw zero. Cross all those E values off your list.
b) Epix=0 ? Well, that CAN be true, but other things are possible and
all of them are E>0. So, most likely E is not 0, but at least a little
bit higher.
c) Epix=1e-6 ?  Yeah, sure, why not?
d) Epix= -1e-6 ?  No. Don't be silly.
e) If I had to guess? Meh. 1 photon per pixel?  That would be k+1

I suppose my objection to E=V=0 is because V=0 implies infinite
confidence in the value of E, and that we don't have. Yes, it is true
that we are quite confident in the fact that we did not see any photons
this time, but the remember that E and V are the mean and variance that
you would see if you did a million experiments under the same
conditions. We are trying to guess those from what we've got. Just
because you've seen zero a hundred times doesn't mean the 101st
experiment won't give you a count.  If it does, then maybe Epatch=0.01
and Epix=0.0001?  But what do you do before you see your first photon?
All you can really do is bracket it.

But what if you come up with a better prior than "I have no idea" ?
Well, we do have other pixels on the detector, and presuming the
background is flat, or at least smooth, maybe the average counts/pixel
is a better prior?

So, let us consider an ideal detector with 1e6 independent pixels. Let
us further say that 1e5 background photons have hit that detector.  I
want to still ignore Bragg photons because those have a very different
prior distribution to the ba

Re: [ccp4bb] am I doing this right?

[Guillaume Gaullier]

I am not qualified to comment on anything in the rest of this discussion, but 
regarding the excerpt quoted below: this way of recording an "image" seems very 
similar to the EER (electron event representation) of the Falcon 4 direct 
electron detector used in cryoEM. See https://doi.org/10.1107/S205225252000929X

Guillaume


On 18 Oct 2021, at 08:30, Frank von Delft 
mailto:frank.vonde...@cmd.ox.ac.uk>> wrote:

Also:  should the detectors change how they read out things, then?  Just write 
out the events with timestamp, rather than dumping all pixels all the time into 
these arbitrary containers called "image".  Or is that what's already happening 
in HDF5 (which I don't understand one bit, I should add).









N?r du har kontakt med oss p? Uppsala universitet med e-post s? inneb?r det att 
vi behandlar dina personuppgifter. F?r att l?sa mer om hur vi g?r det kan du 
l?sa h?r: http://www.uu.se/om-uu/dataskydd-personuppgifter/

E-mailing Uppsala University means that we will process your personal data. For 
more information on how this is performed, please read here: 
http://www.uu.se/en/about-uu/data-protection-policy



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] am I doing this right?

[James Holton]

 to the 
exact information content of a single photon (or lack of a photon), 
and build up from there.  If you are lucky enough to have a large 
number of photons then linear regression will work, and you are back 
to Diederichs (2003). But when you're photon-starved the statistics 
of single photons become more and more important.  This led me to: is 
it k? or k+1 ?  When k=0 getting this wrong could introduce a factor 
of infinity.


So, perhaps the big "consequence of getting it wrong" is embarrassing 
myself by re-making a 200-year old mistake I am not currently aware 
of. I am confident a solution exists, but only recently started 
working on this.  So, I figured ... ask the world?


-James Holton
MAD Scientist


On 10/17/2021 1:51 AM, Frank Von Delft wrote:
James, I've been watching the thread with fascination, but also the 
confusion of wild ignorance. I've finally realised why.


What I've missed is: what exactly makes the question so important?  
I've understood what brought it up, if course, but not the 
consequence of getting it wrong.


Frank

Sent from tiny silly touch screen

*From:* James Holton 
*Sent:* Saturday, 16 October 2021 20:01
*To:* CCP4BB@JISCMAIL.AC.UK
*Subject:* Re: [ccp4bb] am I doing this right?

Thank you everyone for your thoughtful and thought-provoking responses!

But, I am starting to think I was not as clear as I could have been
about my question.  I am actually concerning myself with background, 
not

necessarily Bragg peaks.  With Bragg photons you want the sum, but for
background you want the average.

What I'm getting at is: how does one properly weight a zero-photon
observation when it comes time to combine it with others?  Hopefully
they are not all zero.  If they are, check your shutter.

So, ignoring Bragg photons for the moment (let us suppose it is a
systematic absence) what I am asking is: what is the variance, or,
better yet,what is the WEIGHT one should assign to the observation of
zero photons in a patch of 10x10 pixels?

In the absence of any prior knowledge this is a difficult question, but
a question we kind of need to answer if we want to properly measure 
data

from weak images.  So, what do we do?

Well, with the "I have no idea" uniform prior, it would seem that
expectation (Epix) and variance (Vpix) would be k+1 = 1 for each pixel,
and therefore the sum of Epix and Vpix over the 100 independent 
pixels is:


Epatch=Vpatch=100 photons

I know that seems weird to assume 100 photons should have hit when we
actually saw none, but consider what that zero-photon count, all by
itself, is really telling you:
a) Epix > 20 ? No way. That is "right out". Given we know its Poisson
distributed, and that background is flat, it is VERY unlikely you 
have E

that big when you saw zero. Cross all those E values off your list.
b) Epix=0 ? Well, that CAN be true, but other things are possible and
all of them are E>0. So, most likely E is not 0, but at least a little
bit higher.
c) Epix=1e-6 ?  Yeah, sure, why not?
d) Epix= -1e-6 ?  No. Don't be silly.
e) If I had to guess? Meh. 1 photon per pixel?  That would be k+1

I suppose my objection to E=V=0 is because V=0 implies infinite
confidence in the value of E, and that we don't have. Yes, it is true
that we are quite confident in the fact that we did not see any photons
this time, but the remember that E and V are the mean and variance that
you would see if you did a million experiments under the same
conditions. We are trying to guess those from what we've got. Just
because you've seen zero a hundred times doesn't mean the 101st
experiment won't give you a count.  If it does, then maybe Epatch=0.01
and Epix=0.0001?  But what do you do before you see your first photon?
All you can really do is bracket it.

But what if you come up with a better prior than "I have no idea" ?
Well, we do have other pixels on the detector, and presuming the
background is flat, or at least smooth, maybe the average counts/pixel
is a better prior?

So, let us consider an ideal detector with 1e6 independent pixels. Let
us further say that 1e5 background photons have hit that detector.  I
want to still ignore Bragg photons because those have a very different
prior distribution to the background.  Let us say we have masked off 
all

the Bragg areas.

The average overall background is then 0.1 photons/pixel. Let us assign
that to the prior probability Ppix = 0.1.  Now let us look again at 
that

patch of 10x10 pixels with zero counts on it.  We expected to see 10,
but got 0.  What are the odds of that?  Pretty remote. Less than 1 in a
million.

I suspect in this situation where such an unlikely event has 
occurred it

should perhaps be given a variance larger than 100. Perhaps quite a bit
larger?  Subsequent "sigma-weighted" summation would then squash its
contribution down to effectively 0. So, relative to any other
obs

Re: [ccp4bb] am I doing this right?

[James Holton]

Thank you very much for this Kay!

So, to summarize, you are saying the answer to my question "what is the 
expectation and variance if I observe a 10x10 patch of pixels with zero 
counts?" is:

Iobs = 0.01
sigIobs = 0.01 (defining sigIobs = sqrt(variance(Iobs)))

And for the one-pixel case:
Iobs = 1
sigIobs = 1

but in both cases the distribution is NOT Gaussian, but rather 
exponential. And that means adding variances may not be the way to 
propagate error.


Is that right?

-James Holton
MAD Scientist



On 10/18/2021 7:00 AM, Kay Diederichs wrote:

Hi James,

I'm a bit behind ...

My answer about the basic question ("a patch of 100 pixels each with zero counts - 
what is the variance?") you ask is the following:

1) we all know the Poisson PDF (Probability Distribution Function)  P(k|l) = 
l^k*e^(-l)/k!  (where k stands for for an integer >=0 and l is lambda) which 
tells us the probability of observing k counts if we know l. The PDF is 
normalized: SUM_over_k (P(k|l)) is 1 when k=0...infinity is 1.
2) you don't know before the experiment what l is, and you assume it is some number x 
with 0<=x<=xmax (the xmax limit can be calculated by looking at the physics of 
the experiment; it is finite and less than the overload value of the pixel, otherwise 
you should do a different experiment). Since you don't know that number, all the x 
values are equally likely - you use a uniform prior.
3) what is the PDF P(l|k) of l if we observe k counts?  That can be found with Bayes 
theorem, and it turns out that (due to the uniform prior) the right hand side of the 
formula looks the same as in 1) : P(l|k) = l^k*e^(-l)/k! (again, the ! stands for the 
factorial, it is not a semantic exclamation mark). This is eqs. 7.42 and 7.43 in Agostini 
"Bayesian Reasoning in Data Analysis".
3a) side note: if we calculate the expectation value for l, by multiplying with 
l and integrating over l from 0 to infinity, we obtain E(P(l|k))=k+1, and 
similarly for the variance (Agostini eqs 7.45 and 7.46)
4) for k=0 (zero counts observed in a single pixel), this reduces to 
P(l|0)=e^(-l) for a single observation (pixel). (this is basic math; see also 
§7.4.1 of Agostini.
5) since we have 100 independent pixels, we must multiply the individual PDFs 
to get the overall PDF f, and also normalize to make the integral over that PDF 
to be 1: the result is f(l|all 100 pixels are 0)=n*e^(-n*l). (basic math). A 
more Bayesian procedure would be to realize that the posterior PDF 
P(l|0)=e^(-l) of the first pixel should be used as the prior for the second 
pixel, and so forth until the 100th pixel. This has the same result f(l|all 100 
pixels are 0)=n*e^(-n*l) (Agostini § 7.7.2)!
6) the expectation value INTEGRAL_0_to_infinity over l*n*e^(-n*l) dl is 1/n .  
This is 1 if n=1 as we know from 3a), and 1/100 for 100 pixels with 0 counts.
7) the variance is then INTEGRAL_0_to_infinity over (l-1/n)^2*n*e^(-n*l) dl . 
This is 1/n^2

I find these results quite satisfactory. Please note that they deviate from the 
MLE result: expectation value=0, variance=0 . The problem appears to be that a 
Maximum Likelihood Estimator may give wrong results for small n; something that 
I've read a couple of times but which appears not to be universally 
known/taught. Clearly, the result in 6) and 7) for large n converges towards 0, 
as it should be.
What this also means is that one should really work out the PDF instead of just 
adding expectation values and variances (and arriving at 100 if all 100 pixels 
have zero counts) because it is contradictory to use a uniform prior for all 
the pixels if OTOH these agree perfectly in being 0!

What this means for zero-dose extrapolation I have not thought about. At least 
it prevents infinite weights!

Best,
Kay








To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] am I doing this right?

[Ian Tickle]

All, this was my reply to one of James' emails which I just noticed was to
me only, and I ought to have CC'd it to the BB, since it has relevance to
others' contributions to the discussion.

Cheers

-- Ian


On Mon, 18 Oct 2021 at 11:27, Ian Tickle  wrote:

>
> James, no I don't think so, what does it have to do with the number of
> pixels?  All that matters is the photon flux and the area of measurement.
> The only purpose of the detector is to make the measurement: it cannot
> possibly change the measurement unless of course it's less than ideal.
> Assuming we are using ideal quantum detectors with DQE = 1 you can take
> away the detector and replace it with a different ideal detector with
> different-sized pixels.  The result must be the same for any ideal detector
> assuming a fixed photon flux and measurement box.  Remember that for the
> Poisson distribution (and the true count is Poisson-distributed), the
> expectation equals the variance.  If you're saying that the variance is 100
> then so is the expectation: does that sound sensible given that no counts
> were observed?
>
> I would take heed of Gergely's wise counsel: "It is easy to fall into the
> trap of frequentist thinking and reduce data one step at a time.".  Your
> argument is step 1: estimate expectation and variance for each pixel; step
> 2: add up the expectations and variances for all the pixels.  It doesn't
> work like that!
>
> I stick with my original suggestion that for no observed counts in some
> area the best estimate of expectation and variance is 1 in that area (I'm
> probably assuming an uninformative prior but as I said I haven't been
> through the algebra in detail).
>
> Cheers
>
> -- Ian
>
>
> On Sat, 16 Oct 2021 at 16:04, James Holton  wrote:
>
>> Sorry to be unclear.  I am actually referring to the background.  Assume
>> the spot is a systematic absence, but we still need a variance.  The
>> "units" (if you will pemit) will be photons/pixel.  Not photons/spot.
>>
>> I think in that case my 10x10 patch of independent pixels, all with zero
>> observed counts, would have a variance of 100.  The sum of two patches of
>> 50 would also have a variance of 100.
>>
>> Right?
>>
>>
>> On 10/16/2021 5:35 AM, Ian Tickle wrote:
>>
>>
>> PS Note also that the prior is for the integrated spot intensity, not the
>> individual pixel counts, so we should integrate before applying the +1
>> correction.
>>
>> I.
>>
>>
>> On Sat, 16 Oct 2021 at 10:16, Ian Tickle  wrote:
>>
>>>
>>> James, you're now talking about additivity of the observed or true
>>> counts for different spots whereas I assumed your question still concerned
>>> estimation of the expectation and variance of the true count of a given
>>> spot, assuming some prior distribution of the true count.  As we've already
>>> seen, the latter does not behave in an intuitive way.
>>>
>>> We can use argument by reductio ad absurdum (reduction to absurdity) to
>>> demonstrate this, i.e. assume the contrary (i.e. that the expected counts
>>> and variances are additive), and show that it leads inevitably to a logical
>>> contradiction.  First it should be clear that the result must be
>>> independent of the pixellation of the detector surface, i.e. the pixel size
>>> (it needn't correspond to the hardware pixel detectors), provided it's not
>>> smaller than the hardware pixel and not greater than the spot size.
>>>
>>> That means that we can subdivide the 10x10 area any way we like and we
>>> should always get the same answer for the total expected value and
>>> variance, so 100 1x1 pixels, or 50 2x1, or 25 2x2, or 4 5x5, or 1 10x10
>>> etc.  Given that we accept the estimate of 1 for the expectation and
>>> variance of the true count for zero observed count and assume additivity of
>>> the expected values and variances, these choices give 100, 50, 25, 4 and 1
>>> as the answer!  We can accept possible alternative solutions to a problem
>>> where each solution is correct in its own universe, but not different
>>> solutions that are simultaneously correct in the same universe: that's a
>>> logical contradiction.  So we are forced to abandon additivity for the
>>> expected value and variance of the true count for a single spot.  That of
>>> course has nothing to do with additivity of those values for multiple
>>> different spots: that is still valid.
>>>
>>> BTW I found this paper on Bayesian priors for the Poisson distribution:
>>> "Inferring the intensity of Poisson processes at the limit of the detector
>>> sensitivity": https://arxiv.org/pdf/hep-ex/9909047.pdf .
>>>
>>> Cheers
>>>
>>> -- Ian
>>>
>>>
>>> On Sat, 16 Oct 2021 at 00:25, James Holton  wrote:
>>>
 I don't follow.  How is it that variances are not additive?  We do this
 all the time when we merge data together.


 On 10/15/2021 4:22 PM, Ian Tickle wrote:

 James, also as the question is posed none of the answers is correct
 because a photon count must be an integer (there's no such thing as a
 

Re: [ccp4bb] am I doing this right?

[Kay Diederichs]

Hi James,

I'm a bit behind ...

My answer about the basic question ("a patch of 100 pixels each with zero 
counts - what is the variance?") you ask is the following:

1) we all know the Poisson PDF (Probability Distribution Function)  P(k|l) = 
l^k*e^(-l)/k!  (where k stands for for an integer >=0 and l is lambda) which 
tells us the probability of observing k counts if we know l. The PDF is 
normalized: SUM_over_k (P(k|l)) is 1 when k=0...infinity is 1.
2) you don't know before the experiment what l is, and you assume it is some 
number x with 0<=x<=xmax (the xmax limit can be calculated by looking at the 
physics of the experiment; it is finite and less than the overload value of the 
pixel, otherwise you should do a different experiment). Since you don't know 
that number, all the x values are equally likely - you use a uniform prior.
3) what is the PDF P(l|k) of l if we observe k counts?  That can be found with 
Bayes theorem, and it turns out that (due to the uniform prior) the right hand 
side of the formula looks the same as in 1) : P(l|k) = l^k*e^(-l)/k! (again, 
the ! stands for the factorial, it is not a semantic exclamation mark). This is 
eqs. 7.42 and 7.43 in Agostini "Bayesian Reasoning in Data Analysis".
3a) side note: if we calculate the expectation value for l, by multiplying with 
l and integrating over l from 0 to infinity, we obtain E(P(l|k))=k+1, and 
similarly for the variance (Agostini eqs 7.45 and 7.46)
4) for k=0 (zero counts observed in a single pixel), this reduces to 
P(l|0)=e^(-l) for a single observation (pixel). (this is basic math; see also 
§7.4.1 of Agostini.
5) since we have 100 independent pixels, we must multiply the individual PDFs 
to get the overall PDF f, and also normalize to make the integral over that PDF 
to be 1: the result is f(l|all 100 pixels are 0)=n*e^(-n*l). (basic math). A 
more Bayesian procedure would be to realize that the posterior PDF 
P(l|0)=e^(-l) of the first pixel should be used as the prior for the second 
pixel, and so forth until the 100th pixel. This has the same result f(l|all 100 
pixels are 0)=n*e^(-n*l) (Agostini § 7.7.2)!
6) the expectation value INTEGRAL_0_to_infinity over l*n*e^(-n*l) dl is 1/n .  
This is 1 if n=1 as we know from 3a), and 1/100 for 100 pixels with 0 counts.
7) the variance is then INTEGRAL_0_to_infinity over (l-1/n)^2*n*e^(-n*l) dl . 
This is 1/n^2 

I find these results quite satisfactory. Please note that they deviate from the 
MLE result: expectation value=0, variance=0 . The problem appears to be that a 
Maximum Likelihood Estimator may give wrong results for small n; something that 
I've read a couple of times but which appears not to be universally 
known/taught. Clearly, the result in 6) and 7) for large n converges towards 0, 
as it should be.
What this also means is that one should really work out the PDF instead of just 
adding expectation values and variances (and arriving at 100 if all 100 pixels 
have zero counts) because it is contradictory to use a uniform prior for all 
the pixels if OTOH these agree perfectly in being 0!

What this means for zero-dose extrapolation I have not thought about. At least 
it prevents infinite weights!

Best,
Kay 



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] am I doing this right?

[Frank von Delft]

Thanks, I learnt two things now - one of which being that I'm credited 
with coining that word!  Stap me vittals...


If it's single photon events you're after, isn't it quantum statistics 
where you need to go find that prior?  (Or is that what you're doing in 
this thread - I wouldn't be able to tell.)


Also:  should the detectors change how they read out things, then? Just 
write out the events with timestamp, rather than dumping all pixels all 
the time into these arbitrary containers called "image". Or is that 
what's already happening in HDF5 (which I don't understand one bit, I 
should add).


Frank




On 17/10/2021 18:12, James Holton wrote:


Well Frank, I think it comes down to something I believe you were the 
first to call "dose slicing".


Like fine phi slicing, collecting a larger number of weaker images 
records the same photons, but with more information about the sample 
before it dies. In fine phi slicing the extra information allows you 
to do better background rejection, and in "dose slicing" the extra 
information is about radiation damage. We lose that information when 
we use longer exposures per image, and if you burn up the entire 
useful life of your crystal in one shot, then all information about 
how the spots decayed during the exposure is lost. Your data are also 
rather incomplete.


How much information is lost? Well, how much more disk space would be 
taken up, even after compression, if you collected only 1 photon per 
image?  And kept collecting all the way out to 30 MGy in dose? That's 
about 1 million photons (images) per cubic micron of crystal.  So, I'd 
say the amount of information lost is "quite a bit".


But what makes matters worse is that if you did collect this data set 
and preserved all information available from your crystal you'd have 
no way to process it. This is not because its impossible, its just 
that we don't have the software. Your only choice would be to go find 
images with the same "phi" value and add them together until you have 
enough photons/pixel to index it. Once you've got an indexing solution 
you can map every photon hit to a position in reciprocal space as well 
as give it a time/dose stamp. What do you do with that?  You can do 
zero-dose extrapolation, of course!  Damage-free data! Wouldn't that 
be nice. Or can you?  The data you will have in hand for each 
reciprocal-space pixel might look something like:
tic tic .. tic . tic ... tic tictic ... 
tictic.


So. Eight photons.  With time-of-arrival information.  How do you fit 
a straight line to that?  You could "bin" the data or do some kind of 
smoothing thing, but then you are losing information again. Perhaps 
also making ill-founded assumptions. You need error bars of some kind, 
and, better yet, the shape of the distribution implied by those error 
bars.


And all this makes me think somebody must have already done this. I'm 
willing to bet probably some time in the late 1700s to early 1800s. 
All we're really talking about here is augmenting maximum-likelihood 
estimation of an average value to maximum-likelihood estimation of a 
straight line. That is, slope and intercept, with sigmas on both. I 
suspect the proper approach is to first bring everything down to the 
exact information content of a single photon (or lack of a photon), 
and build up from there.  If you are lucky enough to have a large 
number of photons then linear regression will work, and you are back 
to Diederichs (2003). But when you're photon-starved the statistics of 
single photons become more and more important.  This led me to: is it 
k? or k+1 ?  When k=0 getting this wrong could introduce a factor of 
infinity.


So, perhaps the big "consequence of getting it wrong" is embarrassing 
myself by re-making a 200-year old mistake I am not currently aware 
of. I am confident a solution exists, but only recently started 
working on this.  So, I figured ... ask the world?


-James Holton
MAD Scientist


On 10/17/2021 1:51 AM, Frank Von Delft wrote:
James, I've been watching the thread with fascination, but also the 
confusion of wild ignorance. I've finally realised why.


What I've missed is: what exactly makes the question so important?  
I've understood what brought it up, if course, but not the 
consequence of getting it wrong.


Frank

Sent from tiny silly touch screen

*From:* James Holton 
*Sent:* Saturday, 16 October 2021 20:01
*To:* CCP4BB@JISCMAIL.AC.UK
*Subject:* Re: [ccp4bb] am I doing this right?

Thank you everyone for your thoughtful and thought-provoking responses!

But, I am starting to think I was not as clear as I could have been
about my question.  I am actually concerning myself with background, not
necessarily Bragg peaks.  With Bragg photons you want the sum, but for
background you want the averag

Re: [ccp4bb] am I doing this right?

[Gergely Katona]

Dear James,

No, the prior does not have to be a gamma distribution. As I understand (and I 
am out of my comfort zone here), the reason for choosing gamma is indeed 
convenience. I believe this convenience is rapidly lost as the Bayesian network 
gets complicated and any hope of an analytical expression for a posterior 
disappears. 

The Poisson distribution has a given likelihood function, so let's look for a 
function to pair with that can achieve something. The need for pairing comes 
from the product of the likelihood and the prior probability. The probability 
of the data from the Bayes formula is just a constant as long as we compare the 
posterior probabilities of the parameters given the same data. As you probably 
noticed in my example the posterior distribution has an exponential shape even 
if it was generated by sampling. But, I would also expect that this will 
transform to a bell shape as the rate increases, very much like much like the 
Poisson distribution changes only the gamma distribution is continuous. So 
would not it be nice to have a prior distribution that matches this posterior? 
Then one could recycle the posterior as a new prior when new data is added 
until the end of time.
One could also define the posterior with just two precise parameter values, 
instead of my embarrassing sampling of this simple scenario. One could compare 
these precise parameter values of the prior and posterior etc.

Aside this convenience, I have other reasons to prefer the gamma distribution 
over a uniform distribution: 
1. Lack of region with 0 probability (well, within the positive territory). It 
is possible to convert even strong priors with a lot of evidence, but not a 
prior probability of 0. That value will always default to 0 posterior 
probability the equivalent of a dogma. 
2. With rates perhaps you would like to consider 0.1, 0.01 and 1E-6 with equal 
probability, but a uniform distribution does not say that. So perhaps the 
logarithm of the rates should have equal probabilities... And voila, the 
exponential distribution does not sound like a bad idea, which happens to be 
the same as a gamma function with alpha=1. It is still prudent to choose a flat 
exponential distribution as a weakly informative prior.
 
In the end, mathematical convenience alone should not dictate what a good prior 
is, in my opinion the expertise and crystallographic needs should take 
precedence. In most of the cases there is much to gain with continuing building 
more complicated Bayesian networks. I always underestimate the power of 
sampling methods and get surprised that they work on problems that I believed 
intractable. It is easy to fall into the trap of frequentist thinking and 
reduce data one step at the time.

Best wishes,

Gergely


-Original Message-
From: James Holton  
Sent: 17 October, 2021 19:25
To: Gergely Katona ; CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Thank you Gergely.  That is interesting!

I don't mind at all making this Bayesian, as long as it works!

Something I'm not quite sure about: does the prior distribution HAVE to be a 
gamma distribution? Not that that really narrows things down since there are an 
infinite number of them, but is that really the "i have no idea" prior? Or just 
a convenient closed-form choice? I've only just recently heard of conjugate 
priors.

Much appreciate any thoughts you may have on this,

-James


On 10/16/2021 3:48 PM, Gergely Katona wrote:
> Dear James,
>
> If I understand correctly you are looking for a single rate parameter to 
> describe the pixels in a block. It would also be possible to estimate the 
> rates for individual pixels or estimate the thickness of the sample from the 
> counts if you have a good model, that is where Bayesian methods really shine. 
> I tested the simplest first Bayesian network with 10 and 100 zero count 
> pixels, respectively:
>
> https://colab.research.google.com/drive/1TGJx2YT9I-qyOT1D9_HCC7G7as1KX
> g2e?usp=sharing
>
>
> The two posterior distributions are markedly different even if they start 
> from the same prior distribution, which I find more intuitive than the 
> frequentist treatment of uncertainty. You can test different parameters for 
> the gamma prior or change to another prior distribution. It is possible to 
> reduce the posterior distributions to their mean or posterior maximum, if 
> needed. If you are looking for an alternative to the Bayesian perspective 
> then this will not help, unfortunately.
>
> Best wishes,
>
> Gergely
>
> -Original Message-
> From: CCP4 bulletin board  On Behalf Of James 
> Holton
> Sent: den 16 oktober 2021 21:01
> To: CCP4BB@JISCMAIL.AC.UK
> Subject: Re: [ccp4bb] am I doing this right?
>
> Thank you everyone for your thoughtful and thought-provoking responses!
>
> But, I am starting to think I was not as clear as I cou

Re: [ccp4bb] am I doing this right?

[Nave, Colin (DLSLtd,RAL,LSCI)]

Hi James
For the case under consideration,  isn't the gamma distribution the maximum 
entropy prior i.e. a default with minimum information content. 
Colin

-Original Message-
From: CCP4 bulletin board  On Behalf Of James Holton
Sent: 17 October 2021 18:25
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Thank you Gergely.  That is interesting!

I don't mind at all making this Bayesian, as long as it works!

Something I'm not quite sure about: does the prior distribution HAVE to be a 
gamma distribution? Not that that really narrows things down since there are an 
infinite number of them, but is that really the "i have no idea" prior? Or just 
a convenient closed-form choice? I've only just recently heard of conjugate 
priors.

Much appreciate any thoughts you may have on this,

-James


On 10/16/2021 3:48 PM, Gergely Katona wrote:
> Dear James,
>
> If I understand correctly you are looking for a single rate parameter to 
> describe the pixels in a block. It would also be possible to estimate the 
> rates for individual pixels or estimate the thickness of the sample from the 
> counts if you have a good model, that is where Bayesian methods really shine. 
> I tested the simplest first Bayesian network with 10 and 100 zero count 
> pixels, respectively:
>
> https://colab.research.google.com/drive/1TGJx2YT9I-qyOT1D9_HCC7G7as1KX
> g2e?usp=sharing
>
>
> The two posterior distributions are markedly different even if they start 
> from the same prior distribution, which I find more intuitive than the 
> frequentist treatment of uncertainty. You can test different parameters for 
> the gamma prior or change to another prior distribution. It is possible to 
> reduce the posterior distributions to their mean or posterior maximum, if 
> needed. If you are looking for an alternative to the Bayesian perspective 
> then this will not help, unfortunately.
>
> Best wishes,
>
> Gergely
>
> -Original Message-
> From: CCP4 bulletin board  On Behalf Of James 
> Holton
> Sent: den 16 oktober 2021 21:01
> To: CCP4BB@JISCMAIL.AC.UK
> Subject: Re: [ccp4bb] am I doing this right?
>
> Thank you everyone for your thoughtful and thought-provoking responses!
>
> But, I am starting to think I was not as clear as I could have been about my 
> question.  I am actually concerning myself with background, not necessarily 
> Bragg peaks.  With Bragg photons you want the sum, but for background you 
> want the average.
>
> What I'm getting at is: how does one properly weight a zero-photon 
> observation when it comes time to combine it with others?  Hopefully they are 
> not all zero.  If they are, check your shutter.
>
> So, ignoring Bragg photons for the moment (let us suppose it is a systematic 
> absence) what I am asking is: what is the variance, or, better yet,what is 
> the WEIGHT one should assign to the observation of zero photons in a patch of 
> 10x10 pixels?
>
> In the absence of any prior knowledge this is a difficult question, but a 
> question we kind of need to answer if we want to properly measure data from 
> weak images.  So, what do we do?
>
> Well, with the "I have no idea" uniform prior, it would seem that expectation 
> (Epix) and variance (Vpix) would be k+1 = 1 for each pixel, and therefore the 
> sum of Epix and Vpix over the 100 independent pixels is:
>
> Epatch=Vpatch=100 photons
>
> I know that seems weird to assume 100 photons should have hit when we 
> actually saw none, but consider what that zero-photon count, all by itself, 
> is really telling you:
> a) Epix > 20 ? No way. That is "right out". Given we know its Poisson 
> distributed, and that background is flat, it is VERY unlikely you have E that 
> big when you saw zero. Cross all those E values off your list.
> b) Epix=0 ? Well, that CAN be true, but other things are possible and all of 
> them are E>0. So, most likely E is not 0, but at least a little bit higher.
> c) Epix=1e-6 ?  Yeah, sure, why not?
> d) Epix= -1e-6 ?  No. Don't be silly.
> e) If I had to guess? Meh. 1 photon per pixel?  That would be k+1
>
> I suppose my objection to E=V=0 is because V=0 implies infinite confidence in 
> the value of E, and that we don't have. Yes, it is true that we are quite 
> confident in the fact that we did not see any photons this time, but the 
> remember that E and V are the mean and variance that you would see if you did 
> a million experiments under the same conditions. We are trying to guess those 
> from what we've got. Just because you've seen zero a hundred times doesn't 
> mean the 101st experiment won't give you a count.  If it does, then maybe 
> Epatch=0.01 and Epix=0.0001?  But what do you do before you see your first 
> photon?
> All 

Re: [ccp4bb] am I doing this right?

[Jurgen Bosch]

The EM and Cryo-EM world might have something to say about that perhaps.

Isn’t RIP phasing coming from what you are describing as well?

Just curious,

Jürgen

__
Jürgen Bosch, PhD, MBA
Center for Global Health & Diseases
Case Western Reserve University

https://www.linkedin.com/in/jubosch/

CEO & Co-Founder at InterRayBio, LLC

Johns Hopkins University
Bloomberg School of Public Health
Department of Biochemistry & Molecular Biology

On Oct 17, 2021, at 13:24, James Holton  wrote:

Thank you Gergely.  That is interesting!

I don't mind at all making this Bayesian, as long as it works!

Something I'm not quite sure about: does the prior distribution HAVE to
be a gamma distribution? Not that that really narrows things down since
there are an infinite number of them, but is that really the "i have no
idea" prior? Or just a convenient closed-form choice? I've only just
recently heard of conjugate priors.

Much appreciate any thoughts you may have on this,

-James


On 10/16/2021 3:48 PM, Gergely Katona wrote:

Dear James,


If I understand correctly you are looking for a single rate parameter to
describe the pixels in a block. It would also be possible to estimate the
rates for individual pixels or estimate the thickness of the sample from
the counts if you have a good model, that is where Bayesian methods really
shine. I tested the simplest first Bayesian network with 10 and 100 zero
count pixels, respectively:


https://colab.research.google.com/drive/1TGJx2YT9I-qyOT1D9_HCC7G7as1KXg2e?usp=sharing



The two posterior distributions are markedly different even if they start
from the same prior distribution, which I find more intuitive than the
frequentist treatment of uncertainty. You can test different parameters for
the gamma prior or change to another prior distribution. It is possible to
reduce the posterior distributions to their mean or posterior maximum, if
needed. If you are looking for an alternative to the Bayesian perspective
then this will not help, unfortunately.


Best wishes,


Gergely


-Original Message-

From: CCP4 bulletin board  On Behalf Of James Holton

Sent: den 16 oktober 2021 21:01

To: CCP4BB@JISCMAIL.AC.UK

Subject: Re: [ccp4bb] am I doing this right?


Thank you everyone for your thoughtful and thought-provoking responses!


But, I am starting to think I was not as clear as I could have been about
my question.  I am actually concerning myself with background, not
necessarily Bragg peaks.  With Bragg photons you want the sum, but for
background you want the average.


What I'm getting at is: how does one properly weight a zero-photon
observation when it comes time to combine it with others?  Hopefully they
are not all zero.  If they are, check your shutter.


So, ignoring Bragg photons for the moment (let us suppose it is a
systematic absence) what I am asking is: what is the variance, or, better
yet,what is the WEIGHT one should assign to the observation of zero photons
in a patch of 10x10 pixels?


In the absence of any prior knowledge this is a difficult question, but a
question we kind of need to answer if we want to properly measure data from
weak images.  So, what do we do?


Well, with the "I have no idea" uniform prior, it would seem that
expectation (Epix) and variance (Vpix) would be k+1 = 1 for each pixel, and
therefore the sum of Epix and Vpix over the 100 independent pixels is:


Epatch=Vpatch=100 photons


I know that seems weird to assume 100 photons should have hit when we
actually saw none, but consider what that zero-photon count, all by itself,
is really telling you:

a) Epix > 20 ? No way. That is "right out". Given we know its Poisson
distributed, and that background is flat, it is VERY unlikely you have E
that big when you saw zero. Cross all those E values off your list.

b) Epix=0 ? Well, that CAN be true, but other things are possible and all
of them are E>0. So, most likely E is not 0, but at least a little bit
higher.

c) Epix=1e-6 ?  Yeah, sure, why not?

d) Epix= -1e-6 ?  No. Don't be silly.

e) If I had to guess? Meh. 1 photon per pixel?  That would be k+1


I suppose my objection to E=V=0 is because V=0 implies infinite confidence
in the value of E, and that we don't have. Yes, it is true that we are
quite confident in the fact that we did not see any photons this time, but
the remember that E and V are the mean and variance that you would see if
you did a million experiments under the same conditions. We are trying to
guess those from what we've got. Just because you've seen zero a hundred
times doesn't mean the 101st experiment won't give you a count.  If it
does, then maybe Epatch=0.01 and Epix=0.0001?  But what do you do before
you see your first photon?

All you can really do is bracket it.


But what if you come up with a better prior than "I have no idea" ?

Well, we do have other pixels on the detector, and presuming the background
i

Re: [ccp4bb] am I doing this right?

[James Holton]

Thank you Gergely.  That is interesting!

I don't mind at all making this Bayesian, as long as it works!

Something I'm not quite sure about: does the prior distribution HAVE to 
be a gamma distribution? Not that that really narrows things down since 
there are an infinite number of them, but is that really the "i have no 
idea" prior? Or just a convenient closed-form choice? I've only just 
recently heard of conjugate priors.


Much appreciate any thoughts you may have on this,

-James


On 10/16/2021 3:48 PM, Gergely Katona wrote:

Dear James,

If I understand correctly you are looking for a single rate parameter to 
describe the pixels in a block. It would also be possible to estimate the rates 
for individual pixels or estimate the thickness of the sample from the counts 
if you have a good model, that is where Bayesian methods really shine. I tested 
the simplest first Bayesian network with 10 and 100 zero count pixels, 
respectively:

https://colab.research.google.com/drive/1TGJx2YT9I-qyOT1D9_HCC7G7as1KXg2e?usp=sharing


The two posterior distributions are markedly different even if they start from 
the same prior distribution, which I find more intuitive than the frequentist 
treatment of uncertainty. You can test different parameters for the gamma prior 
or change to another prior distribution. It is possible to reduce the posterior 
distributions to their mean or posterior maximum, if needed. If you are looking 
for an alternative to the Bayesian perspective then this will not help, 
unfortunately.

Best wishes,

Gergely

-Original Message-
From: CCP4 bulletin board  On Behalf Of James Holton
Sent: den 16 oktober 2021 21:01
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Thank you everyone for your thoughtful and thought-provoking responses!

But, I am starting to think I was not as clear as I could have been about my 
question.  I am actually concerning myself with background, not necessarily 
Bragg peaks.  With Bragg photons you want the sum, but for background you want 
the average.

What I'm getting at is: how does one properly weight a zero-photon observation 
when it comes time to combine it with others?  Hopefully they are not all zero. 
 If they are, check your shutter.

So, ignoring Bragg photons for the moment (let us suppose it is a systematic 
absence) what I am asking is: what is the variance, or, better yet,what is the 
WEIGHT one should assign to the observation of zero photons in a patch of 10x10 
pixels?

In the absence of any prior knowledge this is a difficult question, but a 
question we kind of need to answer if we want to properly measure data from 
weak images.  So, what do we do?

Well, with the "I have no idea" uniform prior, it would seem that expectation 
(Epix) and variance (Vpix) would be k+1 = 1 for each pixel, and therefore the sum of Epix 
and Vpix over the 100 independent pixels is:

Epatch=Vpatch=100 photons

I know that seems weird to assume 100 photons should have hit when we actually 
saw none, but consider what that zero-photon count, all by itself, is really 
telling you:
a) Epix > 20 ? No way. That is "right out". Given we know its Poisson 
distributed, and that background is flat, it is VERY unlikely you have E that big when you 
saw zero. Cross all those E values off your list.
b) Epix=0 ? Well, that CAN be true, but other things are possible and all of them 
are E>0. So, most likely E is not 0, but at least a little bit higher.
c) Epix=1e-6 ?  Yeah, sure, why not?
d) Epix= -1e-6 ?  No. Don't be silly.
e) If I had to guess? Meh. 1 photon per pixel?  That would be k+1

I suppose my objection to E=V=0 is because V=0 implies infinite confidence in 
the value of E, and that we don't have. Yes, it is true that we are quite 
confident in the fact that we did not see any photons this time, but the 
remember that E and V are the mean and variance that you would see if you did a 
million experiments under the same conditions. We are trying to guess those 
from what we've got. Just because you've seen zero a hundred times doesn't mean 
the 101st experiment won't give you a count.  If it does, then maybe 
Epatch=0.01 and Epix=0.0001?  But what do you do before you see your first 
photon?
All you can really do is bracket it.

But what if you come up with a better prior than "I have no idea" ?
Well, we do have other pixels on the detector, and presuming the background is 
flat, or at least smooth, maybe the average counts/pixel is a better prior?

So, let us consider an ideal detector with 1e6 independent pixels. Let us 
further say that 1e5 background photons have hit that detector.  I want to 
still ignore Bragg photons because those have a very different prior 
distribution to the background.  Let us say we have masked off all the Bragg 
areas.

The average overall background is then 0.1 photons/pixel. Let us assign that to 
the prior probability Ppix = 0.1.  Now let us look again a

Re: [ccp4bb] am I doing this right?

[James Holton]

Well Frank, I think it comes down to something I believe you were the 
first to call "dose slicing".


Like fine phi slicing, collecting a larger number of weaker images 
records the same photons, but with more information about the sample 
before it dies. In fine phi slicing the extra information allows you to 
do better background rejection, and in "dose slicing" the extra 
information is about radiation damage. We lose that information when we 
use longer exposures per image, and if you burn up the entire useful 
life of your crystal in one shot, then all information about how the 
spots decayed during the exposure is lost. Your data are also rather 
incomplete.


How much information is lost? Well, how much more disk space would be 
taken up, even after compression, if you collected only 1 photon per 
image?  And kept collecting all the way out to 30 MGy in dose? That's 
about 1 million photons (images) per cubic micron of crystal.  So, I'd 
say the amount of information lost is "quite a bit".


But what makes matters worse is that if you did collect this data set 
and preserved all information available from your crystal you'd have no 
way to process it. This is not because its impossible, its just that we 
don't have the software. Your only choice would be to go find images 
with the same "phi" value and add them together until you have enough 
photons/pixel to index it. Once you've got an indexing solution you can 
map every photon hit to a position in reciprocal space as well as give 
it a time/dose stamp. What do you do with that?  You can do zero-dose 
extrapolation, of course! Damage-free data! Wouldn't that be nice. Or 
can you?  The data you will have in hand for each reciprocal-space pixel 
might look something like:
tic tic .. tic . tic ... tic tictic ... 
tictic.


So. Eight photons.  With time-of-arrival information.  How do you fit a 
straight line to that?  You could "bin" the data or do some kind of 
smoothing thing, but then you are losing information again. Perhaps also 
making ill-founded assumptions. You need error bars of some kind, and, 
better yet, the shape of the distribution implied by those error bars.


And all this makes me think somebody must have already done this. I'm 
willing to bet probably some time in the late 1700s to early 1800s. All 
we're really talking about here is augmenting maximum-likelihood 
estimation of an average value to maximum-likelihood estimation of a 
straight line. That is, slope and intercept, with sigmas on both. I 
suspect the proper approach is to first bring everything down to the 
exact information content of a single photon (or lack of a photon), and 
build up from there.  If you are lucky enough to have a large number of 
photons then linear regression will work, and you are back to Diederichs 
(2003). But when you're photon-starved the statistics of single photons 
become more and more important.  This led me to: is it k? or k+1 ?  When 
k=0 getting this wrong could introduce a factor of infinity.


So, perhaps the big "consequence of getting it wrong" is embarrassing 
myself by re-making a 200-year old mistake I am not currently aware of. 
I am confident a solution exists, but only recently started working on 
this.  So, I figured ... ask the world?


-James Holton
MAD Scientist


On 10/17/2021 1:51 AM, Frank Von Delft wrote:
James, I've been watching the thread with fascination, but also the 
confusion of wild ignorance. I've finally realised why.


What I've missed is: what exactly makes the question so important?  
I've understood what brought it up, if course, but not the consequence 
of getting it wrong.


Frank

Sent from tiny silly touch screen

*From:* James Holton 
*Sent:* Saturday, 16 October 2021 20:01
*To:* CCP4BB@JISCMAIL.AC.UK
*Subject:* Re: [ccp4bb] am I doing this right?

Thank you everyone for your thoughtful and thought-provoking responses!

But, I am starting to think I was not as clear as I could have been
about my question.  I am actually concerning myself with background, not
necessarily Bragg peaks.  With Bragg photons you want the sum, but for
background you want the average.

What I'm getting at is: how does one properly weight a zero-photon
observation when it comes time to combine it with others? Hopefully
they are not all zero.  If they are, check your shutter.

So, ignoring Bragg photons for the moment (let us suppose it is a
systematic absence) what I am asking is: what is the variance, or,
better yet,what is the WEIGHT one should assign to the observation of
zero photons in a patch of 10x10 pixels?

In the absence of any prior knowledge this is a difficult question, but
a question we kind of need to answer if we want to properly measure data
from weak images.  So, what do we do?

Well, with the "I have no idea" uniform 

Re: [ccp4bb] am I doing this right?

[Kay Diederichs]

If you write/analyze/improve a data-reduction program in crystallography, then 
these questions are important: how to estimate the intensity of a Bragg spot by 
evaluating the counts in the pixels of the signal area, and those of the 
background area? When calculating mean and standard deviation of counts, is the 
frequentist view the only correct one, or can we improve the result e.g. by 
thinking about priors?
Consequence of getting it wrong is less accurate intensities, and downstream 
results depending on the intensities.

This is at least how I understand James' questions.

Best wishes,
Kay

On Sun, 17 Oct 2021 08:51:30 +, Frank Von Delft 
 wrote:

>James, I've been watching the thread with fascination, but also the confusion 
>of wild ignorance. I've finally realised why.
>
>What I've missed is: what exactly makes the question so important?  I've 
>understood what brought it up, if course, but not the consequence of getting 
>it wrong.
>
>Frank



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] am I doing this right?

[Frank Von Delft]

James, I've been watching the thread with fascination, but also the confusion 
of wild ignorance. I've finally realised why.

What I've missed is: what exactly makes the question so important?  I've 
understood what brought it up, if course, but not the consequence of getting it 
wrong.

Frank

Sent from tiny silly touch screen<http://www.9folders.com/>

From: James Holton 
Sent: Saturday, 16 October 2021 20:01
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Thank you everyone for your thoughtful and thought-provoking responses!

But, I am starting to think I was not as clear as I could have been
about my question.  I am actually concerning myself with background, not
necessarily Bragg peaks.  With Bragg photons you want the sum, but for
background you want the average.

What I'm getting at is: how does one properly weight a zero-photon
observation when it comes time to combine it with others?  Hopefully
they are not all zero.  If they are, check your shutter.

So, ignoring Bragg photons for the moment (let us suppose it is a
systematic absence) what I am asking is: what is the variance, or,
better yet,what is the WEIGHT one should assign to the observation of
zero photons in a patch of 10x10 pixels?

In the absence of any prior knowledge this is a difficult question, but
a question we kind of need to answer if we want to properly measure data
from weak images.  So, what do we do?

Well, with the "I have no idea" uniform prior, it would seem that
expectation (Epix) and variance (Vpix) would be k+1 = 1 for each pixel,
and therefore the sum of Epix and Vpix over the 100 independent pixels is:

Epatch=Vpatch=100 photons

I know that seems weird to assume 100 photons should have hit when we
actually saw none, but consider what that zero-photon count, all by
itself, is really telling you:
a) Epix > 20 ? No way. That is "right out". Given we know its Poisson
distributed, and that background is flat, it is VERY unlikely you have E
that big when you saw zero. Cross all those E values off your list.
b) Epix=0 ? Well, that CAN be true, but other things are possible and
all of them are E>0. So, most likely E is not 0, but at least a little
bit higher.
c) Epix=1e-6 ?  Yeah, sure, why not?
d) Epix= -1e-6 ?  No. Don't be silly.
e) If I had to guess? Meh. 1 photon per pixel?  That would be k+1

I suppose my objection to E=V=0 is because V=0 implies infinite
confidence in the value of E, and that we don't have. Yes, it is true
that we are quite confident in the fact that we did not see any photons
this time, but the remember that E and V are the mean and variance that
you would see if you did a million experiments under the same
conditions. We are trying to guess those from what we've got. Just
because you've seen zero a hundred times doesn't mean the 101st
experiment won't give you a count.  If it does, then maybe Epatch=0.01
and Epix=0.0001?  But what do you do before you see your first photon?
All you can really do is bracket it.

But what if you come up with a better prior than "I have no idea" ?
Well, we do have other pixels on the detector, and presuming the
background is flat, or at least smooth, maybe the average counts/pixel
is a better prior?

So, let us consider an ideal detector with 1e6 independent pixels. Let
us further say that 1e5 background photons have hit that detector.  I
want to still ignore Bragg photons because those have a very different
prior distribution to the background.  Let us say we have masked off all
the Bragg areas.

The average overall background is then 0.1 photons/pixel. Let us assign
that to the prior probability Ppix = 0.1.  Now let us look again at that
patch of 10x10 pixels with zero counts on it.  We expected to see 10,
but got 0.  What are the odds of that?  Pretty remote.  Less than 1 in a
million.

I suspect in this situation where such an unlikely event has occurred it
should perhaps be given a variance larger than 100. Perhaps quite a bit
larger?  Subsequent "sigma-weighted" summation would then squash its
contribution down to effectively 0. So, relative to any other
observation with even a shred of merit it would have no impact. Giving
it V=0, however? That can't be right.

But what if Ppix=0.01 ?  Then we expect to see zero counts on our
100-pixel patch about 1/3 of the time. Same for 1-photon observations.
Giving these two kinds of observations the same weight seems more
sensible, given the prior.

Another prior might be to take the flux and sample thickness into
account.  Given the cross section of light elements the expected
photons/pixel on most any detector would be:

Ppix = 1.2e-5*flux*exposure*thickness*omega/Npixels
where:
Ppix = expected photons/pixel
Npixels = number of pixels on the detector
omega  = fraction of scattered photons that hit it (about 0.5)
thickness = thickness of sample and loop in microns
exposure = exposure time in seconds
flux = inciden

Re: [ccp4bb] am I doing this right?

[Gergely Katona]

Dear James,

If I understand correctly you are looking for a single rate parameter to 
describe the pixels in a block. It would also be possible to estimate the rates 
for individual pixels or estimate the thickness of the sample from the counts 
if you have a good model, that is where Bayesian methods really shine. I tested 
the simplest first Bayesian network with 10 and 100 zero count pixels, 
respectively:

https://colab.research.google.com/drive/1TGJx2YT9I-qyOT1D9_HCC7G7as1KXg2e?usp=sharing


The two posterior distributions are markedly different even if they start from 
the same prior distribution, which I find more intuitive than the frequentist 
treatment of uncertainty. You can test different parameters for the gamma prior 
or change to another prior distribution. It is possible to reduce the posterior 
distributions to their mean or posterior maximum, if needed. If you are looking 
for an alternative to the Bayesian perspective then this will not help, 
unfortunately.

Best wishes,

Gergely

-Original Message-
From: CCP4 bulletin board  On Behalf Of James Holton
Sent: den 16 oktober 2021 21:01
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Thank you everyone for your thoughtful and thought-provoking responses!

But, I am starting to think I was not as clear as I could have been about my 
question.  I am actually concerning myself with background, not necessarily 
Bragg peaks.  With Bragg photons you want the sum, but for background you want 
the average.

What I'm getting at is: how does one properly weight a zero-photon observation 
when it comes time to combine it with others?  Hopefully they are not all zero. 
 If they are, check your shutter.

So, ignoring Bragg photons for the moment (let us suppose it is a systematic 
absence) what I am asking is: what is the variance, or, better yet,what is the 
WEIGHT one should assign to the observation of zero photons in a patch of 10x10 
pixels?

In the absence of any prior knowledge this is a difficult question, but a 
question we kind of need to answer if we want to properly measure data from 
weak images.  So, what do we do?

Well, with the "I have no idea" uniform prior, it would seem that expectation 
(Epix) and variance (Vpix) would be k+1 = 1 for each pixel, and therefore the 
sum of Epix and Vpix over the 100 independent pixels is:

Epatch=Vpatch=100 photons

I know that seems weird to assume 100 photons should have hit when we actually 
saw none, but consider what that zero-photon count, all by itself, is really 
telling you:
a) Epix > 20 ? No way. That is "right out". Given we know its Poisson 
distributed, and that background is flat, it is VERY unlikely you have E that 
big when you saw zero. Cross all those E values off your list.
b) Epix=0 ? Well, that CAN be true, but other things are possible and all of 
them are E>0. So, most likely E is not 0, but at least a little bit higher.
c) Epix=1e-6 ?  Yeah, sure, why not?
d) Epix= -1e-6 ?  No. Don't be silly.
e) If I had to guess? Meh. 1 photon per pixel?  That would be k+1

I suppose my objection to E=V=0 is because V=0 implies infinite confidence in 
the value of E, and that we don't have. Yes, it is true that we are quite 
confident in the fact that we did not see any photons this time, but the 
remember that E and V are the mean and variance that you would see if you did a 
million experiments under the same conditions. We are trying to guess those 
from what we've got. Just because you've seen zero a hundred times doesn't mean 
the 101st experiment won't give you a count.  If it does, then maybe 
Epatch=0.01 and Epix=0.0001?  But what do you do before you see your first 
photon? 
All you can really do is bracket it.

But what if you come up with a better prior than "I have no idea" ? 
Well, we do have other pixels on the detector, and presuming the background is 
flat, or at least smooth, maybe the average counts/pixel is a better prior?

So, let us consider an ideal detector with 1e6 independent pixels. Let us 
further say that 1e5 background photons have hit that detector.  I want to 
still ignore Bragg photons because those have a very different prior 
distribution to the background.  Let us say we have masked off all the Bragg 
areas.

The average overall background is then 0.1 photons/pixel. Let us assign that to 
the prior probability Ppix = 0.1.  Now let us look again at that patch of 10x10 
pixels with zero counts on it.  We expected to see 10, but got 0.  What are the 
odds of that?  Pretty remote.  Less than 1 in a million.

I suspect in this situation where such an unlikely event has occurred it should 
perhaps be given a variance larger than 100. Perhaps quite a bit larger?  
Subsequent "sigma-weighted" summation would then squash its contribution down 
to effectively 0. So, relative to any other observation with even a shred of 
merit it would have no impact. Giving it V=0, however? That can

Re: [ccp4bb] am I doing this right?

[James Holton]

en, and 
results in 0 as the variance, right?

best,
Kay


On Fri, 15 Oct 2021 21:07:26 +, Gergely Katona  wrote:


Dear James,

Uniform distribution sounds like “I have no idea”, but a uniform distribution 
does not go from -inf to +inf. If I believe that every count from 0 to 65535 
has the same probability, then I also expect counts with an average of 32768 on 
the image. It is not an objective belief in the end and probably not a very 
good idea for an X-ray experiment if the number of observations are small. 
Concerning which variance is the right one, the frequentist view requires 
frequencies to be observed. In the absence of frequencies, there is no error 
estimate. Bayesians at least can determine a single distribution as an answer 
without observations and that will be their prior belief of the variance. 
Again, I would avoid a uniform a priori distribution for the variance. For a 
Poisson distribution the convenient conjugate prior is the gamma distribution. 
It can control the magnitude of k and strength of belief with its location and 
scale parameter, respectively.

Best wishes,

Gergely

Gergely Katona, Professor, Chairman of the Chemistry Program Council
Department of Chemistry and Molecular Biology, University of Gothenburg
Box 462, 40530 Göteborg, Sweden
Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
Web: http://katonalab.eu, Email: gergely.kat...@gu.se

From: CCP4 bulletin board  On Behalf Of James Holton
Sent: 15 October, 2021 18:06
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Well I'll be...

Kay Diederichs pointed out to me off-list that the k+1 expectation and variance from observing k 
photons is in "Bayesian Reasoning in Data Analysis: A Critical Introduction" by Giulio D. 
Agostini.  Granted, that is with a uniform prior, which I take as the Bayesean equivalent of 
"I have no idea".

So, if I'm looking to integrate a 10 x 10 patch of pixels on a weak detector 
image, and I find that area has zero counts, what variance shall I put on that 
observation?  Is it:

a) zero
b) 1.0
c) 100

Wish I could say there are no wrong answers, but I think at least two of those 
are incorrect,

-James Holton
MAD Scientist
On 10/13/2021 2:34 PM, Filipe Maia wrote:
I forgot to add probably the most important. James is correct, the expected 
value of u, the true mean, given a single observation k is indeed k+1 and k+1 
is also the mean square error of using k+1 as the estimator of the true mean.

Cheers,
Filipe

On Wed, 13 Oct 2021 at 23:17, Filipe Maia 
mailto:fil...@xray.bmc.uu.se>> wrote:
Hi,

The maximum likelihood estimator for a Poisson distributed variable is equal to 
the mean of the observations. In the case of a single observation, it will be 
equal to that observation. As Graeme suggested, you can calculate the 
probability mass function for a given observation with different Poisson 
parameters (i.e. true means) and see that function peaks when the parameter 
matches the observation.

The root mean squared error of the estimation of the true mean from a single 
observation k seems to be sqrt(k+2). Or to put it in another way, mean squared 
error, that is the expected value of (k-u)**2, for an observation k and a true 
mean u, is equal to k+2.

You can see some example calculations at 
https://colab.research.google.com/drive/1eoaNrDqaPnP-4FTGiNZxMllP7SFHkQuS?usp=sharing

Cheers,
Filipe

On Wed, 13 Oct 2021 at 17:14, Winter, Graeme (DLSLtd,RAL,LSCI) 
<6a19cead4548-dmarc-requ...@jiscmail.ac.uk<mailto:6a19cead4548-dmarc-requ...@jiscmail.ac.uk>>
 wrote:
This rang a bell to me last night, and I think you can derive this from first 
principles

If you assume an observation of N counts, you can calculate the probability of 
such an observation for a given Poisson rate constant X. If you then integrate 
over all possible value of X to work out the central value of the rate constant 
which is most likely to result in an observation of N I think you get X = N+1

I think it is the kind of calculation you can perform on a napkin, if memory 
serves

All the best Graeme


On 13 Oct 2021, at 16:10, Andrew Leslie - MRC LMB 
mailto:and...@mrc-lmb.cam.ac.uk>> wrote:

Hi Ian, James,

  I have a strong feeling that I have seen this result 
before, and it was due to Andy Hammersley at ESRF. I’ve done a literature 
search and there is a paper relating to errors in analysis of counting 
statistics (se below), but I had a quick look at this and could not find the 
(N+1) correction, so it must have been somewhere else. I Have cc’d Andy on this 
Email (hoping that this Email address from 2016 still works) and maybe he can 
throw more light on this. What I remember at the time I saw this was the 
simplicity of the correction.

Cheers,

Andrew

Reducing bias in the analysis of counting statistics data
Hammersley, AP<https://www.webofscience.com/wos/author/record/2665675> (Hammersley, 
AP) A

Re: [ccp4bb] am I doing this right?

[Gergely Katona]

Dear Kay,

Yes, I agree. I am sorry I was too focused on the errors. The question I was 
trying to address is how useful frequentist error estimates are when the 
observed counts are 0. I guess both frequentists and Bayesians agree on the 
definition of descriptive statistics when the entire population is known. The 
disagreement comes from the uncertainty either because we do not see the future 
or somehow we are prevented to see the entire population. It is a bit 
artificial example, but perhaps we only see 100 pixels of the detector and we 
try to describe how the rest of, say, 00 unknown pixels might look like. 
Based on the mean of 0 I have to conclude that the rest of the pixels must also 
have 0 counts since there is no range for errors in the rate parameter. And it 
does not matter if I initially observe 100, 10, or 2 pixels with 0 photons. If 
this is the frequentist interpretation of the error based on 0 counts then I am 
not sure how useful it is. 

Best wishes,

Gergely

-Original Message-
From: Kay Diederichs  
Sent: den 16 oktober 2021 09:48
To: CCP4BB@JISCMAIL.AC.UK; Gergely Katona 
Subject: Re: am I doing this right?

Dear Gergely,

with " 10 x 10 patch of pixels ", I believe James means that he observes 100 
neighbouring pixels each with 0 counts. Thus the frequentist view can be taken, 
and results in 0 as the variance, right?

best,
Kay


On Fri, 15 Oct 2021 21:07:26 +, Gergely Katona  wrote:

>Dear James,
>
>Uniform distribution sounds like “I have no idea”, but a uniform distribution 
>does not go from -inf to +inf. If I believe that every count from 0 to 65535 
>has the same probability, then I also expect counts with an average of 32768 
>on the image. It is not an objective belief in the end and probably not a very 
>good idea for an X-ray experiment if the number of observations are small. 
>Concerning which variance is the right one, the frequentist view requires 
>frequencies to be observed. In the absence of frequencies, there is no error 
>estimate. Bayesians at least can determine a single distribution as an answer 
>without observations and that will be their prior belief of the variance. 
>Again, I would avoid a uniform a priori distribution for the variance. For a 
>Poisson distribution the convenient conjugate prior is the gamma distribution. 
>It can control the magnitude of k and strength of belief with its location and 
>scale parameter, respectively.
>
>Best wishes,
>
>Gergely
>
>Gergely Katona, Professor, Chairman of the Chemistry Program Council 
>Department of Chemistry and Molecular Biology, University of Gothenburg 
>Box 462, 40530 Göteborg, Sweden
>Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>Web: http://katonalab.eu, Email: gergely.kat...@gu.se
>
>From: CCP4 bulletin board  On Behalf Of James 
>Holton
>Sent: 15 October, 2021 18:06
>To: CCP4BB@JISCMAIL.AC.UK
>Subject: Re: [ccp4bb] am I doing this right?
>
>Well I'll be...
>
>Kay Diederichs pointed out to me off-list that the k+1 expectation and 
>variance from observing k photons is in "Bayesian Reasoning in Data Analysis: 
>A Critical Introduction" by Giulio D. Agostini.  Granted, that is with a 
>uniform prior, which I take as the Bayesean equivalent of "I have no idea".
>
>So, if I'm looking to integrate a 10 x 10 patch of pixels on a weak detector 
>image, and I find that area has zero counts, what variance shall I put on that 
>observation?  Is it:
>
>a) zero
>b) 1.0
>c) 100
>
>Wish I could say there are no wrong answers, but I think at least two 
>of those are incorrect,
>
>-James Holton
>MAD Scientist
>On 10/13/2021 2:34 PM, Filipe Maia wrote:
>I forgot to add probably the most important. James is correct, the expected 
>value of u, the true mean, given a single observation k is indeed k+1 and k+1 
>is also the mean square error of using k+1 as the estimator of the true mean.
>
>Cheers,
>Filipe
>
>On Wed, 13 Oct 2021 at 23:17, Filipe Maia 
>mailto:fil...@xray.bmc.uu.se>> wrote:
>Hi,
>
>The maximum likelihood estimator for a Poisson distributed variable is equal 
>to the mean of the observations. In the case of a single observation, it will 
>be equal to that observation. As Graeme suggested, you can calculate the 
>probability mass function for a given observation with different Poisson 
>parameters (i.e. true means) and see that function peaks when the parameter 
>matches the observation.
>
>The root mean squared error of the estimation of the true mean from a single 
>observation k seems to be sqrt(k+2). Or to put it in another way, mean squared 
>error, that is the expected value of (k-u)**2, for an observation k and a true 
>mean u, is equal to k+2.
>
>You can see some example calculations at 
>http

Re: [ccp4bb] am I doing this right?

[Ian Tickle]

Hi Rangana

That's correct.  To get the true population expectation and variance you
would need to sample the entire population, i.e. all the possible values
that the random variable can take at its expected frequency, though in
practice that's obviously not feasible and one has to be satisfied with a
"representative" sample.

Cheers

-- Ian


On Sat, 16 Oct 2021 at 09:07, Rangana Warshamanage 
wrote:

> If a random variable gets the same value in its all occurrences, its
> variance should be zero, isn't it? Or do I not understand that?
>
> Rangana.
>
> On Sat, 16 Oct 2021, 08:49 Kay Diederichs, 
> wrote:
>
>> Dear Gergely,
>>
>> with " 10 x 10 patch of pixels ", I believe James means that he observes
>> 100 neighbouring pixels each with 0 counts. Thus the frequentist view can
>> be taken, and results in 0 as the variance, right?
>>
>> best,
>> Kay
>>
>>
>> On Fri, 15 Oct 2021 21:07:26 +, Gergely Katona 
>> wrote:
>>
>> >Dear James,
>> >
>> >Uniform distribution sounds like “I have no idea”, but a uniform
>> distribution does not go from -inf to +inf. If I believe that every count
>> from 0 to 65535 has the same probability, then I also expect counts with an
>> average of 32768 on the image. It is not an objective belief in the end and
>> probably not a very good idea for an X-ray experiment if the number of
>> observations are small. Concerning which variance is the right one, the
>> frequentist view requires frequencies to be observed. In the absence of
>> frequencies, there is no error estimate. Bayesians at least can determine a
>> single distribution as an answer without observations and that will be
>> their prior belief of the variance. Again, I would avoid a uniform a priori
>> distribution for the variance. For a Poisson distribution the convenient
>> conjugate prior is the gamma distribution. It can control the magnitude of
>> k and strength of belief with its location and scale parameter,
>> respectively.
>> >
>> >Best wishes,
>> >
>> >Gergely
>> >
>> >Gergely Katona, Professor, Chairman of the Chemistry Program Council
>> >Department of Chemistry and Molecular Biology, University of Gothenburg
>> >Box 462, 40530 Göteborg, Sweden
>> >Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> >Web: http://katonalab.eu, Email: gergely.kat...@gu.se
>> >
>> >From: CCP4 bulletin board  On Behalf Of James
>> Holton
>> >Sent: 15 October, 2021 18:06
>> >To: CCP4BB@JISCMAIL.AC.UK
>> >Subject: Re: [ccp4bb] am I doing this right?
>> >
>> >Well I'll be...
>> >
>> >Kay Diederichs pointed out to me off-list that the k+1 expectation and
>> variance from observing k photons is in "Bayesian Reasoning in Data
>> Analysis: A Critical Introduction" by Giulio D. Agostini.  Granted, that is
>> with a uniform prior, which I take as the Bayesean equivalent of "I have no
>> idea".
>> >
>> >So, if I'm looking to integrate a 10 x 10 patch of pixels on a weak
>> detector image, and I find that area has zero counts, what variance shall I
>> put on that observation?  Is it:
>> >
>> >a) zero
>> >b) 1.0
>> >c) 100
>> >
>> >Wish I could say there are no wrong answers, but I think at least two of
>> those are incorrect,
>> >
>> >-James Holton
>> >MAD Scientist
>> >On 10/13/2021 2:34 PM, Filipe Maia wrote:
>> >I forgot to add probably the most important. James is correct, the
>> expected value of u, the true mean, given a single observation k is indeed
>> k+1 and k+1 is also the mean square error of using k+1 as the estimator of
>> the true mean.
>> >
>> >Cheers,
>> >Filipe
>> >
>> >On Wed, 13 Oct 2021 at 23:17, Filipe Maia > fil...@xray.bmc.uu.se>> wrote:
>> >Hi,
>> >
>> >The maximum likelihood estimator for a Poisson distributed variable is
>> equal to the mean of the observations. In the case of a single observation,
>> it will be equal to that observation. As Graeme suggested, you can
>> calculate the probability mass function for a given observation with
>> different Poisson parameters (i.e. true means) and see that function peaks
>> when the parameter matches the observation.
>> >
>> >The root mean squared error of the estimation of the true mean from a
>> single observation k seems to be sqrt(k+2). Or to put it in another way,
>> mean squared error, that is the expected valu

Re: [ccp4bb] am I doing this right?

[Rangana Warshamanage]

If a random variable gets the same value in its all occurrences, its
variance should be zero, isn't it? Or do I not understand that?

Rangana.

On Sat, 16 Oct 2021, 08:49 Kay Diederichs, 
wrote:

> Dear Gergely,
>
> with " 10 x 10 patch of pixels ", I believe James means that he observes
> 100 neighbouring pixels each with 0 counts. Thus the frequentist view can
> be taken, and results in 0 as the variance, right?
>
> best,
> Kay
>
>
> On Fri, 15 Oct 2021 21:07:26 +, Gergely Katona 
> wrote:
>
> >Dear James,
> >
> >Uniform distribution sounds like “I have no idea”, but a uniform
> distribution does not go from -inf to +inf. If I believe that every count
> from 0 to 65535 has the same probability, then I also expect counts with an
> average of 32768 on the image. It is not an objective belief in the end and
> probably not a very good idea for an X-ray experiment if the number of
> observations are small. Concerning which variance is the right one, the
> frequentist view requires frequencies to be observed. In the absence of
> frequencies, there is no error estimate. Bayesians at least can determine a
> single distribution as an answer without observations and that will be
> their prior belief of the variance. Again, I would avoid a uniform a priori
> distribution for the variance. For a Poisson distribution the convenient
> conjugate prior is the gamma distribution. It can control the magnitude of
> k and strength of belief with its location and scale parameter,
> respectively.
> >
> >Best wishes,
> >
> >Gergely
> >
> >Gergely Katona, Professor, Chairman of the Chemistry Program Council
> >Department of Chemistry and Molecular Biology, University of Gothenburg
> >Box 462, 40530 Göteborg, Sweden
> >Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >Web: http://katonalab.eu, Email: gergely.kat...@gu.se
> >
> >From: CCP4 bulletin board  On Behalf Of James
> Holton
> >Sent: 15 October, 2021 18:06
> >To: CCP4BB@JISCMAIL.AC.UK
> >Subject: Re: [ccp4bb] am I doing this right?
> >
> >Well I'll be...
> >
> >Kay Diederichs pointed out to me off-list that the k+1 expectation and
> variance from observing k photons is in "Bayesian Reasoning in Data
> Analysis: A Critical Introduction" by Giulio D. Agostini.  Granted, that is
> with a uniform prior, which I take as the Bayesean equivalent of "I have no
> idea".
> >
> >So, if I'm looking to integrate a 10 x 10 patch of pixels on a weak
> detector image, and I find that area has zero counts, what variance shall I
> put on that observation?  Is it:
> >
> >a) zero
> >b) 1.0
> >c) 100
> >
> >Wish I could say there are no wrong answers, but I think at least two of
> those are incorrect,
> >
> >-James Holton
> >MAD Scientist
> >On 10/13/2021 2:34 PM, Filipe Maia wrote:
> >I forgot to add probably the most important. James is correct, the
> expected value of u, the true mean, given a single observation k is indeed
> k+1 and k+1 is also the mean square error of using k+1 as the estimator of
> the true mean.
> >
> >Cheers,
> >Filipe
> >
> >On Wed, 13 Oct 2021 at 23:17, Filipe Maia  fil...@xray.bmc.uu.se>> wrote:
> >Hi,
> >
> >The maximum likelihood estimator for a Poisson distributed variable is
> equal to the mean of the observations. In the case of a single observation,
> it will be equal to that observation. As Graeme suggested, you can
> calculate the probability mass function for a given observation with
> different Poisson parameters (i.e. true means) and see that function peaks
> when the parameter matches the observation.
> >
> >The root mean squared error of the estimation of the true mean from a
> single observation k seems to be sqrt(k+2). Or to put it in another way,
> mean squared error, that is the expected value of (k-u)**2, for an
> observation k and a true mean u, is equal to k+2.
> >
> >You can see some example calculations at
> https://colab.research.google.com/drive/1eoaNrDqaPnP-4FTGiNZxMllP7SFHkQuS?usp=sharing
> >
> >Cheers,
> >Filipe
> >
> >On Wed, 13 Oct 2021 at 17:14, Winter, Graeme (DLSLtd,RAL,LSCI) <
> 6a19cead4548-dmarc-requ...@jiscmail.ac.uk 6a19cead4548-dmarc-requ...@jiscmail.ac.uk>> wrote:
> >This rang a bell to me last night, and I think you can derive this from
> first principles
> >
> >If you assume an observation of N counts, you can calculate the
> probability of such an observation for a given Poisson rate constant X. If
> you then integrate over all possible value of X to work out the central
&

Re: [ccp4bb] am I doing this right?

[Kay Diederichs]

Dear Gergely,

with " 10 x 10 patch of pixels ", I believe James means that he observes 100 
neighbouring pixels each with 0 counts. Thus the frequentist view can be taken, 
and results in 0 as the variance, right?

best,
Kay


On Fri, 15 Oct 2021 21:07:26 +, Gergely Katona  wrote:

>Dear James,
>
>Uniform distribution sounds like “I have no idea”, but a uniform distribution 
>does not go from -inf to +inf. If I believe that every count from 0 to 65535 
>has the same probability, then I also expect counts with an average of 32768 
>on the image. It is not an objective belief in the end and probably not a very 
>good idea for an X-ray experiment if the number of observations are small. 
>Concerning which variance is the right one, the frequentist view requires 
>frequencies to be observed. In the absence of frequencies, there is no error 
>estimate. Bayesians at least can determine a single distribution as an answer 
>without observations and that will be their prior belief of the variance. 
>Again, I would avoid a uniform a priori distribution for the variance. For a 
>Poisson distribution the convenient conjugate prior is the gamma distribution. 
>It can control the magnitude of k and strength of belief with its location and 
>scale parameter, respectively.
>
>Best wishes,
>
>Gergely
>
>Gergely Katona, Professor, Chairman of the Chemistry Program Council
>Department of Chemistry and Molecular Biology, University of Gothenburg
>Box 462, 40530 Göteborg, Sweden
>Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>Web: http://katonalab.eu, Email: gergely.kat...@gu.se
>
>From: CCP4 bulletin board  On Behalf Of James Holton
>Sent: 15 October, 2021 18:06
>To: CCP4BB@JISCMAIL.AC.UK
>Subject: Re: [ccp4bb] am I doing this right?
>
>Well I'll be...
>
>Kay Diederichs pointed out to me off-list that the k+1 expectation and 
>variance from observing k photons is in "Bayesian Reasoning in Data Analysis: 
>A Critical Introduction" by Giulio D. Agostini.  Granted, that is with a 
>uniform prior, which I take as the Bayesean equivalent of "I have no idea".
>
>So, if I'm looking to integrate a 10 x 10 patch of pixels on a weak detector 
>image, and I find that area has zero counts, what variance shall I put on that 
>observation?  Is it:
>
>a) zero
>b) 1.0
>c) 100
>
>Wish I could say there are no wrong answers, but I think at least two of those 
>are incorrect,
>
>-James Holton
>MAD Scientist
>On 10/13/2021 2:34 PM, Filipe Maia wrote:
>I forgot to add probably the most important. James is correct, the expected 
>value of u, the true mean, given a single observation k is indeed k+1 and k+1 
>is also the mean square error of using k+1 as the estimator of the true mean.
>
>Cheers,
>Filipe
>
>On Wed, 13 Oct 2021 at 23:17, Filipe Maia 
>mailto:fil...@xray.bmc.uu.se>> wrote:
>Hi,
>
>The maximum likelihood estimator for a Poisson distributed variable is equal 
>to the mean of the observations. In the case of a single observation, it will 
>be equal to that observation. As Graeme suggested, you can calculate the 
>probability mass function for a given observation with different Poisson 
>parameters (i.e. true means) and see that function peaks when the parameter 
>matches the observation.
>
>The root mean squared error of the estimation of the true mean from a single 
>observation k seems to be sqrt(k+2). Or to put it in another way, mean squared 
>error, that is the expected value of (k-u)**2, for an observation k and a true 
>mean u, is equal to k+2.
>
>You can see some example calculations at 
>https://colab.research.google.com/drive/1eoaNrDqaPnP-4FTGiNZxMllP7SFHkQuS?usp=sharing
>
>Cheers,
>Filipe
>
>On Wed, 13 Oct 2021 at 17:14, Winter, Graeme (DLSLtd,RAL,LSCI) 
><6a19cead4548-dmarc-requ...@jiscmail.ac.uk<mailto:6a19cead4548-dmarc-requ...@jiscmail.ac.uk>>
> wrote:
>This rang a bell to me last night, and I think you can derive this from first 
>principles
>
>If you assume an observation of N counts, you can calculate the probability of 
>such an observation for a given Poisson rate constant X. If you then integrate 
>over all possible value of X to work out the central value of the rate 
>constant which is most likely to result in an observation of N I think you get 
>X = N+1
>
>I think it is the kind of calculation you can perform on a napkin, if memory 
>serves
>
>All the best Graeme
>
>
>On 13 Oct 2021, at 16:10, Andrew Leslie - MRC LMB 
>mailto:and...@mrc-lmb.cam.ac.uk>> wrote:
>
>Hi Ian, James,
>
>  I have a strong feeling that I have seen this result 
> before, and it was due to Andy Hammersley at ESRF. I’ve

Re: [ccp4bb] am I doing this right?

[Gergely Katona]

Dear James,

Uniform distribution sounds like “I have no idea”, but a uniform distribution 
does not go from -inf to +inf. If I believe that every count from 0 to 65535 
has the same probability, then I also expect counts with an average of 32768 on 
the image. It is not an objective belief in the end and probably not a very 
good idea for an X-ray experiment if the number of observations are small. 
Concerning which variance is the right one, the frequentist view requires 
frequencies to be observed. In the absence of frequencies, there is no error 
estimate. Bayesians at least can determine a single distribution as an answer 
without observations and that will be their prior belief of the variance. 
Again, I would avoid a uniform a priori distribution for the variance. For a 
Poisson distribution the convenient conjugate prior is the gamma distribution. 
It can control the magnitude of k and strength of belief with its location and 
scale parameter, respectively.

Best wishes,

Gergely

Gergely Katona, Professor, Chairman of the Chemistry Program Council
Department of Chemistry and Molecular Biology, University of Gothenburg
Box 462, 40530 Göteborg, Sweden
Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
Web: http://katonalab.eu, Email: gergely.kat...@gu.se

From: CCP4 bulletin board  On Behalf Of James Holton
Sent: 15 October, 2021 18:06
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] am I doing this right?

Well I'll be...

Kay Diederichs pointed out to me off-list that the k+1 expectation and variance 
from observing k photons is in "Bayesian Reasoning in Data Analysis: A Critical 
Introduction" by Giulio D. Agostini.  Granted, that is with a uniform prior, 
which I take as the Bayesean equivalent of "I have no idea".

So, if I'm looking to integrate a 10 x 10 patch of pixels on a weak detector 
image, and I find that area has zero counts, what variance shall I put on that 
observation?  Is it:

a) zero
b) 1.0
c) 100

Wish I could say there are no wrong answers, but I think at least two of those 
are incorrect,

-James Holton
MAD Scientist
On 10/13/2021 2:34 PM, Filipe Maia wrote:
I forgot to add probably the most important. James is correct, the expected 
value of u, the true mean, given a single observation k is indeed k+1 and k+1 
is also the mean square error of using k+1 as the estimator of the true mean.

Cheers,
Filipe

On Wed, 13 Oct 2021 at 23:17, Filipe Maia 
mailto:fil...@xray.bmc.uu.se>> wrote:
Hi,

The maximum likelihood estimator for a Poisson distributed variable is equal to 
the mean of the observations. In the case of a single observation, it will be 
equal to that observation. As Graeme suggested, you can calculate the 
probability mass function for a given observation with different Poisson 
parameters (i.e. true means) and see that function peaks when the parameter 
matches the observation.

The root mean squared error of the estimation of the true mean from a single 
observation k seems to be sqrt(k+2). Or to put it in another way, mean squared 
error, that is the expected value of (k-u)**2, for an observation k and a true 
mean u, is equal to k+2.

You can see some example calculations at 
https://colab.research.google.com/drive/1eoaNrDqaPnP-4FTGiNZxMllP7SFHkQuS?usp=sharing

Cheers,
Filipe

On Wed, 13 Oct 2021 at 17:14, Winter, Graeme (DLSLtd,RAL,LSCI) 
<6a19cead4548-dmarc-requ...@jiscmail.ac.uk<mailto:6a19cead4548-dmarc-requ...@jiscmail.ac.uk>>
 wrote:
This rang a bell to me last night, and I think you can derive this from first 
principles

If you assume an observation of N counts, you can calculate the probability of 
such an observation for a given Poisson rate constant X. If you then integrate 
over all possible value of X to work out the central value of the rate constant 
which is most likely to result in an observation of N I think you get X = N+1

I think it is the kind of calculation you can perform on a napkin, if memory 
serves

All the best Graeme


On 13 Oct 2021, at 16:10, Andrew Leslie - MRC LMB 
mailto:and...@mrc-lmb.cam.ac.uk>> wrote:

Hi Ian, James,

  I have a strong feeling that I have seen this result 
before, and it was due to Andy Hammersley at ESRF. I’ve done a literature 
search and there is a paper relating to errors in analysis of counting 
statistics (se below), but I had a quick look at this and could not find the 
(N+1) correction, so it must have been somewhere else. I Have cc’d Andy on this 
Email (hoping that this Email address from 2016 still works) and maybe he can 
throw more light on this. What I remember at the time I saw this was the 
simplicity of the correction.

Cheers,

Andrew

Reducing bias in the analysis of counting statistics data
Hammersley, AP<https://www.webofscience.com/wos/author/record/2665675> 
(Hammersley, AP) Antoniadis, 
A<https://www.webofscience.com/wos/author/record/13070551> (Antoniadis, A)
NUCLEAR INSTRUME

Re: [ccp4bb] am I doing this right?

[James Holton]

Well I'll be...

Kay Diederichs pointed out to me off-list that the k+1 expectation and 
variance from observing k photons is in "Bayesian Reasoning in Data 
Analysis: A Critical Introduction" by Giulio D. Agostini. Granted, that 
is with a uniform prior, which I take as the Bayesean equivalent of "I 
have no idea".


So, if I'm looking to integrate a 10 x 10 patch of pixels on a weak 
detector image, and I find that area has zero counts, what variance 
shall I put on that observation?  Is it:


a) zero
b) 1.0
c) 100

Wish I could say there are no wrong answers, but I think at least two of 
those are incorrect,


-James Holton
MAD Scientist

On 10/13/2021 2:34 PM, Filipe Maia wrote:
I forgot to add probably the most important. James is correct, the 
expected value of u, the true mean, given a single observation k is 
indeed k+1 and k+1 is also the mean square error of using k+1 as the 
estimator of the true mean.


Cheers,
Filipe

On Wed, 13 Oct 2021 at 23:17, Filipe Maia > wrote:


Hi,

The maximum likelihood estimator for a Poisson distributed
variable is equal to the mean of the observations. In the case of
a single observation, it will be equal to that observation. As
Graeme suggested, you can calculate the probability mass function
for a given observation with different Poisson parameters (i.e.
true means) and see that function peaks when the parameter matches
the observation.

The root mean squared error of the estimation of the true mean
from a single observation k seems to be sqrt(k+2). Or to put it in
another way, mean squared error, that is the expected value of
(k-u)**2, for an observation k and a true mean u, is equal to k+2.

You can see some example calculations at

https://colab.research.google.com/drive/1eoaNrDqaPnP-4FTGiNZxMllP7SFHkQuS?usp=sharing



Cheers,
Filipe

On Wed, 13 Oct 2021 at 17:14, Winter, Graeme (DLSLtd,RAL,LSCI)
<6a19cead4548-dmarc-requ...@jiscmail.ac.uk
> wrote:

This rang a bell to me last night, and I think you can derive
this from first principles

If you assume an observation of N counts, you can calculate
the probability of such an observation for a given Poisson
rate constant X. If you then integrate over all possible value
of X to work out the central value of the rate constant which
is most likely to result in an observation of N I think you
get X = N+1

I think it is the kind of calculation you can perform on a
napkin, if memory serves

All the best Graeme


On 13 Oct 2021, at 16:10, Andrew Leslie - MRC LMB
mailto:and...@mrc-lmb.cam.ac.uk>>
wrote:

Hi Ian, James,

                      I have a strong feeling that I have
seen this result before, and it was due to Andy Hammersley at
ESRF. I’ve done a literature search and there is a paper
relating to errors in analysis of counting statistics (se
below), but I had a quick look at this and could not find the
(N+1) correction, so it must have been somewhere else. I Have
cc’d Andy on this Email (hoping that this Email address from
2016 still works) and maybe he can throw more light on this.
What I remember at the time I saw this was the simplicity of
the correction.

Cheers,

Andrew


Reducing bias in the analysis of counting statistics data


  Hammersley, AP
  
(Hammersley,
  AP) Antoniadis, A
  
(Antoniadis,
  A)


  NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION
  A-ACCELERATORS SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT


  Volume


  394


  Issue

1-2


  Page

219-224


  DOI

10.1016/S0168-9002(97)00668-2


  Published

JUL 11 1997


On 12 Oct 2021, at 18:55, Ian Tickle mailto:ianj...@gmail.com>> wrote:


Hi James

What the Poisson distribution tells you is that if the true
count is N then the expectation and variance are also N. 
That's not the same thing as saying that for an observed
count N the expectation and variance are N.  Consider all
those cases where the observed count is exactly zero.  That
can arise from any number of true counts, though as you
noted larger values become increasingly unlikely.  However
those true counts are all >= 0 which means that the mean and
variance of those true counts must be positive and
non-zero.  From 

Re: [ccp4bb] am I doing this right?

[Filipe Maia]

I forgot to add probably the most important. James is correct, the expected
value of u, the true mean, given a single observation k is indeed k+1 and
k+1 is also the mean square error of using k+1 as the estimator of the true
mean.

Cheers,
Filipe

On Wed, 13 Oct 2021 at 23:17, Filipe Maia  wrote:

> Hi,
>
> The maximum likelihood estimator for a Poisson distributed variable is
> equal to the mean of the observations. In the case of a single observation,
> it will be equal to that observation. As Graeme suggested, you can
> calculate the probability mass function for a given observation with
> different Poisson parameters (i.e. true means) and see that function peaks
> when the parameter matches the observation.
>
> The root mean squared error of the estimation of the true mean from a
> single observation k seems to be sqrt(k+2). Or to put it in another way,
> mean squared error, that is the expected value of (k-u)**2, for an
> observation k and a true mean u, is equal to k+2.
>
> You can see some example calculations at
> https://colab.research.google.com/drive/1eoaNrDqaPnP-4FTGiNZxMllP7SFHkQuS?usp=sharing
>
> Cheers,
> Filipe
>
> On Wed, 13 Oct 2021 at 17:14, Winter, Graeme (DLSLtd,RAL,LSCI) <
> 6a19cead4548-dmarc-requ...@jiscmail.ac.uk> wrote:
>
>> This rang a bell to me last night, and I think you can derive this from
>> first principles
>>
>> If you assume an observation of N counts, you can calculate the
>> probability of such an observation for a given Poisson rate constant X. If
>> you then integrate over all possible value of X to work out the central
>> value of the rate constant which is most likely to result in an observation
>> of N I think you get X = N+1
>>
>> I think it is the kind of calculation you can perform on a napkin, if
>> memory serves
>>
>> All the best Graeme
>>
>> On 13 Oct 2021, at 16:10, Andrew Leslie - MRC LMB <
>> and...@mrc-lmb.cam.ac.uk> wrote:
>>
>> Hi Ian, James,
>>
>>   I have a strong feeling that I have seen this
>> result before, and it was due to Andy Hammersley at ESRF. I’ve done a
>> literature search and there is a paper relating to errors in analysis of
>> counting statistics (se below), but I had a quick look at this and could
>> not find the (N+1) correction, so it must have been somewhere else. I Have
>> cc’d Andy on this Email (hoping that this Email address from 2016 still
>> works) and maybe he can throw more light on this. What I remember at the
>> time I saw this was the simplicity of the correction.
>>
>> Cheers,
>>
>> Andrew
>>
>> Reducing bias in the analysis of counting statistics data
>> Hammersley, AP  
>> (Hammersley,
>> AP) Antoniadis, A
>>  (Antoniadis, A)
>> NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION A-ACCELERATORS
>> SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT Volume 394
>> Issue 1-2
>> Page 219-224
>> DOI 10.1016/S0168-9002(97)00668-2
>> Published JUL 11 1997
>>
>> On 12 Oct 2021, at 18:55, Ian Tickle  wrote:
>>
>>
>> Hi James
>>
>> What the Poisson distribution tells you is that if the true count is N
>> then the expectation and variance are also N.  That's not the same thing as
>> saying that for an observed count N the expectation and variance are N.
>> Consider all those cases where the observed count is exactly zero.  That
>> can arise from any number of true counts, though as you noted larger values
>> become increasingly unlikely.  However those true counts are all >= 0 which
>> means that the mean and variance of those true counts must be positive and
>> non-zero.  From your results they are both 1 though I haven't been through
>> the algebra to prove it.
>>
>> So what you are saying seems correct: for N observed counts we should be
>> taking the best estimate of the true value and variance as N+1.  For
>> reasonably large N the difference is small but if you are concerned with
>> weak images it might start to become significant.
>>
>> Cheers
>>
>> -- Ian
>>
>>
>> On Tue, 12 Oct 2021 at 17:56, James Holton  wrote:
>>
>>> All my life I have believed that if you're counting photons then the
>>> error of observing N counts is sqrt(N).  However, a calculation I just
>>> performed suggests its actually sqrt(N+1).
>>>
>>> My purpose here is to understand the weak-image limit of data
>>> processing. Question is: for a given pixel, if one photon is all you
>>> got, what do you "know"?
>>>
>>> I simulated millions of 1-second experiments. For each I used a "true"
>>> beam intensity (Itrue) between 0.001 and 20 photons/s. That is, for
>>> Itrue= 0.001 the average over a very long exposure would be 1 photon
>>> every 1000 seconds or so. For a 1-second exposure the observed count (N)
>>> is almost always zero. About 1 in 1000 of them will see one photon, and
>>> roughly 1 in a million will get N=2. I do 10,000 such experiments and
>>> put the results into a pile.  I then repeat with 

Re: [ccp4bb] am I doing this right?

[Filipe Maia]

Hi,

The maximum likelihood estimator for a Poisson distributed variable is
equal to the mean of the observations. In the case of a single observation,
it will be equal to that observation. As Graeme suggested, you can
calculate the probability mass function for a given observation with
different Poisson parameters (i.e. true means) and see that function peaks
when the parameter matches the observation.

The root mean squared error of the estimation of the true mean from a
single observation k seems to be sqrt(k+2). Or to put it in another way,
mean squared error, that is the expected value of (k-u)**2, for an
observation k and a true mean u, is equal to k+2.

You can see some example calculations at
https://colab.research.google.com/drive/1eoaNrDqaPnP-4FTGiNZxMllP7SFHkQuS?usp=sharing

Cheers,
Filipe

On Wed, 13 Oct 2021 at 17:14, Winter, Graeme (DLSLtd,RAL,LSCI) <
6a19cead4548-dmarc-requ...@jiscmail.ac.uk> wrote:

> This rang a bell to me last night, and I think you can derive this from
> first principles
>
> If you assume an observation of N counts, you can calculate the
> probability of such an observation for a given Poisson rate constant X. If
> you then integrate over all possible value of X to work out the central
> value of the rate constant which is most likely to result in an observation
> of N I think you get X = N+1
>
> I think it is the kind of calculation you can perform on a napkin, if
> memory serves
>
> All the best Graeme
>
> On 13 Oct 2021, at 16:10, Andrew Leslie - MRC LMB <
> and...@mrc-lmb.cam.ac.uk> wrote:
>
> Hi Ian, James,
>
>   I have a strong feeling that I have seen this result
> before, and it was due to Andy Hammersley at ESRF. I’ve done a literature
> search and there is a paper relating to errors in analysis of counting
> statistics (se below), but I had a quick look at this and could not find
> the (N+1) correction, so it must have been somewhere else. I Have cc’d Andy
> on this Email (hoping that this Email address from 2016 still works) and
> maybe he can throw more light on this. What I remember at the time I saw
> this was the simplicity of the correction.
>
> Cheers,
>
> Andrew
>
> Reducing bias in the analysis of counting statistics data
> Hammersley, AP  
> (Hammersley,
> AP) Antoniadis, A
>  (Antoniadis, A)
> NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION A-ACCELERATORS
> SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT Volume 394
> Issue 1-2
> Page 219-224
> DOI 10.1016/S0168-9002(97)00668-2
> Published JUL 11 1997
>
> On 12 Oct 2021, at 18:55, Ian Tickle  wrote:
>
>
> Hi James
>
> What the Poisson distribution tells you is that if the true count is N
> then the expectation and variance are also N.  That's not the same thing as
> saying that for an observed count N the expectation and variance are N.
> Consider all those cases where the observed count is exactly zero.  That
> can arise from any number of true counts, though as you noted larger values
> become increasingly unlikely.  However those true counts are all >= 0 which
> means that the mean and variance of those true counts must be positive and
> non-zero.  From your results they are both 1 though I haven't been through
> the algebra to prove it.
>
> So what you are saying seems correct: for N observed counts we should be
> taking the best estimate of the true value and variance as N+1.  For
> reasonably large N the difference is small but if you are concerned with
> weak images it might start to become significant.
>
> Cheers
>
> -- Ian
>
>
> On Tue, 12 Oct 2021 at 17:56, James Holton  wrote:
>
>> All my life I have believed that if you're counting photons then the
>> error of observing N counts is sqrt(N).  However, a calculation I just
>> performed suggests its actually sqrt(N+1).
>>
>> My purpose here is to understand the weak-image limit of data
>> processing. Question is: for a given pixel, if one photon is all you
>> got, what do you "know"?
>>
>> I simulated millions of 1-second experiments. For each I used a "true"
>> beam intensity (Itrue) between 0.001 and 20 photons/s. That is, for
>> Itrue= 0.001 the average over a very long exposure would be 1 photon
>> every 1000 seconds or so. For a 1-second exposure the observed count (N)
>> is almost always zero. About 1 in 1000 of them will see one photon, and
>> roughly 1 in a million will get N=2. I do 10,000 such experiments and
>> put the results into a pile.  I then repeat with Itrue=0.002,
>> Itrue=0.003, etc. All the way up to Itrue = 20. At Itrue > 20 I never
>> see N=1, not even in 1e7 experiments. With Itrue=0, I also see no N=1
>> events.
>> Now I go through my pile of results and extract those with N=1, and
>> count up the number of times a given Itrue produced such an event. The
>> histogram of Itrue values in this subset is itself Poisson, but with
>> mean = 2 ! If I similarly count up events 

Re: [ccp4bb] am I doing this right?

[Winter, Graeme (DLSLtd,RAL,LSCI)]

This rang a bell to me last night, and I think you can derive this from first 
principles

If you assume an observation of N counts, you can calculate the probability of 
such an observation for a given Poisson rate constant X. If you then integrate 
over all possible value of X to work out the central value of the rate constant 
which is most likely to result in an observation of N I think you get X = N+1

I think it is the kind of calculation you can perform on a napkin, if memory 
serves

All the best Graeme

On 13 Oct 2021, at 16:10, Andrew Leslie - MRC LMB 
mailto:and...@mrc-lmb.cam.ac.uk>> wrote:

Hi Ian, James,

  I have a strong feeling that I have seen this result 
before, and it was due to Andy Hammersley at ESRF. I’ve done a literature 
search and there is a paper relating to errors in analysis of counting 
statistics (se below), but I had a quick look at this and could not find the 
(N+1) correction, so it must have been somewhere else. I Have cc’d Andy on this 
Email (hoping that this Email address from 2016 still works) and maybe he can 
throw more light on this. What I remember at the time I saw this was the 
simplicity of the correction.

Cheers,

Andrew

Reducing bias in the analysis of counting statistics data
Hammersley, AP 
(Hammersley, AP) Antoniadis, 
A (Antoniadis, A)
NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION A-ACCELERATORS 
SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT
Volume
394
Issue
1-2
Page
219-224
DOI
10.1016/S0168-9002(97)00668-2
Published
JUL 11 1997

On 12 Oct 2021, at 18:55, Ian Tickle 
mailto:ianj...@gmail.com>> wrote:


Hi James

What the Poisson distribution tells you is that if the true count is N then the 
expectation and variance are also N.  That's not the same thing as saying that 
for an observed count N the expectation and variance are N.  Consider all those 
cases where the observed count is exactly zero.  That can arise from any number 
of true counts, though as you noted larger values become increasingly unlikely. 
 However those true counts are all >= 0 which means that the mean and variance 
of those true counts must be positive and non-zero.  From your results they are 
both 1 though I haven't been through the algebra to prove it.

So what you are saying seems correct: for N observed counts we should be taking 
the best estimate of the true value and variance as N+1.  For reasonably large 
N the difference is small but if you are concerned with weak images it might 
start to become significant.

Cheers

-- Ian


On Tue, 12 Oct 2021 at 17:56, James Holton 
mailto:jmhol...@lbl.gov>> wrote:
All my life I have believed that if you're counting photons then the
error of observing N counts is sqrt(N).  However, a calculation I just
performed suggests its actually sqrt(N+1).

My purpose here is to understand the weak-image limit of data
processing. Question is: for a given pixel, if one photon is all you
got, what do you "know"?

I simulated millions of 1-second experiments. For each I used a "true"
beam intensity (Itrue) between 0.001 and 20 photons/s. That is, for
Itrue= 0.001 the average over a very long exposure would be 1 photon
every 1000 seconds or so. For a 1-second exposure the observed count (N)
is almost always zero. About 1 in 1000 of them will see one photon, and
roughly 1 in a million will get N=2. I do 10,000 such experiments and
put the results into a pile.  I then repeat with Itrue=0.002,
Itrue=0.003, etc. All the way up to Itrue = 20. At Itrue > 20 I never
see N=1, not even in 1e7 experiments. With Itrue=0, I also see no N=1
events.
Now I go through my pile of results and extract those with N=1, and
count up the number of times a given Itrue produced such an event. The
histogram of Itrue values in this subset is itself Poisson, but with
mean = 2 ! If I similarly count up events where 2 and only 2 photons
were seen, the mean Itrue is 3. And if I look at only zero-count events
the mean and standard deviation is unity.

Does that mean the error of observing N counts is really sqrt(N+1) ?

I admit that this little exercise assumes that the distribution of Itrue
is uniform between 0.001 and 20, but given that one photon has been
observed Itrue values outside this range are highly unlikely. The
Itrue=0.001 and N=1 events are only a tiny fraction of the whole.  So, I
wold say that even if the prior distribution is not uniform, it is
certainly bracketed. Now, Itrue=0 is possible if the shutter didn't
open, but if the rest of the detector pixels have N=~1, doesn't this
affect the prior distribution of Itrue on our pixel of interest?

Of course, two or more photons are better than one, but these days with
small crystals and big detectors N=1 is no longer a trivial situation.
I look forward to hearing your take on this.  And no, this is not a trick.

-James Holton
MAD Scientist

Re: [ccp4bb] am I doing this right?

[Andrew Leslie - MRC LMB]

Hi Ian, James,

  I have a strong feeling that I have seen this result 
before, and it was due to Andy Hammersley at ESRF. I’ve done a literature 
search and there is a paper relating to errors in analysis of counting 
statistics (se below), but I had a quick look at this and could not find the 
(N+1) correction, so it must have been somewhere else. I Have cc’d Andy on this 
Email (hoping that this Email address from 2016 still works) and maybe he can 
throw more light on this. What I remember at the time I saw this was the 
simplicity of the correction.

Cheers,

Andrew

Reducing bias in the analysis of counting statistics data

Hammersley, AP  
(Hammersley, AP) Antoniadis, A 
 (Antoniadis, A)

NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION A-ACCELERATORS 
SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT

Volume

394

Issue

1-2
Page

219-224
DOI

10.1016/S0168-9002(97)00668-2
Published

JUL 11 1997

> On 12 Oct 2021, at 18:55, Ian Tickle  wrote:
> 
> 
> Hi James
> 
> What the Poisson distribution tells you is that if the true count is N then 
> the expectation and variance are also N.  That's not the same thing as saying 
> that for an observed count N the expectation and variance are N.  Consider 
> all those cases where the observed count is exactly zero.  That can arise 
> from any number of true counts, though as you noted larger values become 
> increasingly unlikely.  However those true counts are all >= 0 which means 
> that the mean and variance of those true counts must be positive and 
> non-zero.  From your results they are both 1 though I haven't been through 
> the algebra to prove it.
> 
> So what you are saying seems correct: for N observed counts we should be 
> taking the best estimate of the true value and variance as N+1.  For 
> reasonably large N the difference is small but if you are concerned with weak 
> images it might start to become significant.
> 
> Cheers
> 
> -- Ian
> 
> 
> On Tue, 12 Oct 2021 at 17:56, James Holton  > wrote:
> All my life I have believed that if you're counting photons then the 
> error of observing N counts is sqrt(N).  However, a calculation I just 
> performed suggests its actually sqrt(N+1).
> 
> My purpose here is to understand the weak-image limit of data 
> processing. Question is: for a given pixel, if one photon is all you 
> got, what do you "know"?
> 
> I simulated millions of 1-second experiments. For each I used a "true" 
> beam intensity (Itrue) between 0.001 and 20 photons/s. That is, for 
> Itrue= 0.001 the average over a very long exposure would be 1 photon 
> every 1000 seconds or so. For a 1-second exposure the observed count (N) 
> is almost always zero. About 1 in 1000 of them will see one photon, and 
> roughly 1 in a million will get N=2. I do 10,000 such experiments and 
> put the results into a pile.  I then repeat with Itrue=0.002, 
> Itrue=0.003, etc. All the way up to Itrue = 20. At Itrue > 20 I never 
> see N=1, not even in 1e7 experiments. With Itrue=0, I also see no N=1 
> events.
> Now I go through my pile of results and extract those with N=1, and 
> count up the number of times a given Itrue produced such an event. The 
> histogram of Itrue values in this subset is itself Poisson, but with 
> mean = 2 ! If I similarly count up events where 2 and only 2 photons 
> were seen, the mean Itrue is 3. And if I look at only zero-count events 
> the mean and standard deviation is unity.
> 
> Does that mean the error of observing N counts is really sqrt(N+1) ?
> 
> I admit that this little exercise assumes that the distribution of Itrue 
> is uniform between 0.001 and 20, but given that one photon has been 
> observed Itrue values outside this range are highly unlikely. The 
> Itrue=0.001 and N=1 events are only a tiny fraction of the whole.  So, I 
> wold say that even if the prior distribution is not uniform, it is 
> certainly bracketed. Now, Itrue=0 is possible if the shutter didn't 
> open, but if the rest of the detector pixels have N=~1, doesn't this 
> affect the prior distribution of Itrue on our pixel of interest?
> 
> Of course, two or more photons are better than one, but these days with 
> small crystals and big detectors N=1 is no longer a trivial situation.  
> I look forward to hearing your take on this.  And no, this is not a trick.
> 
> -James Holton
> MAD Scientist
> 
> 
> 
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 
> 
> 
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB 
> , a mailing list hosted by 
> www.jiscmail.ac.uk , terms & conditions are 
> 

Re: [ccp4bb] am I doing this right?

[Ian Tickle]

Hi James

What the Poisson distribution tells you is that if the true count is N then
the expectation and variance are also N.  That's not the same thing as
saying that for an observed count N the expectation and variance are N.
Consider all those cases where the observed count is exactly zero.  That
can arise from any number of true counts, though as you noted larger values
become increasingly unlikely.  However those true counts are all >= 0 which
means that the mean and variance of those true counts must be positive and
non-zero.  From your results they are both 1 though I haven't been through
the algebra to prove it.

So what you are saying seems correct: for N observed counts we should be
taking the best estimate of the true value and variance as N+1.  For
reasonably large N the difference is small but if you are concerned with
weak images it might start to become significant.

Cheers

-- Ian


On Tue, 12 Oct 2021 at 17:56, James Holton  wrote:

> All my life I have believed that if you're counting photons then the
> error of observing N counts is sqrt(N).  However, a calculation I just
> performed suggests its actually sqrt(N+1).
>
> My purpose here is to understand the weak-image limit of data
> processing. Question is: for a given pixel, if one photon is all you
> got, what do you "know"?
>
> I simulated millions of 1-second experiments. For each I used a "true"
> beam intensity (Itrue) between 0.001 and 20 photons/s. That is, for
> Itrue= 0.001 the average over a very long exposure would be 1 photon
> every 1000 seconds or so. For a 1-second exposure the observed count (N)
> is almost always zero. About 1 in 1000 of them will see one photon, and
> roughly 1 in a million will get N=2. I do 10,000 such experiments and
> put the results into a pile.  I then repeat with Itrue=0.002,
> Itrue=0.003, etc. All the way up to Itrue = 20. At Itrue > 20 I never
> see N=1, not even in 1e7 experiments. With Itrue=0, I also see no N=1
> events.
> Now I go through my pile of results and extract those with N=1, and
> count up the number of times a given Itrue produced such an event. The
> histogram of Itrue values in this subset is itself Poisson, but with
> mean = 2 ! If I similarly count up events where 2 and only 2 photons
> were seen, the mean Itrue is 3. And if I look at only zero-count events
> the mean and standard deviation is unity.
>
> Does that mean the error of observing N counts is really sqrt(N+1) ?
>
> I admit that this little exercise assumes that the distribution of Itrue
> is uniform between 0.001 and 20, but given that one photon has been
> observed Itrue values outside this range are highly unlikely. The
> Itrue=0.001 and N=1 events are only a tiny fraction of the whole.  So, I
> wold say that even if the prior distribution is not uniform, it is
> certainly bracketed. Now, Itrue=0 is possible if the shutter didn't
> open, but if the rest of the detector pixels have N=~1, doesn't this
> affect the prior distribution of Itrue on our pixel of interest?
>
> Of course, two or more photons are better than one, but these days with
> small crystals and big detectors N=1 is no longer a trivial situation.
> I look forward to hearing your take on this.  And no, this is not a trick.
>
> -James Holton
> MAD Scientist
>
> 
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> available at https://www.jiscmail.ac.uk/policyandsecurity/
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

[ccp4bb] am I doing this right?

[James Holton]

All my life I have believed that if you're counting photons then the 
error of observing N counts is sqrt(N).  However, a calculation I just 
performed suggests its actually sqrt(N+1).


My purpose here is to understand the weak-image limit of data 
processing. Question is: for a given pixel, if one photon is all you 
got, what do you "know"?


I simulated millions of 1-second experiments. For each I used a "true" 
beam intensity (Itrue) between 0.001 and 20 photons/s. That is, for 
Itrue= 0.001 the average over a very long exposure would be 1 photon 
every 1000 seconds or so. For a 1-second exposure the observed count (N) 
is almost always zero. About 1 in 1000 of them will see one photon, and 
roughly 1 in a million will get N=2. I do 10,000 such experiments and 
put the results into a pile.  I then repeat with Itrue=0.002, 
Itrue=0.003, etc. All the way up to Itrue = 20. At Itrue > 20 I never 
see N=1, not even in 1e7 experiments. With Itrue=0, I also see no N=1 
events.
Now I go through my pile of results and extract those with N=1, and 
count up the number of times a given Itrue produced such an event. The 
histogram of Itrue values in this subset is itself Poisson, but with 
mean = 2 ! If I similarly count up events where 2 and only 2 photons 
were seen, the mean Itrue is 3. And if I look at only zero-count events 
the mean and standard deviation is unity.


Does that mean the error of observing N counts is really sqrt(N+1) ?

I admit that this little exercise assumes that the distribution of Itrue 
is uniform between 0.001 and 20, but given that one photon has been 
observed Itrue values outside this range are highly unlikely. The 
Itrue=0.001 and N=1 events are only a tiny fraction of the whole.  So, I 
wold say that even if the prior distribution is not uniform, it is 
certainly bracketed. Now, Itrue=0 is possible if the shutter didn't 
open, but if the rest of the detector pixels have N=~1, doesn't this 
affect the prior distribution of Itrue on our pixel of interest?


Of course, two or more photons are better than one, but these days with 
small crystals and big detectors N=1 is no longer a trivial situation.  
I look forward to hearing your take on this.  And no, this is not a trick.


-James Holton
MAD Scientist



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/