Re: [ccp4bb] am I doing this right?

t;> >> >> Gergely Katona, Professor, Chairman of the Chemistry Program Council >> >> Department of Chemistry and Molecular Biology, University of Gothenburg >> >> Box 462, 40530 Göteborg, Sweden >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fa

Re: [ccp4bb] am I doing this right?

priate probability distributions > and priors. Also, have we got clarity about the question being asked? > > > > Thanks to all for the interesting points. > > > > Colin > > *From:* CCP4 bulletin board *On Behalf Of *Randy > John Read > *Sent:* 21 October 2021 13:2

Re: [ccp4bb] am I doing this right?

, Email: gergely.kat...@gu.se From: CCP4 bulletin board On Behalf Of Nave, Colin (DLSLtd,RAL,LSCI) Sent: 21 October, 2021 19:21 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right? Congratulations to James for starting this interesting discussion. For those who are like me

Re: [ccp4bb] am I doing this right?

, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: gergely.kat...@gu.se From: CCP4 bulletin board On Behalf Of Nave, Colin (DLSLtd,RAL,LSCI) Sent: 21 October, 2021 19:21 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right

Re: [ccp4bb] am I doing this right?

for the interesting points. Colin From: CCP4 bulletin board On Behalf Of Randy John Read Sent: 21 October 2021 13:23 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right? Hi Kay, No, I still think the answer should come out the same if you have good reason to believe that all

Re: [ccp4bb] am I doing this right?

I am sorry, this was a dead-end idea multinomial distribution with 0 trials is not defined. Gergely From: CCP4 bulletin board On Behalf Of Gergely Katona Sent: 21 October, 2021 15:29 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right? Dear Randy and Kay, My solution would

Re: [ccp4bb] am I doing this right?

a black belt in statistics so I am not sure what the closed form of the posterior probabilities would be (if such exist). Best wishes, Gergely From: CCP4 bulletin board On Behalf Of Randy John Read Sent: 21 October, 2021 14:23 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing

Re: [ccp4bb] am I doing this right?

Hi Kay, No, I still think the answer should come out the same if you have good reason to believe that all the 100 pixels are equally likely to receive a photon (for instance because your knowledge of the geometry of the source and the detector says the difference in their positions is

Re: [ccp4bb] am I doing this right?

Randy, I must admit that I am not certain about my answer, but I lean toward thinking that the result (of the two thought experiments that you describe) is not the same. I do agree that it makes sense that the expectation value is the same, and the math that I sketched in

Re: [ccp4bb] am I doing this right?

Just to be a bit clearer, I mean that the calculation of the expected value and its variance should give the same answer if you're comparing one pixel for a particular length of exposure with the sum obtained from either a larger number of smaller pixels covering the same area for the same

Re: [ccp4bb] am I doing this right?

I would think that if this problem is being approached correctly, with the right prior, it shouldn't matter whether you collect the same signal distributed over 100 smaller pixels or the same pixel measured for the same length of exposure but with 100 time slices; you should get the same

Re: [ccp4bb] am I doing this right?

Hi Ian, it is Iobs=0.01 and sigIobs=0.01 for one pixel, but adding 100 pixels each with variance=sigIobs^2=0.0001 gives 0.01 , yielding a 100-pixel-sigIobs of 0.1 - different from the 1 you get. As if repeatedly observing the same count of 0 lowers the estimated error by sqrt(n), where n is

Re: [ccp4bb] am I doing this right?

Hi Kay Can I just confirm that your result Iobs=0.01 sigIobs=0.01 is the estimate of the true average intensity *per pixel* for a patch of 100 pixels? So then the total count for all 100 pixels is 1 with variance also 1, or in general for k observed counts in the patch, expectation = variance =

Re: [ccp4bb] am I doing this right?

ober, 2021 21:41 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right? Thank you very much for this Kay! So, to summarize, you are saying the answer to my question "what is the expectation and variance if I observe a 10x10 patch of pixels with zero counts?" is: Iobs = 0.

Re: [ccp4bb] am I doing this right?

James, I am saying that my answer to "what is the expectation and variance if I observe a 10x10 patch of pixels with zero counts?" is Iobs=0.01 sigIobs=0.01 (and Iobs=sigIobs=1 if there is only one pixel) IF the uniform prior applies. I agree with Gergely and others that this prior (with its

Re: [ccp4bb] am I doing this right?

e -Original Message- From: CCP4 bulletin board On Behalf Of James Holton Sent: 18 October, 2021 21:41 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right? Thank you very much for this Kay! So, to summarize, you are saying the answer to my question "what is the expect

Re: [ccp4bb] am I doing this right?

the question so important? I've understood what brought it up, if course, but not the consequence of getting it wrong. Frank Sent from tiny silly touch screen *From:* James Holton *Sent:* Saturday, 16 October 2021 20:01 *To

Re: [ccp4bb] am I doing this right?

I am not qualified to comment on anything in the rest of this discussion, but regarding the excerpt quoted below: this way of recording an "image" seems very similar to the EER (electron event representation) of the Falcon 4 direct electron detector used in cryoEM. See

Re: [ccp4bb] am I doing this right?

rought it up, if course, but not the consequence of getting it wrong. Frank Sent from tiny silly touch screen *From:* James Holton *Sent:* Saturday, 16 October 2021 20:01 *To:* CCP4BB@JISCMAIL.AC.UK *Subject:* Re: [cc

Re: [ccp4bb] am I doing this right?

Thank you very much for this Kay! So, to summarize, you are saying the answer to my question "what is the expectation and variance if I observe a 10x10 patch of pixels with zero counts?" is: Iobs = 0.01 sigIobs = 0.01 (defining sigIobs = sqrt(variance(Iobs))) And for the one-pixel case:

Re: [ccp4bb] am I doing this right?

All, this was my reply to one of James' emails which I just noticed was to me only, and I ought to have CC'd it to the BB, since it has relevance to others' contributions to the discussion. Cheers -- Ian On Mon, 18 Oct 2021 at 11:27, Ian Tickle wrote: > > James, no I don't think so, what

Re: [ccp4bb] am I doing this right?

Hi James, I'm a bit behind ... My answer about the basic question ("a patch of 100 pixels each with zero counts - what is the variance?") you ask is the following: 1) we all know the Poisson PDF (Probability Distribution Function) P(k|l) = l^k*e^(-l)/k! (where k stands for for an integer

Re: [ccp4bb] am I doing this right?

-- *From:* James Holton *Sent:* Saturday, 16 October 2021 20:01 *To:* CCP4BB@JISCMAIL.AC.UK *Subject:* Re: [ccp4bb] am I doing this right? Thank you everyone for your thoughtful and thought-provoking responses! But, I am starting to think I was not as c

Re: [ccp4bb] am I doing this right?

of frequentist thinking and reduce data one step at the time. Best wishes, Gergely -Original Message- From: James Holton Sent: 17 October, 2021 19:25 To: Gergely Katona ; CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right? Thank you Gergely.  That is interesting! I

Re: [ccp4bb] am I doing this right?

: Re: [ccp4bb] am I doing this right? Thank you Gergely.  That is interesting! I don't mind at all making this Bayesian, as long as it works! Something I'm not quite sure about: does the prior distribution HAVE to be a gamma distribution? Not that that really narrows things down since

Re: [ccp4bb] am I doing this right?

-Original Message- From: CCP4 bulletin board On Behalf Of James Holton Sent: den 16 oktober 2021 21:01 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right? Thank you everyone for your thoughtful and thought-provoking responses! But, I am starting to think I was n

Re: [ccp4bb] am I doing this right?

erspective then this will not help, unfortunately. Best wishes, Gergely -Original Message- From: CCP4 bulletin board On Behalf Of James Holton Sent: den 16 oktober 2021 21:01 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right? Thank you everyone for your thoughtful a

Re: [ccp4bb] am I doing this right?

Saturday, 16 October 2021 20:01 *To:* CCP4BB@JISCMAIL.AC.UK *Subject:* Re: [ccp4bb] am I doing this right? Thank you everyone for your thoughtful and thought-provoking responses! But, I am starting to think I was not as clear as I could have been about my question.  I am actually concerning myse

Re: [ccp4bb] am I doing this right?

If you write/analyze/improve a data-reduction program in crystallography, then these questions are important: how to estimate the intensity of a Bragg spot by evaluating the counts in the pixels of the signal area, and those of the background area? When calculating mean and standard deviation

Re: [ccp4bb] am I doing this right?

Sent from tiny silly touch screen<http://www.9folders.com/> From: James Holton Sent: Saturday, 16 October 2021 20:01 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right? Thank you everyone for your thoughtful and thought-provoking responses!

Re: [ccp4bb] am I doing this right?

are looking for an alternative to the Bayesian perspective then this will not help, unfortunately. Best wishes, Gergely -Original Message- From: CCP4 bulletin board On Behalf Of James Holton Sent: den 16 oktober 2021 21:01 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right

Re: [ccp4bb] am I doing this right?

: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: gergely.kat...@gu.se From: CCP4 bulletin board On Behalf Of James Holton Sent: 15 October, 2021 18:06 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right? Well I'll be... Kay Dieder

Re: [ccp4bb] am I doing this right?

>Box 462, 40530 Göteborg, Sweden >Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >Web: http://katonalab.eu, Email: gergely.kat...@gu.se > >From: CCP4 bulletin board On Behalf Of James >Holton >Sent: 15 October, 2021 18:06 >To: CCP4BB@JISCMAIL.AC.UK >

Re: [ccp4bb] am I doing this right?

lecular Biology, University of Gothenburg >> >Box 462, 40530 Göteborg, Sweden >> >Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> >Web: http://katonalab.eu, Email: gergely.kat...@gu.se >> > >> >From: CCP4 bulletin board On Be

Re: [ccp4bb] am I doing this right?

d Molecular Biology, University of Gothenburg > >Box 462, 40530 Göteborg, Sweden > >Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 > >Web: http://katonalab.eu, Email: gergely.kat...@gu.se > > > >From: CCP4 bulletin board On Behalf Of James > Holton

Re: [ccp4bb] am I doing this right?

Biology, University of Gothenburg >Box 462, 40530 Göteborg, Sweden >Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >Web: http://katonalab.eu, Email: gergely.kat...@gu.se > >From: CCP4 bulletin board On Behalf Of James Holton >Sent: 15 October, 2021 18:06 >

Re: [ccp4bb] am I doing this right?

/ M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: gergely.kat...@gu.se From: CCP4 bulletin board On Behalf Of James Holton Sent: 15 October, 2021 18:06 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] am I doing this right? Well I'll be... Kay Diederichs pointed out

Re: [ccp4bb] am I doing this right?

Well I'll be... Kay Diederichs pointed out to me off-list that the k+1 expectation and variance from observing k photons is in "Bayesian Reasoning in Data Analysis: A Critical Introduction" by Giulio D. Agostini. Granted, that is with a uniform prior, which I take as the Bayesean equivalent

Re: [ccp4bb] am I doing this right?

I forgot to add probably the most important. James is correct, the expected value of u, the true mean, given a single observation k is indeed k+1 and k+1 is also the mean square error of using k+1 as the estimator of the true mean. Cheers, Filipe On Wed, 13 Oct 2021 at 23:17, Filipe Maia wrote:

Re: [ccp4bb] am I doing this right?

Hi, The maximum likelihood estimator for a Poisson distributed variable is equal to the mean of the observations. In the case of a single observation, it will be equal to that observation. As Graeme suggested, you can calculate the probability mass function for a given observation with different

Re: [ccp4bb] am I doing this right?

This rang a bell to me last night, and I think you can derive this from first principles If you assume an observation of N counts, you can calculate the probability of such an observation for a given Poisson rate constant X. If you then integrate over all possible value of X to work out the

Re: [ccp4bb] am I doing this right?

Hi Ian, James, I have a strong feeling that I have seen this result before, and it was due to Andy Hammersley at ESRF. I’ve done a literature search and there is a paper relating to errors in analysis of counting statistics (se below), but I had a quick look at this and

Re: [ccp4bb] am I doing this right?

Hi James What the Poisson distribution tells you is that if the true count is N then the expectation and variance are also N. That's not the same thing as saying that for an observed count N the expectation and variance are N. Consider all those cases where the observed count is exactly zero.

[ccp4bb] am I doing this right?

All my life I have believed that if you're counting photons then the error of observing N counts is sqrt(N).  However, a calculation I just performed suggests its actually sqrt(N+1). My purpose here is to understand the weak-image limit of data processing. Question is: for a given pixel, if