Re: Software re-binned PD data

[Reinhard Kleeberg]

The problem of falsified counting statistics (deviating from Poisson 
distribution) sometimes arises even from (wrong) conversion of 0D 
detector data (typically when originally cps have been stored and the 
counting time got lost in the conversion), and is quite common when 1D 
detector data are exported to 3rd party formats and the number of active 
channels is not considered correctly in the export routine.


As a tool for a quick coarse check of the correct noise (like Matthew 
has suggested below), Nicola Doebelin has integrated a "noise" cursor in 
his PROFEX software

https://profex.doebelin.org/
simply showing the +-sqrt(n)  bars.
If the noise of a pattern significantly deviates from this interval, 
something was going wrong, either in instrumental data 
collection/pretreatment or during export or conversion. No big science, 
but very helpful to identify bad or manipulated data.


Reinhard


Am 27/09/2019 um 08:15 schrieb Matthew Rowles:

Hi Tony

If you want to have a look at what the uncertainties are doing, then 
try scanning over a peak a couple of dozen times (maybe with a few 
different mA settings on the tube, maybe with some different step 
times) to collect a range of different intensities. The standard 
deviation of the "raw" counts (not raw CPS) should approximately the 
square root of the number of counts. If it is different, then 
something squirrelly is going on.


Matthew

On Fri, 27 Sep 2019 at 13:46, iangie > wrote:


Dear Rietvelders,

Thanks for your opinions!
The "re-binning" of 1D data was done by my measurement software
automatically, rather than by analysis software.
The CPS is unchanged after its "re-binning". This means, rather
than adding counts of neighboring steps, it is *averaging* my data
(sum counts up then divided by the number of combined bins)!
I have a feeling what my measurement software doing is not correct...

--
*Dr. Xiaodong (Tony) Wang*
*Research Infrastructure Specialist (XRD)*
Central Analytical Research Facility (CARF)   |   Institute for
Future Environments
Queensland University of Technology
*
*

在 2019-09-27 10:31:45，alancoe...@bigpond.com
 写道：

Hi Tony

>My I ask is this re-bined data from the measurement software considered as 
"raw data"
or "treated data"?

I’m not sure what is meant by treated data. Almost all neutron
data and synchrotron data with area detectors are “treated data”.

If the detector has a slit width in the equatorial plane that
is 0.03 degrees 2Th then it makes little sense using a step
size that is less than 0.03/2 degrees 2Th. If rebinning is
done correctly (see rebin_with_dx_of in the Technical
Reference) then rebinning is basically collecting redoing the
experiment with a wider slit.

In the case of your PSD then the resolution of the PSD would
be the smallest slit width. If the data has broad features
relative to the slit width then rebinning (or using a bigger
slit width) should not change the results. You could simulate
all this using TOPAS to see the difference. Correct rebinning
should not affect parameter errors.

This is a question that is not simple to answer and if there’s
concern then:

 1. Simulating data with the small step size and performing a fit
 2. And then rebinning with various slit widths and then fitting
 3. And then comparing parameters errors and parameter values
for all the refinements should shine light on the area.

I don’t know where but I feeling is that there should be
papers on this.

Cheers

Alan

*From:*rietveld_l-requ...@ill.fr
 mailto:rietveld_l-requ...@ill.fr>> *On Behalf Of *iangie
*Sent:* Thursday, September 26, 2019 1:40 PM
*To:* rietveld_l@ill.fr 
*Subject:* Software re-binned PD data

Dear Rietvelder,

I hope you are doing well.

It is generally acknolwdged that Rietveld refinement should be
performed on raw data, without any data processing.
One of our diffractometer/PSD  scans data at its minimal step
size (users can see that the step size during scan is much
smaller than what was set), and upon finishing, the
measurement software re-bin the counts to the step size what
users set (so the data also looks smoother, after re-bin).
My I ask is this re-bined data from the measurement software
considered as "raw data" or "treated data"? And can we apply
Rietveld refinement on this data?

Any comments are welcome. :)

--

*Dr. Xiaodong (Tony) Wang*

*Research Infrastructure Specialist (XRD)*

Central Analytical Research 

Re: Software re-binned PD data

[Jonathan WRIGHT]

If this rule is broken, such as the mode in 
FIT2D that smooths patterns by pixel-splitting) then one introduces 
correlation between points in the pattern 


Hi Brian, your comment has triggered me it seems :-)

For SAXS and PDF it could be better to under-sample the data (see end), 
but for profile fitting I find this story is causing more headaches than 
it solves. Splitting pixels was (and still is?) a pragmatic way to 
address the following:


- To fit peak positions and widths for strain you need >3 points within 
the FWHM. Output bins may be required as smaller than pixel sizes.


- Detector pixels may fall halfway between two output bins, so which 
output bin should they go into? If a bunch of pixels "jump" from one bin 
to another suddenly, so do your results. Your colleague, with a tiny 
change in calibration, might get wildly different answers to you.


Fitting the (raw) 2D images also overcomes that but runs into other 
problems: what are the weights for pixels with zero counts? Binning 
makes it is easier to look at a fit with ~5 points within a FWHM 
compared to 5000. Note that pixel splitting convolutes the pixel (or 
bin) shapes into the final peak shape, so you come out with broader and 
smoother peaks that tend to be more robust to numerical issues.


Serial correlation in the final fit remains a problem whether you split 
pixels or not, from a long time ago, e.g.:


Berar and Lelann: J. Appl. Cryst. (1991). 24, 1-5
https://doi.org/10.1107/S0021889890008391

Note that pixels from some detectors do not arrive as statistically 
independent quantities anyway. When spatial corrections have been done 
for you the patterns are obvious, but less so when a single X-ray photon 
is spread over several CCD pixels.


Is someone aware of a Rietveld example where "not splitting is better" 
while keeping the same bin size? I see the point for statistical 
analysis of the noise in SAXS and PDF data, but I do not get it when 
fitting strong peaks where the signal is greater than the noise level 
anyway. The reality for a lot of 2D detector data is that you counted 
way too many photons. Systematic errors in the fit are much larger than 
the statistical noise anyway.


Best,

Jon
===

PS: The SAXS argument against splitting pixels is here:
"Correlation Map, a goodness-of-fit test for one-dimensional X-ray 
scattering spectra"

Daniel Franke, Cy M Jeffries & Dmitri I Svergun
Nature Methods 12, 419–422 (2015)
https://doi.org/10.1038/nmeth.3358

The PDF case is here:
"On the estimation of statistical uncertainties on powder diffraction 
and small-angle scattering data from two-dimensional X-ray detectors"

X. Yang, P. Juhás and S. J. L. Billinge
J. Appl. Cryst. (2014). 47, 1273-1283
https://doi.org/10.1107/S1600576714010516

++
Please do NOT attach files to the whole list 
Send commands to  eg: HELP as the subject with no body text
The Rietveld_L list archive is on http://www.mail-archive.com/rietveld_l@ill.fr/
++

Re: RE: Software re-binned PD data

[Alan Hewat]

Tony. If you want the statistics to be the same as for larger steps you
should be adding the counts from your PSD channels, not averaging them.
That said, with an x-ray PSD you perhaps have so much intensity that
uncertainties in your refined parameters due to counting statistics may be
small compared to systematic errors, so it may not be important that your
PSD averages adjacent channels instead of summing them. Perhaps the PSD
manufacturer chose averaging to make the data look better:-)
Alan.

On Fri, 27 Sep 2019 at 07:45, iangie  wrote:

> Dear Rietvelders,
>
> Thanks for your opinions!
> The "re-binning" of 1D data was done by my measurement software
> automatically, rather than by analysis software.
> The CPS is unchanged after its "re-binning". This means, rather than
> adding counts of neighboring steps, it is *averaging* my data (sum counts
> up then divided by the number of combined bins)!
> I have a feeling what my measurement software doing is not correct...
>
> --
> *Dr. Xiaodong (Tony) Wang*
> *Research Infrastructure Specialist (XRD)*
> Central Analytical Research Facility (CARF)   |   Institute for Future
> Environments
> Queensland University of Technology
>
>
> 在 2019-09-27 10:31:45，alancoe...@bigpond.com 写道：
>
> Hi Tony
>
>
>
> >My I ask is this re-bined data from the measurement software considered
> as "raw data" or "treated data"?
>
>
>
> I’m not sure what is meant by treated data. Almost all neutron data and
> synchrotron data with area detectors are “treated data”.
>
>
>
> If the detector has a slit width in the equatorial plane that is 0.03
> degrees 2Th then it makes little sense using a step size that is less than
> 0.03/2 degrees 2Th. If rebinning is done correctly (see rebin_with_dx_of in
> the Technical Reference) then rebinning is basically collecting redoing the
> experiment with a wider slit.
>
>
>
> In the case of your PSD then the resolution of the PSD would be the
> smallest slit width. If the data has broad features relative to the slit
> width then rebinning (or using a bigger slit width) should not change the
> results. You could simulate all this using TOPAS to see the difference.
> Correct rebinning should not affect parameter errors.
>
>
>
> This is a question that is not simple to answer and if there’s concern
> then:
>
>
>
>1. Simulating data with the small step size and performing a fit
>2. And then rebinning with various slit widths and then fitting
>3. And then comparing parameters errors and parameter values for all
>the refinements should shine light on the area.
>
>
>
> I don’t know where but I feeling is that there should be papers on this.
>
>
>
> Cheers
>
> Alan
>
>
>
>
>
> *From:* rietveld_l-requ...@ill.fr  *On Behalf
> Of *iangie
> *Sent:* Thursday, September 26, 2019 1:40 PM
> *To:* rietveld_l@ill.fr
> *Subject:* Software re-binned PD data
>
>
>
> Dear Rietvelder,
>
>
>
> I hope you are doing well.
>
> It is generally acknolwdged that Rietveld refinement should be performed
> on raw data, without any data processing.
> One of our diffractometer/PSD  scans data at its minimal step size (users
> can see that the step size during scan is much smaller than what was set),
> and upon finishing, the measurement software re-bin the counts to the step
> size what users set (so the data also looks smoother, after re-bin).
> My I ask is this re-bined data from the measurement software considered as
> "raw data" or "treated data"? And can we apply Rietveld refinement on this
> data?
>
>
>
> Any comments are welcome. :)
>
> --
>
> *Dr. Xiaodong (Tony) Wang*
>
> *Research Infrastructure Specialist (XRD)*
>
> Central Analytical Research Facility (CARF)   |   Institute for Future
> Environments
>
> Queensland University of Technology
>
> ++
> Please do NOT attach files to the whole list  >
> Send commands to  eg: HELP as the subject with no body
> text
> The Rietveld_L list archive is on
> http://www.mail-archive.com/rietveld_l@ill.fr/
> ++
>
>

-- 
__
*   Dr Alan Hewat, NeutronOptics, Grenoble, FRANCE *
 +33.476.98.41.68
http://www.NeutronOptics.com/hewat TVA FR 79499450856
__
++
Please do NOT attach files to the whole list 
Send commands to  eg: HELP as the subject with no body text
The Rietveld_L list archive is on http://www.mail-archive.com/rietveld_l@ill.fr/
++

Re: RE: Software re-binned PD data

[Matthew Rowles]

Hi Tony

If you want to have a look at what the uncertainties are doing, then try
scanning over a peak a couple of dozen times (maybe with a few different mA
settings on the tube, maybe with some different step times) to collect a
range of different intensities. The standard deviation of the "raw" counts
(not raw CPS) should approximately the square root of the number of counts.
If it is different, then something squirrelly is going on.

Matthew

On Fri, 27 Sep 2019 at 13:46, iangie  wrote:

> Dear Rietvelders,
>
> Thanks for your opinions!
> The "re-binning" of 1D data was done by my measurement software
> automatically, rather than by analysis software.
> The CPS is unchanged after its "re-binning". This means, rather than
> adding counts of neighboring steps, it is *averaging* my data (sum counts
> up then divided by the number of combined bins)!
> I have a feeling what my measurement software doing is not correct...
>
> --
> *Dr. Xiaodong (Tony) Wang*
> *Research Infrastructure Specialist (XRD)*
> Central Analytical Research Facility (CARF)   |   Institute for Future
> Environments
> Queensland University of Technology
>
>
> 在 2019-09-27 10:31:45，alancoe...@bigpond.com 写道：
>
> Hi Tony
>
>
>
> >My I ask is this re-bined data from the measurement software considered
> as "raw data" or "treated data"?
>
>
>
> I’m not sure what is meant by treated data. Almost all neutron data and
> synchrotron data with area detectors are “treated data”.
>
>
>
> If the detector has a slit width in the equatorial plane that is 0.03
> degrees 2Th then it makes little sense using a step size that is less than
> 0.03/2 degrees 2Th. If rebinning is done correctly (see rebin_with_dx_of in
> the Technical Reference) then rebinning is basically collecting redoing the
> experiment with a wider slit.
>
>
>
> In the case of your PSD then the resolution of the PSD would be the
> smallest slit width. If the data has broad features relative to the slit
> width then rebinning (or using a bigger slit width) should not change the
> results. You could simulate all this using TOPAS to see the difference.
> Correct rebinning should not affect parameter errors.
>
>
>
> This is a question that is not simple to answer and if there’s concern
> then:
>
>
>
>1. Simulating data with the small step size and performing a fit
>2. And then rebinning with various slit widths and then fitting
>3. And then comparing parameters errors and parameter values for all
>the refinements should shine light on the area.
>
>
>
> I don’t know where but I feeling is that there should be papers on this.
>
>
>
> Cheers
>
> Alan
>
>
>
>
>
> *From:* rietveld_l-requ...@ill.fr  *On Behalf
> Of *iangie
> *Sent:* Thursday, September 26, 2019 1:40 PM
> *To:* rietveld_l@ill.fr
> *Subject:* Software re-binned PD data
>
>
>
> Dear Rietvelder,
>
>
>
> I hope you are doing well.
>
> It is generally acknolwdged that Rietveld refinement should be performed
> on raw data, without any data processing.
> One of our diffractometer/PSD  scans data at its minimal step size (users
> can see that the step size during scan is much smaller than what was set),
> and upon finishing, the measurement software re-bin the counts to the step
> size what users set (so the data also looks smoother, after re-bin).
> My I ask is this re-bined data from the measurement software considered as
> "raw data" or "treated data"? And can we apply Rietveld refinement on this
> data?
>
>
>
> Any comments are welcome. :)
>
> --
>
> *Dr. Xiaodong (Tony) Wang*
>
> *Research Infrastructure Specialist (XRD)*
>
> Central Analytical Research Facility (CARF)   |   Institute for Future
> Environments
>
> Queensland University of Technology
>
> ++
> Please do NOT attach files to the whole list  >
> Send commands to  eg: HELP as the subject with no body
> text
> The Rietveld_L list archive is on
> http://www.mail-archive.com/rietveld_l@ill.fr/
> ++
>
>
++
Please do NOT attach files to the whole list 
Send commands to  eg: HELP as the subject with no body text
The Rietveld_L list archive is on http://www.mail-archive.com/rietveld_l@ill.fr/
++

RE: Software re-binned PD data

[alancoelho]

Hi Tony

 

>My I ask is this re-bined data from the measurement software considered as
"raw data" or "treated data"?

 

I'm not sure what is meant by treated data. Almost all neutron data and
synchrotron data with area detectors are "treated data".

 

If the detector has a slit width in the equatorial plane that is 0.03
degrees 2Th then it makes little sense using a step size that is less than
0.03/2 degrees 2Th. If rebinning is done correctly (see rebin_with_dx_of in
the Technical Reference) then rebinning is basically collecting redoing the
experiment with a wider slit.

 

In the case of your PSD then the resolution of the PSD would be the smallest
slit width. If the data has broad features relative to the slit width then
rebinning (or using a bigger slit width) should not change the results. You
could simulate all this using TOPAS to see the difference. Correct rebinning
should not affect parameter errors. 

 

This is a question that is not simple to answer and if there's concern then:

 

1.  Simulating data with the small step size and performing a fit
2.  And then rebinning with various slit widths and then fitting
3.  And then comparing parameters errors and parameter values for all
the refinements should shine light on the area.

 

I don't know where but I feeling is that there should be papers on this. 

 

Cheers

Alan

 

 

From: rietveld_l-requ...@ill.fr  On Behalf Of
iangie
Sent: Thursday, September 26, 2019 1:40 PM
To: rietveld_l@ill.fr
Subject: Software re-binned PD data

 

Dear Rietvelder,

 

I hope you are doing well.

It is generally acknolwdged that Rietveld refinement should be performed on
raw data, without any data processing. 
One of our diffractometer/PSD  scans data at its minimal step size (users
can see that the step size during scan is much smaller than what was set),
and upon finishing, the measurement software re-bin the counts to the step
size what users set (so the data also looks smoother, after re-bin).  
My I ask is this re-bined data from the measurement software considered as
"raw data" or "treated data"? And can we apply Rietveld refinement on this
data?

 

Any comments are welcome. :)

--

Dr. Xiaodong (Tony) Wang

Research Infrastructure Specialist (XRD)

Central Analytical Research Facility (CARF)   |   Institute for Future
Environments

Queensland University of Technology

++
Please do NOT attach files to the whole list 
Send commands to  eg: HELP as the subject with no body text
The Rietveld_L list archive is on http://www.mail-archive.com/rietveld_l@ill.fr/
++

Re: Software re-binned PD data

[Toby, Brian H.]

Tony,

These days many if not most powder instruments (~100% of those at 
synchrotron and neutron sources) do some sort of data processing to take raw 
measurements and convert them to diffractograms that are used in Rietveld. The 
key for how to know if this is done properly is to ask if any experimental 
measurement ends up in being used in more than one bin in the final 
diffractogram. If this rule is broken, such as the mode in FIT2D that smooths 
patterns by pixel-splitting) then one introduces correlation between points in 
the pattern and the statistically expected R becomes overestimated (and reduced 
chi-squared values can drop below 1). That could be handled in theory (using a 
weight matrix in construction of the Hessian), but no Rietveld code does.

Brian



On Sep 25, 2019, at 10:40 PM, iangie mailto:ian...@126.com>> 
wrote:

Dear Rietvelder,

I hope you are doing well.
It is generally acknolwdged that Rietveld refinement should be performed on raw 
data, without any data processing.
One of our diffractometer/PSD  scans data at its minimal step size (users can 
see that the step size during scan is much smaller than what was set), and upon 
finishing, the measurement software re-bin the counts to the step size what 
users set (so the data also looks smoother, after re-bin).
My I ask is this re-bined data from the measurement software considered as "raw 
data" or "treated data"? And can we apply Rietveld refinement on this data?

Any comments are welcome. :)

--
Dr. Xiaodong (Tony) Wang
Research Infrastructure Specialist (XRD)
Central Analytical Research Facility (CARF)   |   Institute for Future 
Environments
Queensland University of Technology
++
Please do NOT attach files to the whole list 
mailto:alan.he...@neutronoptics.com>>
Send commands to mailto:lists...@ill.fr>> eg: HELP as the 
subject with no body text
The Rietveld_L list archive is on http://www.mail-archive.com/rietveld_l@ill.fr/
++


++
Please do NOT attach files to the whole list 
Send commands to  eg: HELP as the subject with no body text
The Rietveld_L list archive is on http://www.mail-archive.com/rietveld_l@ill.fr/
++

Re: Software re-binned PD data

[Daniel M . Többens]

Hello,

the term "raw data" has become kind of fuzzy in recent times. Most 
powder diffractometers nowadays are equipped with some kind of linear or 
area detector. The 1D powder pattern in this case is always the result 
of some integration, merging, and binning.


I would take "raw" to mean that major corrections like absorption or 
Lorentz-Polarisation should not be applied beforehand. In particular, 
removing the background was done in old times quite a lot, making it 
hard to judge the real quality of the data.


Yours,
Daniel Többens

Am 26.09.2019 um 05:40 schrieb iangie:

Dear Rietvelder,

I hope you are doing well.
It is generally acknolwdged that Rietveld refinement should be 
performed on raw data, without any data processing.
One of our diffractometer/PSD  scans data at its minimal step size 
(users can see that the step size during scan is much smaller than 
what was set), and upon finishing, the measurement software re-bin the 
counts to the step size what users set (so the data also looks 
smoother, after re-bin).
My I ask is this re-bined data from the measurement software 
considered as "raw data" or "treated data"? And can we apply Rietveld 
refinement on this data?


Any comments are welcome. :)

--
*Dr. Xiaodong (Tony) Wang*
*Research Infrastructure Specialist (XRD)*
Central Analytical Research Facility (CARF)   |   Institute for Future 
Environments

Queensland University of Technology

++
Please do NOT attach files to the whole list 
Send commands to  eg: HELP as the subject with no body text
The Rietveld_L list archive is on http://www.mail-archive.com/rietveld_l@ill.fr/
++




--
Helmholtz-Zentrum Berlin für Materialien und Energie GmbH

Mitglied der Hermann von Helmholtz-Gemeinschaft Deutscher Forschungszentren e.V.

Aufsichtsrat: Vorsitzender Dr. Volkmar Dietz, stv. Vorsitzende Dr. Jutta 
Koch-Unterseher
Geschäftsführung: Prof. Dr. Bernd Rech (Sprecher), Prof. Dr. Jan Lüning, Thomas 
Frederking

Sitz Berlin, AG Charlottenburg, 89 HRB 5583

Postadresse:
Hahn-Meitner-Platz 1
D-14109 Berlin



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Software re-binned PD data

[Alan Hewat]

Tony, if you bin (add) the raw data you can perform Rietveld refinement as
if you had taken larger steps.
Regards, Alan.

On Thu, 26 Sep 2019 at 05:42, iangie  wrote:

> Dear Rietvelder,
>
> I hope you are doing well.
> It is generally acknolwdged that Rietveld refinement should be performed
> on raw data, without any data processing.
> One of our diffractometer/PSD  scans data at its minimal step size (users
> can see that the step size during scan is much smaller than what was set),
> and upon finishing, the measurement software re-bin the counts to the step
> size what users set (so the data also looks smoother, after re-bin).
> My I ask is this re-bined data from the measurement software considered as
> "raw data" or "treated data"? And can we apply Rietveld refinement on this
> data?
>
> Any comments are welcome. :)
>
> --
> *Dr. Xiaodong (Tony) Wang*
> *Research Infrastructure Specialist (XRD)*
> Central Analytical Research Facility (CARF)   |   Institute for Future
> Environments
> Queensland University of Technology
> ++
> Please do NOT attach files to the whole list  >
> Send commands to  eg: HELP as the subject with no body
> text
> The Rietveld_L list archive is on
> http://www.mail-archive.com/rietveld_l@ill.fr/
> ++
>
>

-- 
__
*   Dr Alan Hewat, NeutronOptics, Grenoble, FRANCE *
 +33.476.98.41.68
http://www.NeutronOptics.com/hewat TVA FR 79499450856
__
++
Please do NOT attach files to the whole list 
Send commands to  eg: HELP as the subject with no body text
The Rietveld_L list archive is on http://www.mail-archive.com/rietveld_l@ill.fr/
++