Re: [Ifeffit] doubt on number of variables

[Jesús Eduardo Vega Castillo]

Thank you very much Bruce and Scott,

Your answers are really helpful.

2016-05-26 11:28 GMT-03:00 Bruce Ravel :

> On 05/26/2016 10:02 AM, Jesús Eduardo Vega Castillo wrote:
>
>> Just for clarifying, this is not what I have been doing. I just was
>> asked to add new independent variables (individual DW factors for each
>> path) while I am already at the limit of independent points.
>>
>> I have been using a number of parameter always lower than the number of
>> independent points. I have also been managing high correlations by not
>> varying two strongly correlated parameters at the time. But at the end I
>> do a final fit setting to guess all the parameters, no matter the
>> correlations, in order to obtain a "true" reduced Chi2 to report which
>> includes all parameters. Do you consider this procedure right?
>>
>
> Well, "truth" is a difficult concept and isn't really what EXAFS analysis
> is all about.
>
> When we do an EXAFS analysis we are testing the extent to which data are
> consistent with a fitting model.  The goal is to find a fitting model that
> is a defensible description of the data and which yields fitting results
> and uncertainties that are, themselves, defensible and which tell us
> something that we want to know about the sample which was measured.
>
> Some parameters are highly correlated and there is nothing you can do
> about it, regardless of how much you might want to.  In a one-shell fit,
> S02 and coordination number are 100% correlated.  It doesn't matter how
> much your thesis advisor wants you to get the "true" coordination number
> from the one-shell fit, it won't happen.
>
> So, in the end, you settle upon a fitting model and write a paper.  In
> that paper you report on the fitting results.  If the referee asks why you
> chose to fix a certain parameter, you better have a good reason. That's
> what I mean by "defensible" -- that you can explain and justify all the
> choices you made when fitting the data.
>
> The "procedure" is that you define a fitting model, press the big Fit
> button, and interpret the results.  The parameters that you float, the
> parameters that you fix ... they are defensible when you can defend them.
> "Because the boss wanted two more sigma^2 parameters" is probably not a
> defensible argument.
>
> Here are a couple of my papers:
>
> http://dx.doi.org/10.1016/j.radphyschem.2009.05.024
> http://dx.doi.org/10.1107/S1600577514014982
>
> I refer you to these not because they are necessarily relevant to your
> work or even because they are especially good papers.  But, in each, I
> discuss defensibility of fitting model in a situation where there is not
> nearly enough information in the measured data to properly describe the
> actual structure of the sample.  Perhaps you'll find it helpful to see how
> I address this issue.
>
>
> B
>
> --
>  Bruce Ravel   bra...@bnl.gov
>
>  National Institute of Standards and Technology
>  Synchrotron Science Group at NSLS-II
>  Building 535A
>  Upton NY, 11973
>
>  Homepage:http://bruceravel.github.io/home/
>  Software:https://github.com/bruceravel
>  Demeter: http://bruceravel.github.io/demeter/
> ___
> Ifeffit mailing list
> Ifeffit@millenia.cars.aps.anl.gov
> http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
> Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
>
___
Ifeffit mailing list
Ifeffit@millenia.cars.aps.anl.gov
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

Re: [Ifeffit] doubt on number of variables

[Bruce Ravel]

On 05/26/2016 10:02 AM, Jesús Eduardo Vega Castillo wrote:

Just for clarifying, this is not what I have been doing. I just was
asked to add new independent variables (individual DW factors for each
path) while I am already at the limit of independent points.

I have been using a number of parameter always lower than the number of
independent points. I have also been managing high correlations by not
varying two strongly correlated parameters at the time. But at the end I
do a final fit setting to guess all the parameters, no matter the
correlations, in order to obtain a "true" reduced Chi2 to report which
includes all parameters. Do you consider this procedure right?


Well, "truth" is a difficult concept and isn't really what EXAFS 
analysis is all about.


When we do an EXAFS analysis we are testing the extent to which data are 
consistent with a fitting model.  The goal is to find a fitting model 
that is a defensible description of the data and which yields fitting 
results and uncertainties that are, themselves, defensible and which 
tell us something that we want to know about the sample which was measured.


Some parameters are highly correlated and there is nothing you can do 
about it, regardless of how much you might want to.  In a one-shell fit, 
S02 and coordination number are 100% correlated.  It doesn't matter how 
much your thesis advisor wants you to get the "true" coordination number 
from the one-shell fit, it won't happen.


So, in the end, you settle upon a fitting model and write a paper.  In 
that paper you report on the fitting results.  If the referee asks why 
you chose to fix a certain parameter, you better have a good reason. 
That's what I mean by "defensible" -- that you can explain and justify 
all the choices you made when fitting the data.


The "procedure" is that you define a fitting model, press the big Fit 
button, and interpret the results.  The parameters that you float, the 
parameters that you fix ... they are defensible when you can defend 
them.  "Because the boss wanted two more sigma^2 parameters" is probably 
not a defensible argument.


Here are a couple of my papers:

http://dx.doi.org/10.1016/j.radphyschem.2009.05.024
http://dx.doi.org/10.1107/S1600577514014982

I refer you to these not because they are necessarily relevant to your 
work or even because they are especially good papers.  But, in each, I 
discuss defensibility of fitting model in a situation where there is not 
nearly enough information in the measured data to properly describe the 
actual structure of the sample.  Perhaps you'll find it helpful to see 
how I address this issue.


B

--
 Bruce Ravel   bra...@bnl.gov

 National Institute of Standards and Technology
 Synchrotron Science Group at NSLS-II
 Building 535A
 Upton NY, 11973

 Homepage:http://bruceravel.github.io/home/
 Software:https://github.com/bruceravel
 Demeter: http://bruceravel.github.io/demeter/
___
Ifeffit mailing list
Ifeffit@millenia.cars.aps.anl.gov
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

Re: [Ifeffit] doubt on number of variables

[Scott Calvin]

Hi Jesus,

Yes, that procedure is fine. As long as the final, reported, fit guesses 
everything which is varied, whatever you do with earlier fits amounts to 
exploration and diagnostics.


On May 26, 2016, at 10:02 AM, Jesús Eduardo Vega Castillo 
> wrote:

Thanks Bruce for your answer,

Just for clarifying, this is not what I have been doing. I just was asked to 
add new independent variables (individual DW factors for each path) while I am 
already at the limit of independent points.

I have been using a number of parameter always lower than the number of 
independent points. I have also been managing high correlations by not varying 
two strongly correlated parameters at the time. But at the end I do a final fit 
setting to guess all the parameters, no matter the correlations, in order to 
obtain a "true" reduced Chi2 to report which includes all parameters. Do you 
consider this procedure right?








2016-05-26 10:36 GMT-03:00 Bruce Ravel >:
On 05/26/2016 09:17 AM, Jesús Eduardo Vega Castillo wrote:
As far as I understand the number of variables refers to those
parameters which have been "guessed" in the fit and it is limited by the
number of independent points.

Correct.

If I have a group of variables but only set to guess a few of them while
fixing the others, then fix some of the obtained values and guess other
variables; I might include a large number of parameters in the resulting
fit (even more than the independent points) without increasing the
Number of variables of each individual fit. This also would keep reduced
chi2 from increasing too much.

Would this be cheating?

The answer, almost certainly, is "yes".

The total number of parameters included is always restricted by the
Number of independent points?

I'm going with this one ...

or
Can I include all the parameters I want as long as I set to guess no
more that the number of independent points at the time?

... and I really don't like this one.


The thing you seem not to be considering is that parameters have correlations.  
The correlations between parameters are an important part of the assessment of 
uncertainty.

If you artificially suppress correlations between parameters in the way that 
you are describing, it is unlikely that the fitted results and their 
uncertainties would be defensible.

While it is possible to probe correlations between parameters in a defensible 
manner by selectively setting and guessing parameters in a lengthy sequence of 
fits, your brief description does not give me confidence that that is what you 
are doing.

The trick to EXAFS analysis (and any non-linear minimization analysis) is to 
defensibly get what information you can from the data.  Rarely do we get 
everything we want.  In those situations where we aspire to  more than our data 
can provide, it's important to remember that something is better than nothing.

B

--
 Bruce Ravel   
bra...@bnl.gov

 National Institute of Standards and Technology
 Synchrotron Science Group at NSLS-II
 Building 535A
 Upton NY, 11973

 Homepage:http://bruceravel.github.io/home/
 Software:https://github.com/bruceravel
 Demeter: http://bruceravel.github.io/demeter/
___
Ifeffit mailing list
Ifeffit@millenia.cars.aps.anl.gov
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

___
Ifeffit mailing list
Ifeffit@millenia.cars.aps.anl.gov
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

___
Ifeffit mailing list
Ifeffit@millenia.cars.aps.anl.gov
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

Re: [Ifeffit] doubt on number of variables

[Jesús Eduardo Vega Castillo]

Thanks Bruce for your answer,

Just for clarifying, this is not what I have been doing. I just was asked
to add new independent variables (individual DW factors for each path)
while I am already at the limit of independent points.

I have been using a number of parameter always lower than the number of
independent points. I have also been managing high correlations by not
varying two strongly correlated parameters at the time. But at the end I do
a final fit setting to guess all the parameters, no matter the
correlations, in order to obtain a "true" reduced Chi2 to report which
includes all parameters. Do you consider this procedure right?








2016-05-26 10:36 GMT-03:00 Bruce Ravel :

> On 05/26/2016 09:17 AM, Jesús Eduardo Vega Castillo wrote:
>
>> As far as I understand the number of variables refers to those
>> parameters which have been "guessed" in the fit and it is limited by the
>> number of independent points.
>>
>
> Correct.
>
> If I have a group of variables but only set to guess a few of them while
>> fixing the others, then fix some of the obtained values and guess other
>> variables; I might include a large number of parameters in the resulting
>> fit (even more than the independent points) without increasing the
>> Number of variables of each individual fit. This also would keep reduced
>> chi2 from increasing too much.
>>
>> Would this be cheating?
>>
>
> The answer, almost certainly, is "yes".
>
> The total number of parameters included is always restricted by the
>> Number of independent points?
>>
>
> I'm going with this one ...
>
> or
>> Can I include all the parameters I want as long as I set to guess no
>> more that the number of independent points at the time?
>>
>
> ... and I really don't like this one.
>
>
> The thing you seem not to be considering is that parameters have
> correlations.  The correlations between parameters are an important part of
> the assessment of uncertainty.
>
> If you artificially suppress correlations between parameters in the way
> that you are describing, it is unlikely that the fitted results and their
> uncertainties would be defensible.
>
> While it is possible to probe correlations between parameters in a
> defensible manner by selectively setting and guessing parameters in a
> lengthy sequence of fits, your brief description does not give me
> confidence that that is what you are doing.
>
> The trick to EXAFS analysis (and any non-linear minimization analysis) is
> to defensibly get what information you can from the data.  Rarely do we get
> everything we want.  In those situations where we aspire to  more than our
> data can provide, it's important to remember that something is better than
> nothing.
>
> B
>
> --
>  Bruce Ravel   bra...@bnl.gov
>
>  National Institute of Standards and Technology
>  Synchrotron Science Group at NSLS-II
>  Building 535A
>  Upton NY, 11973
>
>  Homepage:http://bruceravel.github.io/home/
>  Software:https://github.com/bruceravel
>  Demeter: http://bruceravel.github.io/demeter/
> ___
> Ifeffit mailing list
> Ifeffit@millenia.cars.aps.anl.gov
> http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
> Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit
>
___
Ifeffit mailing list
Ifeffit@millenia.cars.aps.anl.gov
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit