Re: [NMusers] Obtaining RSE%

[Santosh]

Hi Ken

Many thanks for the insights on the standard errors & their
interpretations.  Yes, a careful analysis is required to interpret SE’s of
transformed space in untransformed space.

Best regards
Santosh

On Mon, Jul 29, 2024 at 4:43 PM  wrote:

> Hi Santosh,
>
>
>
> It’s important to note the distinction between transformations of the
> parameters and transformations of the data such as the
> log-transform-both-sides approach for a PK model to assume the residual
> errors are log-normally distributed.  Here we are specifically focusing on
> transformations of the parameters not the data.  Note that the likelihood
> is invariant to transformations of the parameters, so you will get the same
> fit and OFV whether you estimate log(CL) or CL as your theta.  However,
> Wald-based standard errors are very much dependent on the parameter
> transformation.
>
>
>
> For example, suppose we estimate the typical value of CL as THETA(1).
> Assuming the maximum likelihood estimate of THETA(1) is asymptotically
> normal then we could construct a confidence interval to reflect the
> uncertainty in that parameter estimate as
>
>
>
> theta1 +/- Zalpha(SE_theta1)
>
>
>
> where SE_theta1 is the Wald-based SE for theta1 and Zalpha is the
> two-sided critical value of the standard normal distribution to obtain a
> 100x(1 – alpha)% confidence interval.  Note that this confidence interval
> is symmetric about the estimate theta1.
>
>
>
> Now consider a log-transformation of CL such that THETA(1) corresponds to
> log(CL).  The Wald-based confidence interval for log(CL) would now be
>
>
>
> theta1 +/- Zalpha(SE_theta1)
>
>
>
> which is symmetric in the log(CL) scale.  However, the corresponding
> confidence interval for CL requires exponentiating the endpoints of the
> log(CL) confidence interval to obtain the confidence interval in the
> original CL scale.  That is,
>
>
>
> exp(theta1 +/- Zalpha(SE_theta1) )
>
>
>
> which will be asymmetric about the estimate of the typical value of CL,
> exp(theta1).
>
>
>
> When the parameter estimate space is highly asymmetric, transformations
> can help with this asymmetry so that the transformed estimates are more
> likely to be symmetric and normally distributed.  So, to answer your
> question, the precision of the estimates may still be valid, but we need to
> recognize that the uncertainty in the estimates may be asymmetric in the
> untransformed (original) space.
>
>
>
> Best,
>
>
>
> Ken
>
>
>
> *From:* owner-nmus...@globomaxnm.com  *On
> Behalf Of *Santosh
> *Sent:* Monday, July 29, 2024 1:11 PM
> *To:* nmusers@globomaxnm.com
> *Subject:* Re: [NMusers] Obtaining RSE%
>
>
>
> Dear Prof Holford, Ken, Alan, Jeroen & others,
>
>
>
> Thanks for the engaging discussions.
>
>
>
> In context of monitoring at the iteration level, I vaguely recall that in
> NMUSERS or in one of ACOP conferences , there was a presentation &
> demonstration with R scripts on looking at the convergence and other
> parameters in real time.
>
>
>
> The interpretations of SEs is interesting based on linear or non-linear
> models, and also based on size of variance of parameters.
>
>
>
> On a different note, I am also interested in hearing from you about SEs
> when estimated based on transformed distribution space and their values &
> interpretations in back-transformed space. Would the notion of precision
>  still be valid  when viewing both transformed and untransformed space?
> This is in context of dealing with untransformed space of non-normal or
> non-lognormal distributions.
>
>
>
> Best regards
>
> Santosh
>
>
>
>
>
> On Mon, Jul 29, 2024 at 8:52 AM Nick Holford 
> wrote:
>
> Hi Jeroen,
>
> A small correction. Please re-read my email to nmusers on 12 Feb 2015
> which I quote here. Sorry I cannot show the original but the 1999 URL is
> not available to me anymore.
>
> =  start quote ===
> Nick Holford Thu, 12 Feb 2015 11:54:59 -0800
> Hi,
> The original quote about electrons comes from a remark I made in 1999 on
> nmusers.
> http://www.cognigencorp.com/nonmem/nm/99nov121999.html
> Lewis Sheiner agreed in the same thread. Thanks to the wonders of living
> on a sphere Lewis appears to agree with me the day before I made the
> comment :-)
> =  end quote ===
>
> I had been meaning to add to Ken's great email which confirms my original
> assertion about electrons.
>
> If Santosh really wanted to calculate SE's after every "iteration" (which
> I think was Ken's interpretation of  e

RE: [NMusers] Obtaining RSE%

[kgkowalski58]

Hi Santosh,

 

It’s important to note the distinction between transformations of the 
parameters and transformations of the data such as the log-transform-both-sides 
approach for a PK model to assume the residual errors are log-normally 
distributed.  Here we are specifically focusing on transformations of the 
parameters not the data.  Note that the likelihood is invariant to 
transformations of the parameters, so you will get the same fit and OFV whether 
you estimate log(CL) or CL as your theta.  However, Wald-based standard errors 
are very much dependent on the parameter transformation.  

 

For example, suppose we estimate the typical value of CL as THETA(1).  Assuming 
the maximum likelihood estimate of THETA(1) is asymptotically normal then we 
could construct a confidence interval to reflect the uncertainty in that 
parameter estimate as 

 

theta1 +/- Zalpha(SE_theta1) 

 

where SE_theta1 is the Wald-based SE for theta1 and Zalpha is the two-sided 
critical value of the standard normal distribution to obtain a 100x(1 – alpha)% 
confidence interval.  Note that this confidence interval is symmetric about the 
estimate theta1.  

 

Now consider a log-transformation of CL such that THETA(1) corresponds to 
log(CL).  The Wald-based confidence interval for log(CL) would now be

 

theta1 +/- Zalpha(SE_theta1) 

 

which is symmetric in the log(CL) scale.  However, the corresponding confidence 
interval for CL requires exponentiating the endpoints of the log(CL) confidence 
interval to obtain the confidence interval in the original CL scale.  That is,

 

exp(theta1 +/- Zalpha(SE_theta1) )

 

which will be asymmetric about the estimate of the typical value of CL, 
exp(theta1).

 

When the parameter estimate space is highly asymmetric, transformations can 
help with this asymmetry so that the transformed estimates are more likely to 
be symmetric and normally distributed.  So, to answer your question, the 
precision of the estimates may still be valid, but we need to recognize that 
the uncertainty in the estimates may be asymmetric in the untransformed 
(original) space.

 

Best,

 

Ken

 

From: owner-nmus...@globomaxnm.com  On Behalf Of 
Santosh
Sent: Monday, July 29, 2024 1:11 PM
To: nmusers@globomaxnm.com
Subject: Re: [NMusers] Obtaining RSE%

 

Dear Prof Holford, Ken, Alan, Jeroen & others, 

 

Thanks for the engaging discussions.

 

In context of monitoring at the iteration level, I vaguely recall that in 
NMUSERS or in one of ACOP conferences , there was a presentation & 
demonstration with R scripts on looking at the convergence and other parameters 
in real time. 

 

The interpretations of SEs is interesting based on linear or non-linear models, 
and also based on size of variance of parameters. 

 

On a different note, I am also interested in hearing from you about SEs when 
estimated based on transformed distribution space and their values & 
interpretations in back-transformed space. Would the notion of precision  still 
be valid  when viewing both transformed and untransformed space? This is in 
context of dealing with untransformed space of non-normal or non-lognormal 
distributions.

 

Best regards 

Santosh

 

 

On Mon, Jul 29, 2024 at 8:52 AM Nick Holford mailto:n.holf...@auckland.ac.nz> > wrote:

Hi Jeroen,

A small correction. Please re-read my email to nmusers on 12 Feb 2015 which I 
quote here. Sorry I cannot show the original but the 1999 URL is not available 
to me anymore. 

=  start quote ===
Nick Holford Thu, 12 Feb 2015 11:54:59 -0800
Hi,
The original quote about electrons comes from a remark I made in 1999 on 
nmusers.
http://www.cognigencorp.com/nonmem/nm/99nov121999.html
Lewis Sheiner agreed in the same thread. Thanks to the wonders of living on a 
sphere Lewis appears to agree with me the day before I made the comment :-)
=  end quote ===

I had been meaning to add to Ken's great email which confirms my original 
assertion about electrons.

If Santosh really wanted to calculate SE's after every "iteration" (which I 
think was Ken's interpretation of  every "estimation") then this can be done by 
running a non-parametric bootstrap with the parameter estimates produced after 
every iteration. 

I wonder if Santosh would like to spend a few hours doing that and adding to 
the nmusers collection about standard errors by reporting the results to us?


Best wishes,
Nick


--
Nick Holford, Professor Emeritus Clinical Pharmacology, MBChB, FRACP
mobile: NZ+64(21) 46 23 53 ; FR+33(6) 62 32 46 72
email: n.holf...@auckland.ac.nz <mailto:n.holf...@auckland.ac.nz> 
web: http://holford.fmhs.auckland.ac.nz/

-Original Message-
From: owner-nmus...@globomaxnm.com <mailto:owner-nmus...@globomaxnm.com>  
mailto:owner-nmus...@globomaxnm.com> > On Behalf 
Of Jeroen Elassaiss-Schaap (PD-value B.V.)
Sent: Monday, July 29, 2024 3:37 P

RE: [NMusers] Obtaining RSE%

[kgkowalski58]

Hi Dennis,

 

I suspect that Stu’s and perhaps Lewis’ position on the value of the COV step 
is even more nuanced than what you describe below in that even if the COV step 
runs such that the standard errors can be produced this does not assure us that 
the estimates are truly at the global minimum.  However, the COV step does have 
value in assessing the stability of our models where it is more likely that if 
we have a stable model then the estimation algorithm can more easily converge 
to the global minimum.  My position which I think Stu and Lewis would probably 
support is that there is diagnostic value in the COV step output including the 
standard errors, but we may not be able to use these standard errors in formal 
statistical inference (e.g., hypothesis testing and confidence interval 
coverage).

 

Best,

 

Ken

 

From: Dennis Fisher  
Sent: Monday, July 29, 2024 2:29 PM
To: kgkowalsk...@gmail.com
Cc: Nick Holford ; Jeroen Elassaiss-Schaap (PD-value 
B.V.) ; Santosh ; 
nmusers@globomaxnm.com; Alan Maloney ; Pyry Välitalo 

Subject: Re: [NMusers] Obtaining RSE%

 

Ken

 

During Lewis’ irregular research conferences at UCSF, this issue came up 
periodically.  Lewis and Stu Beal added some nuance: the parameter space was 
often asymmetric; hence, the NONMEM estimates were not meaningful.  But Stu 
felt strongly (and Lewis appeared to agree) that obtaining standard errors 
assured that the estimates were truly the global minimum, i.e., the value of 
the covariance step was not the estimate of SE.

 

Dennis

 


Dennis Fisher MD
P < (The "P Less Than" Company)
Phone / Fax: 1-866-PLessThan (1-866-753-7784)
www.PLessThan.com <http://www.PLessThan.com> 





On Jul 29, 2024, at 11:12 AM, mailto:kgkowalsk...@gmail.com> > mailto:kgkowalsk...@gmail.com> > wrote:

 

Hi Nick,

I don't want to rehash old debates with you about the diagnostic value of the 
COV step.  However, your statement about SEs "they are not worth the electrons 
expended to compute them" seems hyperbolic to me.  I suspect that what Lewis 
agreed to was the general sentiment that we need to be cautious in how we use 
and interpret the SEs generated by NONMEM.  I doubt that he felt that they have 
absolutely no value.  Indeed, in many of Lewis' papers where he published 
modeling results, he reports the standard errors of these estimates from NONMEM.

It certainly was not my intent to assert that the SEs and the COV step in 
general, have no value.  I believe they still do, even if we may not be able to 
use them say to construct confidence intervals and expect them to have the 
proper coverage probabilities for purposes of statistical inference.

I do not think a non-parametric bootstrap with the parameter estimates produced 
after every iteration is going to tell us anything.  If for no other reason 
that the iteration search path itself is dependent on the starting values used. 
 That is, the parameter estimates after each iteration will depend on where you 
start.  Whereas the maximum likelihood estimates obtained at convergence to the 
global minimum OFV, should be somewhat invariant to the starting values 
provided the starting values are reasonable. The theory behind the 
non-parametric bootstrap standard errors still requires that you obtain the 
maximum likelihood estimates for each bootstrap dataset.

Best,

Ken

-Original Message-
From: Nick Holford mailto:n.holf...@auckland.ac.nz> 
> 
Sent: Monday, July 29, 2024 11:52 AM
To: Jeroen Elassaiss-Schaap (PD-value B.V.) mailto:jer...@pd-value.com> >; kgkowalsk...@gmail.com 
<mailto:kgkowalsk...@gmail.com> ; 'Santosh' mailto:santosh2...@gmail.com> >; nmusers@globomaxnm.com 
<mailto:nmusers@globomaxnm.com> 
Cc: 'Alan Maloney' mailto:al_in_swe...@hotmail.com> 
>; Pyry Välitalo mailto:pyry.valit...@gmail.com> >
Subject: RE: [NMusers] Obtaining RSE%

Hi Jeroen,

A small correction. Please re-read my email to nmusers on 12 Feb 2015 which I 
quote here. Sorry I cannot show the original but the 1999 URL is not available 
to me anymore. 

=  start quote === Nick Holford Thu, 12 Feb 
2015 11:54:59 -0800 Hi, The original quote about electrons comes from a remark 
I made in 1999 on nmusers.
http://www.cognigencorp.com/nonmem/nm/99nov121999.html
Lewis Sheiner agreed in the same thread. Thanks to the wonders of living on a 
sphere Lewis appears to agree with me the day before I made the comment :-) 
=  end quote ===

I had been meaning to add to Ken's great email which confirms my original 
assertion about electrons.

If Santosh really wanted to calculate SE's after every "iteration" (which I 
think was Ken's interpretation of  every "estimation") then this can be done by 
running a non-parametric bootstrap with the parameter estimates produced after 
every iteration. 

I 

Re: [NMusers] Obtaining RSE%

[Dennis Fisher]

Ken

During Lewis’ irregular research conferences at UCSF, this issue came up 
periodically.  Lewis and Stu Beal added some nuance: the parameter space was 
often asymmetric; hence, the NONMEM estimates were not meaningful.  But Stu 
felt strongly (and Lewis appeared to agree) that obtaining standard errors 
assured that the estimates were truly the global minimum, i.e., the value of 
the covariance step was not the estimate of SE.

Dennis


Dennis Fisher MD
P < (The "P Less Than" Company)
Phone / Fax: 1-866-PLessThan (1-866-753-7784)
www.PLessThan.com

> On Jul 29, 2024, at 11:12 AM,  
>  wrote:
> 
> Hi Nick,
> 
> I don't want to rehash old debates with you about the diagnostic value of the 
> COV step.  However, your statement about SEs "they are not worth the 
> electrons expended to compute them" seems hyperbolic to me.  I suspect that 
> what Lewis agreed to was the general sentiment that we need to be cautious in 
> how we use and interpret the SEs generated by NONMEM.  I doubt that he felt 
> that they have absolutely no value.  Indeed, in many of Lewis' papers where 
> he published modeling results, he reports the standard errors of these 
> estimates from NONMEM.
> 
> It certainly was not my intent to assert that the SEs and the COV step in 
> general, have no value.  I believe they still do, even if we may not be able 
> to use them say to construct confidence intervals and expect them to have the 
> proper coverage probabilities for purposes of statistical inference.
> 
> I do not think a non-parametric bootstrap with the parameter estimates 
> produced after every iteration is going to tell us anything.  If for no other 
> reason that the iteration search path itself is dependent on the starting 
> values used.  That is, the parameter estimates after each iteration will 
> depend on where you start.  Whereas the maximum likelihood estimates obtained 
> at convergence to the global minimum OFV, should be somewhat invariant to the 
> starting values provided the starting values are reasonable. The theory 
> behind the non-parametric bootstrap standard errors still requires that you 
> obtain the maximum likelihood estimates for each bootstrap dataset.
> 
> Best,
> 
> Ken
> 
> -Original Message-
> From: Nick Holford  
> Sent: Monday, July 29, 2024 11:52 AM
> To: Jeroen Elassaiss-Schaap (PD-value B.V.) ; 
> kgkowalsk...@gmail.com; 'Santosh' ; 
> nmusers@globomaxnm.com
> Cc: 'Alan Maloney' ; Pyry Välitalo 
> 
> Subject: RE: [NMusers] Obtaining RSE%
> 
> Hi Jeroen,
> 
> A small correction. Please re-read my email to nmusers on 12 Feb 2015 which I 
> quote here. Sorry I cannot show the original but the 1999 URL is not 
> available to me anymore. 
> 
> =  start quote === Nick Holford Thu, 12 Feb 
> 2015 11:54:59 -0800 Hi, The original quote about electrons comes from a 
> remark I made in 1999 on nmusers.
> http://www.cognigencorp.com/nonmem/nm/99nov121999.html
> Lewis Sheiner agreed in the same thread. Thanks to the wonders of living on a 
> sphere Lewis appears to agree with me the day before I made the comment :-) 
> =  end quote ===
> 
> I had been meaning to add to Ken's great email which confirms my original 
> assertion about electrons.
> 
> If Santosh really wanted to calculate SE's after every "iteration" (which I 
> think was Ken's interpretation of  every "estimation") then this can be done 
> by running a non-parametric bootstrap with the parameter estimates produced 
> after every iteration. 
> 
> I wonder if Santosh would like to spend a few hours doing that and adding to 
> the nmusers collection about standard errors by reporting the results to us?
> 
> 
> Best wishes,
> Nick
> 
> 
> --
> Nick Holford, Professor Emeritus Clinical Pharmacology, MBChB, FRACP
> mobile: NZ+64(21) 46 23 53 ; FR+33(6) 62 32 46 72
> email: n.holf...@auckland.ac.nz
> web: http://holford.fmhs.auckland.ac.nz/
> 
> -Original Message-
> From: owner-nmus...@globomaxnm.com  On Behalf 
> Of Jeroen Elassaiss-Schaap (PD-value B.V.)
> Sent: Monday, July 29, 2024 3:37 PM
> To: kgkowalsk...@gmail.com; 'Santosh' ; 
> nmusers@globomaxnm.com
> Cc: 'Alan Maloney' ; Pyry Välitalo 
> 
> Subject: Re: [NMusers] Obtaining RSE%
> 
> [Some people who received this message don't often get email from 
> jer...@pd-value.com. Learn why this is important at 
> https://aka.ms/LearnAboutSenderIdentification ]
> 
> Dear NMusers,
> 
> This is a great reminder for us to consider the reliability of standard 
> errors in our models, thanks Ken & Alan. The more non

RE: [NMusers] Obtaining RSE%

[kgkowalski58]

Hi Nick,

I don't want to rehash old debates with you about the diagnostic value of the 
COV step.  However, your statement about SEs "they are not worth the electrons 
expended to compute them" seems hyperbolic to me.  I suspect that what Lewis 
agreed to was the general sentiment that we need to be cautious in how we use 
and interpret the SEs generated by NONMEM.  I doubt that he felt that they have 
absolutely no value.  Indeed, in many of Lewis' papers where he published 
modeling results, he reports the standard errors of these estimates from NONMEM.

It certainly was not my intent to assert that the SEs and the COV step in 
general, have no value.  I believe they still do, even if we may not be able to 
use them say to construct confidence intervals and expect them to have the 
proper coverage probabilities for purposes of statistical inference.

I do not think a non-parametric bootstrap with the parameter estimates produced 
after every iteration is going to tell us anything.  If for no other reason 
that the iteration search path itself is dependent on the starting values used. 
 That is, the parameter estimates after each iteration will depend on where you 
start.  Whereas the maximum likelihood estimates obtained at convergence to the 
global minimum OFV, should be somewhat invariant to the starting values 
provided the starting values are reasonable. The theory behind the 
non-parametric bootstrap standard errors still requires that you obtain the 
maximum likelihood estimates for each bootstrap dataset.

Best,

Ken

-Original Message-
From: Nick Holford  
Sent: Monday, July 29, 2024 11:52 AM
To: Jeroen Elassaiss-Schaap (PD-value B.V.) ; 
kgkowalsk...@gmail.com; 'Santosh' ; 
nmusers@globomaxnm.com
Cc: 'Alan Maloney' ; Pyry Välitalo 

Subject: RE: [NMusers] Obtaining RSE%

Hi Jeroen,

A small correction. Please re-read my email to nmusers on 12 Feb 2015 which I 
quote here. Sorry I cannot show the original but the 1999 URL is not available 
to me anymore. 

=  start quote === Nick Holford Thu, 12 Feb 
2015 11:54:59 -0800 Hi, The original quote about electrons comes from a remark 
I made in 1999 on nmusers.
http://www.cognigencorp.com/nonmem/nm/99nov121999.html
Lewis Sheiner agreed in the same thread. Thanks to the wonders of living on a 
sphere Lewis appears to agree with me the day before I made the comment :-) 
=  end quote ===

I had been meaning to add to Ken's great email which confirms my original 
assertion about electrons.

If Santosh really wanted to calculate SE's after every "iteration" (which I 
think was Ken's interpretation of  every "estimation") then this can be done by 
running a non-parametric bootstrap with the parameter estimates produced after 
every iteration. 

I wonder if Santosh would like to spend a few hours doing that and adding to 
the nmusers collection about standard errors by reporting the results to us?


Best wishes,
Nick


--
Nick Holford, Professor Emeritus Clinical Pharmacology, MBChB, FRACP
mobile: NZ+64(21) 46 23 53 ; FR+33(6) 62 32 46 72
email: n.holf...@auckland.ac.nz
web: http://holford.fmhs.auckland.ac.nz/

-Original Message-
From: owner-nmus...@globomaxnm.com  On Behalf Of 
Jeroen Elassaiss-Schaap (PD-value B.V.)
Sent: Monday, July 29, 2024 3:37 PM
To: kgkowalsk...@gmail.com; 'Santosh' ; 
nmusers@globomaxnm.com
Cc: 'Alan Maloney' ; Pyry Välitalo 

Subject: Re: [NMusers] Obtaining RSE%

[Some people who received this message don't often get email from 
jer...@pd-value.com. Learn why this is important at 
https://aka.ms/LearnAboutSenderIdentification ]

Dear NMusers,

This is a great reminder for us to consider the reliability of standard errors 
in our models, thanks Ken & Alan. The more non-linear the models become, the 
less reliable and the more important other perspectives on parameter values 
such as sensitivity analysis and prior knowledge.

The nmusers archive has many great threads on the topic that are available to 
review such as 
https://www.mail-archive.com/nmusers@globomaxnm.com/msg05423.html and related 
https://www.mail-archive.com/nmusers@globomaxnm.com/msg05419.html . In summary, 
log-transformation only can get you so far but can perhaps be seen as a sort of 
minimal effort.

To add to the Lewis's quote about SEs - "they are not worth the electrons used 
to compute them" (see the links), Pyry had some very interesting observations 
he shared with me about the SE of the CV of a log-normal omega: it inflates 
with higher values of omega compared to the SE of omega itself.

Best regards,

Jeroen

http://pd-value.com
jer...@pd-value.com
@PD_value
+31 6 23118438
-- More value out of your data!

On 29-07-2024 14:41, kgkowalsk...@gmail.com wrote:
>
> Dear NMusers,
>
> It was recently pointed out to me by a statistical colleague t

Re: [NMusers] Obtaining RSE%

[Santosh]

Dear Prof Holford, Ken, Alan, Jeroen & others,

Thanks for the engaging discussions.

In context of monitoring at the iteration level, I vaguely recall that in
NMUSERS or in one of ACOP conferences , there was a presentation &
demonstration with R scripts on looking at the convergence and other
parameters in real time.

The interpretations of SEs is interesting based on linear or non-linear
models, and also based on size of variance of parameters.

On a different note, I am also interested in hearing from you about SEs
when estimated based on transformed distribution space and their values &
interpretations in back-transformed space. Would the notion of precision
 still be valid  when viewing both transformed and untransformed space?
This is in context of dealing with untransformed space of non-normal or
non-lognormal distributions.

Best regards
Santosh


On Mon, Jul 29, 2024 at 8:52 AM Nick Holford 
wrote:

> Hi Jeroen,
>
> A small correction. Please re-read my email to nmusers on 12 Feb 2015
> which I quote here. Sorry I cannot show the original but the 1999 URL is
> not available to me anymore.
>
> =  start quote ===
> Nick Holford Thu, 12 Feb 2015 11:54:59 -0800
> Hi,
> The original quote about electrons comes from a remark I made in 1999 on
> nmusers.
> http://www.cognigencorp.com/nonmem/nm/99nov121999.html
> Lewis Sheiner agreed in the same thread. Thanks to the wonders of living
> on a sphere Lewis appears to agree with me the day before I made the
> comment :-)
> =  end quote ===
>
> I had been meaning to add to Ken's great email which confirms my original
> assertion about electrons.
>
> If Santosh really wanted to calculate SE's after every "iteration" (which
> I think was Ken's interpretation of  every "estimation") then this can be
> done by running a non-parametric bootstrap with the parameter estimates
> produced after every iteration.
>
> I wonder if Santosh would like to spend a few hours doing that and adding
> to the nmusers collection about standard errors by reporting the results to
> us?
>
>
> Best wishes,
> Nick
>
>
> --
> Nick Holford, Professor Emeritus Clinical Pharmacology, MBChB, FRACP
> mobile: NZ+64(21) 46 23 53 ; FR+33(6) 62 32 46 72
> email: n.holf...@auckland.ac.nz
> web: http://holford.fmhs.auckland.ac.nz/
>
> -Original Message-
> From: owner-nmus...@globomaxnm.com  On
> Behalf Of Jeroen Elassaiss-Schaap (PD-value B.V.)
> Sent: Monday, July 29, 2024 3:37 PM
> To: kgkowalsk...@gmail.com; 'Santosh' ;
> nmusers@globomaxnm.com
> Cc: 'Alan Maloney' ; Pyry Välitalo <
> pyry.valit...@gmail.com>
> Subject: Re: [NMusers] Obtaining RSE%
>
> [Some people who received this message don't often get email from
> jer...@pd-value.com. Learn why this is important at
> https://aka.ms/LearnAboutSenderIdentification ]
>
> Dear NMusers,
>
> This is a great reminder for us to consider the reliability of standard
> errors in our models, thanks Ken & Alan. The more non-linear the models
> become, the less reliable and the more important other perspectives on
> parameter values such as sensitivity analysis and prior knowledge.
>
> The nmusers archive has many great threads on the topic that are available
> to review such as
> https://www.mail-archive.com/nmusers@globomaxnm.com/msg05423.html and
> related https://www.mail-archive.com/nmusers@globomaxnm.com/msg05419.html
> . In summary, log-transformation only can get you so far but can perhaps be
> seen as a sort of minimal effort.
>
> To add to the Lewis's quote about SEs - "they are not worth the electrons
> used to compute them" (see the links), Pyry had some very interesting
> observations he shared with me about the SE of the CV of a log-normal
> omega: it inflates with higher values of omega compared to the SE of omega
> itself.
>
> Best regards,
>
> Jeroen
>
> http://pd-value.com
> jer...@pd-value.com
> @PD_value
> +31 6 23118438
> -- More value out of your data!
>
> On 29-07-2024 14:41, kgkowalsk...@gmail.com wrote:
> >
> > Dear NMusers,
> >
> > It was recently pointed out to me by a statistical colleague that my
> > recent NMusers post about the inverse Hessian (R matrix) evaluated at
> > the maximum likelihood estimates is a consistent estimator of the
> > covariance matrix (i.e., converges to the true value with large N) is
> > only true for linear models.  For nonlinear models, the standard
> > errors produced by NONMEM and other nonlinear estimation software are
> > not only asymptotic but also approximate.  Moreover, how well that
> > approximation works

RE: [NMusers] Obtaining RSE%

[Nick Holford]

Hi Jeroen,

A small correction. Please re-read my email to nmusers on 12 Feb 2015 which I 
quote here. Sorry I cannot show the original but the 1999 URL is not available 
to me anymore. 

=  start quote ===
Nick Holford Thu, 12 Feb 2015 11:54:59 -0800
Hi,
The original quote about electrons comes from a remark I made in 1999 on 
nmusers.
http://www.cognigencorp.com/nonmem/nm/99nov121999.html
Lewis Sheiner agreed in the same thread. Thanks to the wonders of living on a 
sphere Lewis appears to agree with me the day before I made the comment :-)
=  end quote ===

I had been meaning to add to Ken's great email which confirms my original 
assertion about electrons.

If Santosh really wanted to calculate SE's after every "iteration" (which I 
think was Ken's interpretation of  every "estimation") then this can be done by 
running a non-parametric bootstrap with the parameter estimates produced after 
every iteration. 

I wonder if Santosh would like to spend a few hours doing that and adding to 
the nmusers collection about standard errors by reporting the results to us?


Best wishes,
Nick


--
Nick Holford, Professor Emeritus Clinical Pharmacology, MBChB, FRACP
mobile: NZ+64(21) 46 23 53 ; FR+33(6) 62 32 46 72
email: n.holf...@auckland.ac.nz
web: http://holford.fmhs.auckland.ac.nz/

-Original Message-
From: owner-nmus...@globomaxnm.com  On Behalf Of 
Jeroen Elassaiss-Schaap (PD-value B.V.)
Sent: Monday, July 29, 2024 3:37 PM
To: kgkowalsk...@gmail.com; 'Santosh' ; 
nmusers@globomaxnm.com
Cc: 'Alan Maloney' ; Pyry Välitalo 

Subject: Re: [NMusers] Obtaining RSE%

[Some people who received this message don't often get email from 
jer...@pd-value.com. Learn why this is important at 
https://aka.ms/LearnAboutSenderIdentification ]

Dear NMusers,

This is a great reminder for us to consider the reliability of standard errors 
in our models, thanks Ken & Alan. The more non-linear the models become, the 
less reliable and the more important other perspectives on parameter values 
such as sensitivity analysis and prior knowledge.

The nmusers archive has many great threads on the topic that are available to 
review such as 
https://www.mail-archive.com/nmusers@globomaxnm.com/msg05423.html and related 
https://www.mail-archive.com/nmusers@globomaxnm.com/msg05419.html . In summary, 
log-transformation only can get you so far but can perhaps be seen as a sort of 
minimal effort.

To add to the Lewis's quote about SEs - "they are not worth the electrons used 
to compute them" (see the links), Pyry had some very interesting observations 
he shared with me about the SE of the CV of a log-normal omega: it inflates 
with higher values of omega compared to the SE of omega itself.

Best regards,

Jeroen

http://pd-value.com
jer...@pd-value.com
@PD_value
+31 6 23118438
-- More value out of your data!

On 29-07-2024 14:41, kgkowalsk...@gmail.com wrote:
>
> Dear NMusers,
>
> It was recently pointed out to me by a statistical colleague that my 
> recent NMusers post about the inverse Hessian (R matrix) evaluated at 
> the maximum likelihood estimates is a consistent estimator of the 
> covariance matrix (i.e., converges to the true value with large N) is 
> only true for linear models.  For nonlinear models, the standard 
> errors produced by NONMEM and other nonlinear estimation software are 
> not only asymptotic but also approximate.  Moreover, how well that 
> approximation works will also depend on the parameterization.  This I 
> believe is one of the motivations behind “mu referencing” in NONMEM 
> and the use of log transformations of the parameters to help improve 
> Wald-based approximations.  I thank Alan Maloney for pointing this out 
> to me.
>
> Kind regards,
>
> Ken
>
> *From:*kgkowalsk...@gmail.com 
> *Sent:* Saturday, July 27, 2024 12:36 PM
> *To:* 'Santosh' ; nmusers@globomaxnm.com
> *Subject:* RE: [NMusers] Obtaining RSE%
>
> Dear Santosh,
>
> There is a good reason for this.  Wald (1943) has shown that the 
> inverse of the Hessian (R matrix) evaluated at the maximum likelihood 
> estimates is a consistent estimator of the covariance matrix.  It is 
> based on Wald’s approximation that the likelihood surface locally near 
> the maximum likelihood estimates can be approximated by a quadratic 
> function in the parameters.  This theory does not hold for any set of 
> parameter estimates along the algorithm’s search path prior to 
> convergence to the maximum likelihood estimates. Moreover,  inverting 
> the Hessian evaluated at an interim step prior to convergence would 
> likely be a poor approximation especially early in the search path 
> where the gradients are large (i.e., large changes in OFV for a given 
> change in the parameters 

Re: [NMusers] Obtaining RSE%

[Jeroen Elassaiss-Schaap (PD-value B.V.)]

Dear NMusers,

This is a great reminder for us to consider the reliability of standard 
errors in our models, thanks Ken & Alan. The more non-linear the models 
become, the less reliable and the more important other perspectives on 
parameter values such as sensitivity analysis and prior knowledge.


The nmusers archive has many great threads on the topic that are 
available to review such as 
https://www.mail-archive.com/nmusers@globomaxnm.com/msg05423.html and 
related 
https://www.mail-archive.com/nmusers@globomaxnm.com/msg05419.html . In 
summary, log-transformation only can get you so far but can perhaps be 
seen as a sort of minimal effort.


To add to the Lewis's quote about SEs - "they are not worth the 
electrons used to compute them" (see the links), Pyry had some very 
interesting observations he shared with me about the SE of the CV of a 
log-normal omega: it inflates with higher values of omega compared to 
the SE of omega itself.


Best regards,

Jeroen

http://pd-value.com
jer...@pd-value.com
@PD_value
+31 6 23118438
-- More value out of your data!

On 29-07-2024 14:41, kgkowalsk...@gmail.com wrote:


Dear NMusers,

It was recently pointed out to me by a statistical colleague that my 
recent NMusers post about the inverse Hessian (R matrix) evaluated at 
the maximum likelihood estimates is a consistent estimator of the 
covariance matrix (i.e., converges to the true value with large N) is 
only true for linear models.  For nonlinear models, the standard 
errors produced by NONMEM and other nonlinear estimation software are 
not only asymptotic but also approximate.  Moreover, how well that 
approximation works will also depend on the parameterization.  This I 
believe is one of the motivations behind “mu referencing” in NONMEM 
and the use of log transformations of the parameters to help improve 
Wald-based approximations.  I thank Alan Maloney for pointing this out 
to me.


Kind regards,

Ken

*From:*kgkowalsk...@gmail.com 
*Sent:* Saturday, July 27, 2024 12:36 PM
*To:* 'Santosh' ; nmusers@globomaxnm.com
*Subject:* RE: [NMusers] Obtaining RSE%

Dear Santosh,

There is a good reason for this.  Wald (1943) has shown that the 
inverse of the Hessian (R matrix) evaluated at the maximum likelihood 
estimates is a consistent estimator of the covariance matrix.  It is 
based on Wald’s approximation that the likelihood surface locally near 
the maximum likelihood estimates can be approximated by a quadratic 
function in the parameters.  This theory does not hold for any set of 
parameter estimates along the algorithm’s search path prior to 
convergence to the maximum likelihood estimates. Moreover,  inverting 
the Hessian evaluated at an interim step prior to convergence would 
likely be a poor approximation especially early in the search path 
where the gradients are large (i.e., large changes in OFV for a given 
change in the parameters would probably have substantial curvature and 
not be well approximated by a quadratic model in the parameters).


Thus, the COV step in NONMEM is only applied once convergence is 
obtained during the EST step.


Wald, A. “Tests of statistical hypotheses concerning several 
parameters when the number of observations is large.” /Trans. Amer. 
Math. Soc./ 1943;54:426.


Best,

Ken

Kenneth G. Kowalski

President

Kowalski PMetrics Consulting, LLC

Email: kgkowalsk...@gmail.com <mailto:kgkowalsk...@gmail.com>

Cell:  248-207-5082

*From:*owner-nmus...@globomaxnm.com 
<mailto:owner-nmus...@globomaxnm.com><mailto:owner-nmus...@globomaxnm.com>> *On Behalf Of *Santosh

*Sent:* Friday, July 26, 2024 3:38 AM
*To:* nmusers@globomaxnm.com <mailto:nmusers@globomaxnm.com>
*Subject:* [NMusers] Obtaining RSE%

 Dear esteemed experts!

When using one or more estimation methods & covariance step in a 
NONMEM control stream, the resulting ext file contains final estimate 
(for all estimation steps)  & standard error (only for the last 
estimation step).


Is there a way that standard error is generated for every estimation step?

TIA

Santosh

RE: [NMusers] Obtaining RSE%

[kgkowalski58]

Dear NMusers,

 

It was recently pointed out to me by a statistical colleague that my recent 
NMusers post about the inverse Hessian (R matrix) evaluated at the maximum 
likelihood estimates is a consistent estimator of the covariance matrix (i.e., 
converges to the true value with large N) is only true for linear models.  For 
nonlinear models, the standard errors produced by NONMEM and other nonlinear 
estimation software are not only asymptotic but also approximate.  Moreover, 
how well that approximation works will also depend on the parameterization.  
This I believe is one of the motivations behind “mu referencing” in NONMEM and 
the use of log transformations of the parameters to help improve Wald-based 
approximations.  I thank Alan Maloney for pointing this out to me. 

 

Kind regards,

 

Ken

 

From: kgkowalsk...@gmail.com  
Sent: Saturday, July 27, 2024 12:36 PM
To: 'Santosh' ; nmusers@globomaxnm.com
Subject: RE: [NMusers] Obtaining RSE%

 

Dear Santosh,

 

There is a good reason for this.  Wald (1943) has shown that the inverse of the 
Hessian (R matrix) evaluated at the maximum likelihood estimates is a 
consistent estimator of the covariance matrix.  It is based on Wald’s 
approximation that the likelihood surface locally near the maximum likelihood 
estimates can be approximated by a quadratic function in the parameters.  This 
theory does not hold for any set of parameter estimates along the algorithm’s 
search path prior to convergence to the maximum likelihood estimates.  
Moreover,  inverting the Hessian evaluated at an interim step prior to 
convergence would likely be a poor approximation especially early in the search 
path where the gradients are large (i.e., large changes in OFV for a given 
change in the parameters would probably have substantial curvature and not be 
well approximated by a quadratic model in the parameters).

 

Thus, the COV step in NONMEM is only applied once convergence is obtained 
during the EST step.

 

Wald, A. “Tests of statistical hypotheses concerning several parameters when 
the number of observations is large.” Trans. Amer. Math. Soc. 1943;54:426.

 

Best,

 

Ken

 

Kenneth G. Kowalski

President

Kowalski PMetrics Consulting, LLC

Email:  <mailto:kgkowalsk...@gmail.com> kgkowalsk...@gmail.com

Cell:  248-207-5082

 

 

From:  <mailto:owner-nmus...@globomaxnm.com> owner-nmus...@globomaxnm.com < 
<mailto:owner-nmus...@globomaxnm.com> owner-nmus...@globomaxnm.com> On Behalf 
Of Santosh
Sent: Friday, July 26, 2024 3:38 AM
To:  <mailto:nmusers@globomaxnm.com> nmusers@globomaxnm.com
Subject: [NMusers] Obtaining RSE%

 

 Dear esteemed experts!

 

When using one or more estimation methods & covariance step in a NONMEM control 
stream, the resulting ext file contains final estimate (for all estimation 
steps)  & standard error (only for the last estimation step). 

 

Is there a way that standard error is generated for every estimation step?

 

TIA

Santosh 

Re: [NMusers] Obtaining RSE%

[Santosh]

Thanks Leonid & Ken for quick responses.

I did try with multiple $COV steps and submitted the jobs with
Perl-speaks-NONMEM (PsN).. PsN reorganized the NONMEM blocks are changed
the order of $COV steps..  I’ll try with nmfe way…

Hope PsN developers look into this issue and preserve the order of the
lines of codes where they matter, especially the sequence of $ESTIMATION &
$COVARIANCE steps.

TIA
Santosh

On Sat, Jul 27, 2024 at 6:44 PM  wrote:

> Aah – I see that I misunderstood Santosh’s question.  I thought Santosh
> was asking about reporting standard errors at each iteration step within
> the estimation algorithm.
>
>
>
> Best,
>
>
>
> Ken
>
>
>
> *From:* Leonid Gibiansky 
> *Sent:* Saturday, July 27, 2024 4:18 PM
> *To:* Ken Kowalski 
> *Cc:* Santosh ; nmusers 
> *Subject:* Re: [NMusers] Obtaining RSE%
>
>
>
> It can be done if you add extra $cov statement after each estimation
> method record
>
> Thank you
>
> Leonid
>
>
>
>
>
> On Sat, Jul 27, 2024, 12:47 PM  wrote:
>
> Dear Santosh,
>
>
>
> There is a good reason for this.  Wald (1943) has shown that the inverse
> of the Hessian (R matrix) evaluated at the maximum likelihood estimates is
> a consistent estimator of the covariance matrix.  It is based on Wald’s
> approximation that the likelihood surface locally near the maximum
> likelihood estimates can be approximated by a quadratic function in the
> parameters.  This theory does not hold for any set of parameter estimates
> along the algorithm’s search path prior to convergence to the maximum
> likelihood estimates.  Moreover,  inverting the Hessian evaluated at an
> interim step prior to convergence would likely be a poor approximation
> especially early in the search path where the gradients are large (i.e.,
> large changes in OFV for a given change in the parameters would probably
> have substantial curvature and not be well approximated by a quadratic
> model in the parameters).
>
>
>
> Thus, the COV step in NONMEM is only applied once convergence is obtained
> during the EST step.
>
>
>
> Wald, A. “Tests of statistical hypotheses concerning several parameters
> when the number of observations is large.” *Trans. Amer. Math. Soc.*
> 1943;54:426.
>
>
>
> Best,
>
>
>
> Ken
>
>
>
> Kenneth G. Kowalski
>
> President
>
> Kowalski PMetrics Consulting, LLC
>
> Email: kgkowalsk...@gmail.com
>
> Cell:  248-207-5082
>
>
>
>
>
> *From:* owner-nmus...@globomaxnm.com  *On
> Behalf Of *Santosh
> *Sent:* Friday, July 26, 2024 3:38 AM
> *To:* nmusers@globomaxnm.com
> *Subject:* [NMusers] Obtaining RSE%
>
>
>
>  Dear esteemed experts!
>
> When using one or more estimation methods & covariance step in a NONMEM
> control stream, the resulting ext file contains final estimate (for all
> estimation steps)  & standard error (only for the last estimation step).
>
>
>
> Is there a way that standard error is generated for every estimation step?
>
>
>
> TIA
>
> Santosh
>
>

RE: [NMusers] Obtaining RSE%

[kgkowalski58]

Aah – I see that I misunderstood Santosh’s question.  I thought Santosh was 
asking about reporting standard errors at each iteration step within the 
estimation algorithm.

 

Best,

 

Ken

 

From: Leonid Gibiansky  
Sent: Saturday, July 27, 2024 4:18 PM
To: Ken Kowalski 
Cc: Santosh ; nmusers 
Subject: Re: [NMusers] Obtaining RSE%

 

It can be done if you add extra $cov statement after each estimation method 
record

Thank you

Leonid

 

 

On Sat, Jul 27, 2024, 12:47 PM mailto:kgkowalsk...@gmail.com> > wrote:

Dear Santosh,

 

There is a good reason for this.  Wald (1943) has shown that the inverse of the 
Hessian (R matrix) evaluated at the maximum likelihood estimates is a 
consistent estimator of the covariance matrix.  It is based on Wald’s 
approximation that the likelihood surface locally near the maximum likelihood 
estimates can be approximated by a quadratic function in the parameters.  This 
theory does not hold for any set of parameter estimates along the algorithm’s 
search path prior to convergence to the maximum likelihood estimates.  
Moreover,  inverting the Hessian evaluated at an interim step prior to 
convergence would likely be a poor approximation especially early in the search 
path where the gradients are large (i.e., large changes in OFV for a given 
change in the parameters would probably have substantial curvature and not be 
well approximated by a quadratic model in the parameters).

 

Thus, the COV step in NONMEM is only applied once convergence is obtained 
during the EST step.

 

Wald, A. “Tests of statistical hypotheses concerning several parameters when 
the number of observations is large.” Trans. Amer. Math. Soc. 1943;54:426.

 

Best,

 

Ken

 

Kenneth G. Kowalski

President

Kowalski PMetrics Consulting, LLC

Email: kgkowalsk...@gmail.com <mailto:kgkowalsk...@gmail.com> 

Cell:  248-207-5082

 

 

From: owner-nmus...@globomaxnm.com <mailto:owner-nmus...@globomaxnm.com>  
mailto:owner-nmus...@globomaxnm.com> > On Behalf 
Of Santosh
Sent: Friday, July 26, 2024 3:38 AM
To: nmusers@globomaxnm.com <mailto:nmusers@globomaxnm.com> 
Subject: [NMusers] Obtaining RSE%

 

 Dear esteemed experts!

When using one or more estimation methods & covariance step in a NONMEM control 
stream, the resulting ext file contains final estimate (for all estimation 
steps)  & standard error (only for the last estimation step). 

 

Is there a way that standard error is generated for every estimation step?

 

TIA

Santosh 

Re: [NMusers] Obtaining RSE%

[Leonid Gibiansky]

It can be done if you add extra $cov statement after each estimation method
record
Thank you
Leonid


On Sat, Jul 27, 2024, 12:47 PM  wrote:

> Dear Santosh,
>
>
>
> There is a good reason for this.  Wald (1943) has shown that the inverse
> of the Hessian (R matrix) evaluated at the maximum likelihood estimates is
> a consistent estimator of the covariance matrix.  It is based on Wald’s
> approximation that the likelihood surface locally near the maximum
> likelihood estimates can be approximated by a quadratic function in the
> parameters.  This theory does not hold for any set of parameter estimates
> along the algorithm’s search path prior to convergence to the maximum
> likelihood estimates.  Moreover,  inverting the Hessian evaluated at an
> interim step prior to convergence would likely be a poor approximation
> especially early in the search path where the gradients are large (i.e.,
> large changes in OFV for a given change in the parameters would probably
> have substantial curvature and not be well approximated by a quadratic
> model in the parameters).
>
>
>
> Thus, the COV step in NONMEM is only applied once convergence is obtained
> during the EST step.
>
>
>
> Wald, A. “Tests of statistical hypotheses concerning several parameters
> when the number of observations is large.” *Trans. Amer. Math. Soc.*
> 1943;54:426.
>
>
>
> Best,
>
>
>
> Ken
>
>
>
> Kenneth G. Kowalski
>
> President
>
> Kowalski PMetrics Consulting, LLC
>
> Email: kgkowalsk...@gmail.com
>
> Cell:  248-207-5082
>
>
>
>
>
> *From:* owner-nmus...@globomaxnm.com  *On
> Behalf Of *Santosh
> *Sent:* Friday, July 26, 2024 3:38 AM
> *To:* nmusers@globomaxnm.com
> *Subject:* [NMusers] Obtaining RSE%
>
>
>
>  Dear esteemed experts!
>
> When using one or more estimation methods & covariance step in a NONMEM
> control stream, the resulting ext file contains final estimate (for all
> estimation steps)  & standard error (only for the last estimation step).
>
>
>
> Is there a way that standard error is generated for every estimation step?
>
>
>
> TIA
>
> Santosh
>

RE: [NMusers] Obtaining RSE%

[kgkowalski58]

Dear Santosh,

 

There is a good reason for this.  Wald (1943) has shown that the inverse of the 
Hessian (R matrix) evaluated at the maximum likelihood estimates is a 
consistent estimator of the covariance matrix.  It is based on Wald’s 
approximation that the likelihood surface locally near the maximum likelihood 
estimates can be approximated by a quadratic function in the parameters.  This 
theory does not hold for any set of parameter estimates along the algorithm’s 
search path prior to convergence to the maximum likelihood estimates.  
Moreover,  inverting the Hessian evaluated at an interim step prior to 
convergence would likely be a poor approximation especially early in the search 
path where the gradients are large (i.e., large changes in OFV for a given 
change in the parameters would probably have substantial curvature and not be 
well approximated by a quadratic model in the parameters).

 

Thus, the COV step in NONMEM is only applied once convergence is obtained 
during the EST step.

 

Wald, A. “Tests of statistical hypotheses concerning several parameters when 
the number of observations is large.” Trans. Amer. Math. Soc. 1943;54:426.

 

Best,

 

Ken

 

Kenneth G. Kowalski

President

Kowalski PMetrics Consulting, LLC

Email: kgkowalsk...@gmail.com  

Cell:  248-207-5082

 

 

From: owner-nmus...@globomaxnm.com  On Behalf Of 
Santosh
Sent: Friday, July 26, 2024 3:38 AM
To: nmusers@globomaxnm.com
Subject: [NMusers] Obtaining RSE%

 

 Dear esteemed experts!

When using one or more estimation methods & covariance step in a NONMEM control 
stream, the resulting ext file contains final estimate (for all estimation 
steps)  & standard error (only for the last estimation step). 

 

Is there a way that standard error is generated for every estimation step?

 

TIA

Santosh