Nick,
Could you elaborate on how you reason around the necessity of showing a
priori power when you find a significant effects from the study data? How
would you show it?
Best regards,
Mats
Mats Karlsson, PhD
Professor of Pharmacometrics
Dept of Pharmaceutical Biosciences
Uppsala University
Box 591
751 24 Uppsala Sweden
phone: +46 18 4714105
fax: +46 18 471 4003
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Nick Holford
Sent: Tuesday, August 25, 2009 11:54 PM
To: nmusers
Subject: Re: [NMusers] What does convergence/covariance show?
Mats,
You are right - I replied before the coffee had started working so I was
indeed in a strange world!
Nevertheless the isolated finding of P<0.05 should not be uncritically
interpreted as being of clinical relevance without other considerations
such as adequate a priori power and if possible some plausible mechanism
even if the P value suggests an increased hazard of death.
Nick
Mats Karlsson wrote:
Nick,
You're living in a strange world if killing patients is benefit :)
Mats
Mats Karlsson, PhD
Professor of Pharmacometrics
Dept of Pharmaceutical Biosciences
Uppsala University
Box 591
751 24 Uppsala Sweden
phone: +46 18 4714105
fax: +46 18 471 4003
*From:* Nick Holford [mailto:[email protected]]
*Sent:* Tuesday, August 25, 2009 11:15 PM
*To:* Mats Karlsson
*Subject:* Re: [NMusers] What does convergence/covariance show?
Mats,
If the trial was powered to test the effect of the treatment on
survival then I would think that it would be reasonable to consider
some practical consequences. However, FDA would not accept one trial
alone as evidence of benefit without other strong supporting evidence
from a different trial i.e. the OFV alone is not enough to accept
clinical importance.
Nick
Mats Karlsson wrote:
Nick,
If the hazard of patients are dying is significantly (p<0.05) higher on
the
new treatment compared to reference, I don't think you need other evidence
before it has practical consequences. Without mechanistic understanding,
would you ignore it and move on to the next analysis?
Mats
Mats Karlsson, PhD
Professor of Pharmacometrics
Dept of Pharmaceutical Biosciences
Uppsala University
Box 591
751 24 Uppsala Sweden
phone: +46 18 4714105
fax: +46 18 471 4003
-----Original Message-----
From: [email protected] <mailto:[email protected]>
[mailto:[email protected]] On
Behalf Of Nick Holford
Sent: Tuesday, August 25, 2009 10:29 PM
To: nmusers
Subject: Re: [NMusers] What does convergence/covariance show?
Mats,
Thanks for stating more clearly what I tried to say before. Once again
-- I agree that OFV is not a measure of clinical importance. But it is
correlated with discernible differences in model predictions that may be
of clinical importance.
A change of OFV of 5 in a survival model may well be useful to reject a
null hypothesis and point to some explanatory variable. There are
numerous 'statistically significant' findings in the clinical literature
like this that have no practical impact. You do not indicate what else
in the survival analysis convinced you that the OFV was associated with
something of practical consequence. I trust your decision was not based
only on the OFV!
Nick
Mats Karlsson wrote:
Nick,
I agree that small changes (5-10) in OFV often are not practically
important
and and big changes more often are. However, my point is that OFV is
not
the
right scale to judge importance. You should judge it on the
consequence of
you additional complexity to the model (the magnitude of the found
drug
effect/covariate/etc). Just the other day did I analyze survival data
where
a small (5) change in OFV is of practical consequence.
A true treatment effect of a certain size will improve the OFV in
relation
to the size of the dataset. The larger the data set, the larger the
change
in OFV. However, the estimate of the treatment effect does not change
systematically with the size of the data set. The size of the
treatment
effect is what is more appropriate diagnostic for practical
consequences.
OFV we would use only to make sure that we have found the effect by
chance.
Best regards,
Mats
Mats Karlsson, PhD
Professor of Pharmacometrics
Dept of Pharmaceutical Biosciences
Uppsala University
Box 591
751 24 Uppsala Sweden
phone: +46 18 4714105
fax: +46 18 471 4003
-----Original Message-----
From: [email protected]
<mailto:[email protected]> [mailto:[email protected]]
On
Behalf Of Nick Holford
Sent: Tuesday, August 25, 2009 7:25 AM
To: nmusers
Subject: Re: [NMusers] What does convergence/covariance show?
Mats,
When I referred to a change of 50 being needed to detect something of
practical importance I was not saying that was of clinical relevance.
That cannot be judged from the OFV alone. But small OFV changes are
rarely if ever indicators of something that is clinically relevant.
I expect you will agree on this point :-)
Nick
Mats Karlsson wrote:
Nick,
I too would use OFV as the most important goodness-of-fit
diagnostic when
comparing models, especially when deeming something to be
redundant. If
adding a component doesn't reduce OFV, I see no reason to include
it (I
think we're agreeing on something!). However, you write
" Small (5-10) changes in OBJ are not of much interest. A change
of OBJ
of
at least 50 is usually needed to detect anything of practical
importance."
Today we use population methods for everything from very rich pop
pk
meta-analyses to very sparsely informative data sets on survival.
To use
OFV
as a measure of goodness-of-fit is central and look at the risk
something
improved the fit by chance, but I would not use it as measure of
clinical
importance.
Best regards,
Mats
Mats Karlsson, PhD
Professor of Pharmacometrics
Dept of Pharmaceutical Biosciences
Uppsala University
Box 591
751 24 Uppsala Sweden
phone: +46 18 4714105
fax: +46 18 471 4003
-----Original Message-----
From: [email protected]
<mailto:[email protected]> [mailto:[email protected]]
On
Behalf Of Nick Holford
Sent: Tuesday, August 25, 2009 12:14 AM
To: nmusers
Subject: Re: [NMusers] What does convergence/covariance show?
Mats, Leonid,
Thanks for your definitions. I think I prefer that provided by
Mats but
he doesn't say what his test for goodness-of-fit might be.
Leonid already assumes that convergence/covariance are diagnostic
so it
doesnt help at all with an independent definition of
overparameterization. Correlation of random effects is often a
very
important part of a model -- especially for future predictions --
so I
dont see that as a useful test -- unless you restrict it to
pathological
values eg. |correlation|>0.9?. Even with very high correlations I
sometimes leave them in the model because setting the covariance
to zero
often makes quite a big worsening of the OBJ.
My own view is that "overparameterization" is not a black and
white
entity. Parameters can be estimated with decreasing degrees of
confidence depending on many things such as the design and the
adequacy
of the model. Parameter confidence intervals (preferably by
bootstrap)
are the way i would evaluate how well parameters are estimated. I
usually rely on OBJ changes alone during model development with a
VPC
and boostrap confidence interval when I seem to have extracted all
I can
from the data. The VPC and CIs may well prompt further model
development
and the cycle continues.
Nick
Leonid Gibiansky wrote:
Hi Nick,
I am not sure how you build the models but I am using
convergence,
relative standard errors, correlation matrix of parameter
estimates
(reported by the covariance step), and correlation of random
effects
quite extensively when I decide whether I need extra
compartments,
extra random effects, nonlinearity in the model, etc. For me
they are
very useful as diagnostic of over-parameterization. This is
the direct
evidence (proof?) that they are useful :)
For new modelers who are just starting to learn how to do it,
or have
limited experience, or have problems on the way, I would
advise to pay
careful attention to these issues since they often help me to
detect
problems. You seem to disagree with me; that is fine, I am not
trying
to impose on you or anybody else my way of doing the analysis.
This is
just an advise: you (and others) are free to use it or ignore
it :)
Thanks
Leonid
Mats Karlsson wrote:
<<I would say that if you can remove parameters/model
components without
detriment to goodness-of-fit then the model is
overparameterized. >>
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New
Zealand
[email protected] <mailto:[email protected]>
tel:+64(9)923-6730 fax:+64(9)373-7090
mobile: +64 21 46 23 53
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
