Mohamed,
When the number of subjects is small then any confidence interval is
going to be wide and probably no-one is really interested in it. With
studies more suitable for population analysis (at least 25 subjects and
preferably over 100 if you want to look for covariate effects) then the
CIs may be more interesting.
With linear models or parameters which are nearly linear in non-linear
models then I would expect quite good agreement between CIs obtained by
bootstrapping and by using NONMEM SEs. But the models get interesting
when one tries to estimate non-linear parameters e.g. EC50 in an Emax
model. In that case the CIs will often be asymmetrical and the normal
distribution assumption used to compute SEs from CIs will be wrong.
Leonid does not discuss the issue of assymetry of CIs in his poster --
but when I look at Figure 3 I see evidence for disagreement between
bootstrap and NONMEM SE based CIs. The scatter of bootstrap points
relative to the solid line shows an excess of bootstrap upper CI values
above the SE prediction. For the lower CI prediction there also seem to
be more bootstrap values above the SE predictions. Its hard to be sure
that these upper and lower bootstrap predictions belong to the same
parameters but if so this would be evidence for asymmetry of the
bootstrap CI. This is exactly what one would expect because the SE
method has to assume symmetrical CIs yet the bootstrap estimate is not
restricted in this way.
I think this poster is a nice example of why correlation coefficients
are a very poor way to compare predictions (as pointed out by Sheiner
and Beal in their classic paper Sheiner LB, Beal SL. Some Suggestions
for Measuring Predictive Performance. J Pharmacokinet Biopharm.
1981;9(4):503-12.). A better way would be to compute the prediction
error for the absolute larger CI arm and smaller CI arm obtained by
bootstrapping to the symmetrial CI from the SE. If bootstraps CIs are
indeed asymmetrical then there would be a difference shown by the mean
prediction error ('bias'). Note that I use absolute value larger and
smaller CI arm to refer to the larger or smaller part of the CI that is
constructed around zero. I dont mean the upper and lower parts of the CI
interval.
Best wishes,
Nick
Leonid Gibiansky wrote:
I cannot make any general statements but here is the summary of the 13
different models that I tested for comparison of bootstrap and nonmem CI.
http://www.quantpharm.com/pdf_files/2572-GibianskyPage2007Poster2007final.pdf
Note that all bootstrap samples were appropriately stratified by major
covariates (such as study, dose, weight as necessary, etc.).
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
[EMAIL PROTECTED] wrote:
Dear Dr. Holford,
Please correct me if I am wrong, however my understanding is that
asymptotic distribution implied by NONMEM's covariance step
approaches normality as the sample size gets larger or we have more
data. However, a non parametric bootstrap distribution may have poor
coverage with a small sample size as well, since it relies on
sampling subjects with repalcement in the data set. So both
distributions have problems when sample size is small (e.g. N<30).
Therefore I would think when N is large the wald based Confidence
Intervals from NONMEM are appropriate enough. It would be helpful to
know the criteria when generating a non parametric bootstrap
distribution is really advantageous.
Thanks, Mohamed
Quoting Nick Holford <[EMAIL PROTECTED]>:
--
Nick Holford, Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
[EMAIL PROTECTED] tel:+64(9)373-7599x86730 fax:+64(9)373-7090
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
