Dear Joanna,
One option would be to use 95%CI of the covariate parameter estimates.
IMP method should provide you with the estimate and SEs that can be used
to compute those 95%CI (if you would like to test at 0.05 level). You
may say that the effect is significant if 95% CI does not include the
null value. Alternatively, you may discuss it in terms of the effect
value and uncertainty of the estimate.
If you would like to use OF directly, you may increase the ISAMPLE
number to 3000 or even higher: this should decrease stochastic
fluctuation of the OF. If you open the file root.cvn where root is the
name of the control stream, you will find there the OBJ value and SD
(averaged over the iterations). Those values can be used for model
comparison.
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
On 9/16/2013 4:36 PM, Lewis, Joanna wrote:
Dear NMusers,
I have been trying to test for covariate effects in a complex model
described by stiff ODEs, and with poorly-defined parameter values.
I have found by trial and error that FOCE has difficulty fitting my
model when covariate effects are included so I have been using SAEM for
parameter estimation, followed by an importance sampling step for
evaluating the objective function. I have then compared OFVs obtained by
importance sampling, comparing the difference in OFV between two models
to a chi-squared distribution.
Initially, I used the following commands for estimation and OFV
evaluation, with five importance sampling iterations as suggested in the
NONMEM 7.2.0 user guide:
$ESTIMATION METHOD=SAEM INTER PRINT=1 NBURN=1000 ISAMPLE=2 NITER=500 CTYPE=3
$ESTIMATION METHOD=IMP EONLY=1 ISAMPLE=1000 NITER=5
However, when I examined the OFV at each of the six iterations 1-5 I
found that the OFV seemed still to be decreasing so I increased NITER in
the second $ESTIMATION step to 150:
$ESTIMATION METHOD=SAEM INTER PRINT=1 NBURN=1000 ISAMPLE=2 NITER=500 CTYPE=3
$ESTIMATION METHOD=IMP EONLY=1 ISAMPLE=1000 NITER=150
I found that the objective function drifts over approximately the first
10 iterations and then settles down to values around a constant level,
with a standard deviation of around 3-4 OFV units. Averaged over samples
50-150, some covariate models had lower OFVs than the basic model with
no covariates, but others had higher.
There are two questions I would like to ask the community. Firstly, are
these results what you would expect, or does anything in my description
suggest a problem with my model or the way it has been coded? Secondly,
if OFVs may vary randomly by 3-4 units within a model, and p=0.05
corresponds to 3.84 units if one additional degree of freedom has been
introduced, how can a genuinely better model be distinguished from an
importance sampling step which happened to have a lower OFV?
I would be interested and grateful to hear your thoughts on either or
both questions.
With best wishes
Joanna Lewis
2020 Science Research Fellow
+44 (0)20 7679 5300
http://www.2020science.net/people/joanna-lewis
