Hi,
As pointed out by others I agree it is essential to consider the
existence of random effect correlations if you wish to make model
predictions e.g. to use a VPC to evaluate a model.
I agree with Jeroen that this should be primarily be an informed choice
based on physiology/pharmacology. 'Blue sky' searches for correlations
which when would have no rational explanation or interpretation should
be done with a great deal of caution.
It can be tricky to explore all possible combinations using the change
in OFV (e.g. with the likelihood ratio test) to guide model selection. A
more straightforward approach is to bootstrap the model with a full
covariance block for all the random effects you suspect may be correlated.
Bootstrapping today is usually a practical option because runs can be
easily performed in parallel on multiple processors on the same machine
or on a cluster. I typically use 100 bootstrap replicates for this
purpose and look for correlations which include zero in the 95%
bootstrap confidence interval. If I find such correlations then I know I
should be able to remove those covariances from the covariance block. I
can then re-run the bootstrap and obtain confidence intervals on all the
parameters including the correlations. Confidence intervals calculated
from asymptotic standard errors (if you can get them) are usually
unreliable compared with parametric bootstrap confidence intervals
(http://www.page-meeting.org/default.asp?abstract=3143).
i don't agree with Ken that "ill-conditioning" or "not stable" based on
failure of the $COVARIANCE step should be used to judge the adequacy of
the results. Experimentally it has been shown that the bootstrap
distribution of parameter uncertainty is not different when comparing
runs which terminated and those which were successful or which completed
the $COVARIANCE step.
http://www.mail-archive.com/nmusers%40globomaxnm.com/msg03401.html. See
also
http://holford.fmhs.auckland.ac.nz/docs/bootstrap-and-confidence-intervals.pdf
slides 24 to 31.
Best wishes,
Nick
On 1/10/2014 7:57 a.m., Ken Kowalski wrote:
Hi Jeroen,
I think we might be on the same page but I wanted to get clarification
about your suggestion that we “not apply the concept of
over-parameterization” with respect to evaluating the omega
structure. I’m assuming by ‘over-parameterization’ you mean a model
that has more elements in omega than might be necessary to be
parsimonious. If so, I certainly agree but I wouldn’t call such a
model that has more parameters than necessary to be parsimonious as
necessarily over-parameterized. An over-parameterized model is one in
which there can be an infinite set of solutions to the parameter
values that yields the same fit. Such a setting can occur when the
R-matrix in NONMEM is singular. Such over-parameterized models are
often also referred to as being ill-conditioned or not stable. I
think we should always avoid over-parameterization, ill-conditioning
and unstable models regardless of the source (i.e., fixed effects, IIV
random effects and omega-structure, or residual error structure).
However, I do agree that parsimony in omega is probably not as
important as say looking for a parsimonious set of covariate parameter
fixed effects when performing covariate modeling to obtain a final
model for prediction purposes. This is why in my earlier response
below I suggested fitting the “largest omega structure that can be
supported by the data”. What I meant by this statement is that we
fit the largest number of elements of omega while avoiding
over-parameterization or ill-conditioning. Such an omega structure
might not be parsimonious (i.e., the smallest omega structure that
adequately describes the features in the data). The point I was
trying to make is that the smallest omega structure that adequately
describes the features in the data may not be a diagonal omega
structure (i.e., when correlations do exist) particularly if we are
interested in describing the variation in the data and not just in
predictions of central tendency.
Best,
Ken
*From:*[email protected]
[mailto:[email protected]] *On Behalf Of *Jeroen
Elassaiss-Schaap
*Sent:* Monday, September 29, 2014 7:00 PM
*To:* [email protected]; [email protected];
[email protected]; [email protected]; [email protected]
*Subject:* Re: [NMusers] OMEGA matrix
Dear Pavel, others,
The underlying technical difference is that SAEM is in its core a
sampling methodology. Off-diagonal elements (as explained by Bob
Bauer) are available as sample correlations and do not have to be
separately computed in contrast to linearization approaches such as FOCE.
The more interesting question to me, as also eluted to by Ken, is what
criteria to set up for inclusion of an off-diagonal element. I
completely support his argument for simulation performance of the
model, as e.g. judged using a VPC. Whether to score it as an
additional degree of freedom may be up to debate. An off-diagonal
element in essence limits the freedom of the model as the random space
in which samples can be generated will be smaller. In that perspective
one could argue to retain any off-diagonal element that is
sufficiently deviating from zero regardless of ofv changes, and to not
apply the concept of over-parametrization (or at least not in
comparison to other types of parameters). In practice inclusion of an
important off-diagonal is mostly accompanied by a sound improvement in
ofv anyway.
More can be found in earlier discussions we had on this list, see e.g.
https://www.mail-archive.com/[email protected]/msg02736.html for
quite an extensive one from 2010. Here also an r-script to visualize
the parameter space impact can be found ;-).
In cases where a larger full or banded omega block is found, I would
advice to explore its properties further using matrix decomposition
approaches (PCA etc) to evaluate propagated correlations across the
matrix. But also on the basis of physiology/pharmacology as a data
sample may not be informative enough to support robust interpretation
of correlations. A discussion along those lines in reporting seems the
more fruitful to me.
Best regards,
Jeroen
http://pd-value.com
-- More value out of your data!
-----Original Message-----
From:[email protected] <mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Standing Joseph (GREAT ORMOND
STREET HOSPITAL FOR CHILDREN NHS FOUNDATION TRUST)
Sent: Friday, September 26, 2014 09:15
To: Kowalski, Ken; 'Eleveld, DJ'; 'Pavel Belo';[email protected]
<mailto:[email protected]>
Subject: RE: [NMusers] OMEGA matrix
Dear Pavel,
To answer your question I suggest you go on Bob Bauer's NONMEM 7 course.
The understanding I gleaned from that course (which I think was enhanced by the
excellent wine we had at lunch in Alicante) was that with appropriate MU
parameterisation there is virtually no computational disadvantage to estimating
the full block with the newer algorithms. So you might as well do it, at least
in early runs where you want an idea of which parameter correlations might be
useful/reasonably estimated.
BW,
Joe
Joseph F Standing
MRC Fellow, UCL Institute of Child Health
Antimicrobial Pharmacist, Great Ormond Street Hospital
Tel: +44(0)207 905 2370
Mobile: +44(0)7970 572435
------------------------------------------------------------------------
From:[email protected] <mailto:[email protected]>
[[email protected] <mailto:[email protected]>] On Behalf Of Ken
Kowalski [[email protected] <mailto:[email protected]>]
Sent: 25 September 2014 22:43
To: 'Eleveld, DJ'; 'Pavel Belo';[email protected]
<mailto:[email protected]>
Subject: RE: [NMusers] OMEGA matrix
Warning: This message contains unverified links which may not be safe. You
should only click links if you are sure they are from a trusted source.
Hi Douglas,
My own thinking is that you should fit the largest omega structure that can
be supported by the data rather than just always assuming a diagonal omega
structure. This does not necessarily mean always fitting a full block omega
structure, as it can often lead to an ill-conditioned model, however, there
may be a reduced block omega structure that is more parsimonious than the
diagonal omega structure. Getting the omega structure right is particularly
important for simulation of individual responses. For example, if you
always simulate from a diagonal omega structure for CL and V when there is
evidence that the random effects are highly positively correlated then you
may end up simulating individual PK profiles for combinations of individual
CLs and Vs that are not represented in your data (i.e., high correlation
would suggest that individuals with high CL will tend to also have high V
and vice versa whereas a simulation assuming that they are independent will
result in simulating for some individuals with high CL and low V and some
individuals with low CL and high V that might not be represented in your
data). This could lead to simulations that over-predict the variation in
the concentration-time profiles even though the diagonal omega may be
sufficient for purposes of predicting central tendency in the PK profile.
You can confirm this by VPC looking at your ability to predict say the 10th
and 90th percentiles in comparison to the observed 10th and 90th percentiles
in your data. That is, if you simulate from the diagonal omega when there
is correlation in the random effects you may find that your prediction of
the 10th and 90th percentiles are more extreme than that in your observed
data. I see this all the time in VPC plots where the majority of the
observed data are well within the predictions of the 10th and 90th
percentiles when we should expect about 10% of our data above the 90th
percentile prediction and 10% below the 10th percentile prediction.
Best regards,
Ken
Kenneth G. Kowalski
President & CEO
A2PG - Ann Arbor Pharmacometrics Group, Inc.
110 Miller Ave., Garden Suite
Ann Arbor, MI 48104
Work: 734-274-8255
Cell: 248-207-5082
Fax: 734-913-0230
[email protected] <mailto:[email protected]>
www.a2pg.com <http://www.a2pg.com>
-----Original Message-----
From:[email protected] <mailto:[email protected]>
[mailto:[email protected]] On
Behalf Of Eleveld, DJ
Sent: Thursday, September 25, 2014 4:36 PM
To: Pavel Belo;[email protected] <mailto:[email protected]>
Subject: RE: [NMusers] OMEGA matrix
Hi Pavel,
My question is: Why is it desirable to fit a complete omega matrix if its
physical interpretation is unclear? Etas are variation of unknown origin
i.e. not explained by the structural model. A full omega matrix allows the
unknown variation of one paramater to have a (linear?) relationship with
some other thing that is also unknown. If unknown A is found to have a
linear relationship with unknown B, then what knowlegde is gained? I do
think it can be instructive to to look at correlations and use this
information to make a better structural model. But I think diagonal OMEGA
matrix is more desirable if it works ok.
warm regards,
Douglas Eleveld
------------------------------------------------------------------------
From:[email protected] <mailto:[email protected]>
[[email protected] <mailto:[email protected]>] on behalf
of Pavel Belo [[email protected] <mailto:[email protected]>]
Sent: Thursday, September 25, 2014 4:24 PM
To:[email protected] <mailto:[email protected]>
Subject: [NMusers] OMEGA matrix
Hello Nonmem Community,
It seems like NONMEM developers may advise to start with full OMEGA matrix
at the beginning of model development. Monolix developers may advise to
start with a diagonal matrix. Is there something different in NONMEM SAEM
algorithms that makes model stable when a lot of statistically insignificant
correlations/covariances are estimated in the model?
It seems like NONMEM SAEM can be very stable in very "hard cases" (a lot of
outliers, partially misspecified model, overparameterized model, etc.). The
omega matrix is a part of the puzzle.
When it is impossible to test every correlation coefficient for significance
due to some limitations, it becomes a regulatory issue. We may need to be
able to make a statement that the model is safe and sound even when OMEGA
matrix can be overparameterized (tries to estimate too many insignificant
parameters within the OMEGA matrix).
Kind regards,
Pavel
------------------------------------------------------------------------
De inhoud van dit bericht is vertrouwelijk en alleen bestemd voor de
geadresseerde(n). Anderen dan de geadresseerde(n) mogen geen gebruik maken
van dit bericht, het niet openbaar maken of op enige wijze verspreiden of
vermenigvuldigen. Het UMCG kan niet aansprakelijk gesteld worden voor een
incomplete aankomst of vertraging van dit verzonden bericht.
The contents of this message are confidential and only intended for the eyes
of the addressee(s). Others than the addressee(s) are not allowed to use
this message, to make it public or to distribute or multiply this message in
any way. The UMCG cannot be held responsible for incomplete reception or
delay of this transferred message.
********************************************************************************************************************
This message may contain confidential information. If you are not the
intended recipient please inform the
sender that you have received the message in error before deleting it.
Please do not disclose, copy or distribute information in this e-mail or
take any action in reliance on its contents:
to do so is strictly prohibited and may be unlawful.
Thank you for your co-operation.
NHSmail is the secure email and directory service available for all NHS
staff in England and Scotland
NHSmail is approved for exchanging patient data and other sensitive
information with NHSmail and GSi recipients
NHSmail provides an email address for your career in the NHS and can be
accessed anywhere
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology, Bldg 503 Room 302A
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
office:+64(9)923-6730 mobile:NZ +64(21)46 23 53
email: [email protected]
http://holford.fmhs.auckland.ac.nz/
Holford SD, Allegaert K, Anderson BJ, Kukanich B, Sousa AB, Steinman A, Pypendop,
B., Mehvar, R., Giorgi, M., Holford,N.H.G. Parent-metabolite pharmacokinetic models
- tests of assumptions and predictions. Journal of Pharmacology & Clinical
Toxicology. 2014;2(2):1023-34.
Ribba B, Holford N, Magni P, Trocóniz I, Gueorguieva I, Girard P, Sarr,C.,
Elishmereni,M., Kloft,C., Friberg,L. A review of mixed-effects models of tumor
growth and effects of anticancer drug treatment for population analysis. CPT:
pharmacometrics & systems pharmacology. 2014;Accepted 15-Mar-2014.
