Leonid wrote privately to Mark but Mark posted to nmusers:
"We were discussing the usefulness of nonmem error messages: what to do
if $COV failed, or even if estimation step has not converged
successfully (e.e., infinite OF message).
Nick point is that we just ignore the error messages.
My point is that we erro messages prompt us to study the model because
more often than not, error messages point out to real problem (although
sometimes they need to be ignored if you are happy with the model)."
Nick replied:
My point is not "ignore the error messages". It is "Do not use the
termination messages as a guide to whether the model is good or bad."
(Note that NONMEM does not list them as ERROR messages. They are simply
messages about NONMEM's view of the world when it decided to finish the
estimation step. Some messages can be ignored (ROUNDING ERRORS) while
others (EXCEEDED NUMBER OF FUNCTION EVALUATIONS) probably mean you
should restart the model where it finished and keep going. Other
messages in NMVI about boundary conditions are usually just a nuisance
but if you did happen to be asleep and do not look at your parameter
estimates then this is a reminder to wake up.
Leonid suggests we use these messages to examine the model. But there is
no clue in these messages as to which part of the model (or the data)
should be examined. So they are worthless except to remind you that you
should be thinking about your model and data.
But you MUST think about the model and the data ALWAYS! It makes no
difference what termination message you get you must continue to think
(the hard part <grin>) and remember the advice of Box:
"All models are wrong but some are useful".
NONMEM has no idea if your model is wrong. It is always wrong but it
seems Leonid is mislead into thinking it is not wrong when NONMEM says
MINIMIZATION SUCCESSFUL.
NONMEM especially has no idea if your model is useful. Only you and your
colleagues who want to use the results of the modelling can decide if
its useful. Usefulness can be investigated by model evaluation
procedures (e.g. VPC, NPDE, etc) but the final decision will rest with a
human brain not NONMEM's randomly generated minimization messages.
Nick
Mark Sale - Next Level Solutions wrote:
Leonid,
I agree with your point that failure to converge/and or covariance
is a message that the model is a prompt to study the model. I object
to those who claim that model that fails covariance is not useful
despite data to the contrary (just went around and around with a
sponsor about this - actually their stats consultant who basically
just kept insisting on the theory regardless of data that we presented
to the contrary). But, I think that the messages are completely
non-specific - they tell you something is less than ideal, but give no
clue as to what. I suspect that graphics are likely to be much more
consistently informative, telling you not only that something is less
than ideal, but some clue what to do to fix it. As such, I'm not sure
that convergence and covariance messages add anything to the process
(anything that a good and thorough analyst would have known already,
based on VPC, NPC, various post hoc plots etc).
Mark
Mark Sale MD
Next Level Solutions, LLC
www.NextLevelSolns.com <http://www.nextlevelsolns.com/>
919-846-9185
-------- Original Message --------
Subject: RE: [NMusers] Models that abort before convergence
From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Date: Wed, November 19, 2008 10:23 pm
To: [EMAIL PROTECTED]
Mark,
I am sorry, I simply do not understand what you are saying. I do
not want
to bother the group, it could be that I am the only one who is
missing your
point, but could you repeat what exactly you are trying to say?
We were discussing the usefulness of nonmem error messages: what
to do if
$COV failed, or even if estimation step has not converged successfully
(e.e., infinite OF message).
Nick point is that we just ignore the error messages.
My point is that we erro messages prompt us to study the model
because more
often than not, error messages point out to real problem (although
sometimes they need to be ignored if you are happy with the
model). What is
your opinion?
Thanks
Leonid
Original Message:
-----------------
From: Mark Sale - Next Level Solutions [EMAIL PROTECTED]
Date: Wed, 19 Nov 2008 06:48:36 -0700
To: [email protected]
Subject: RE: [NMusers] Models that abort before convergence
<!-- wmLetter_head_start -->
<table align="center" style="empty-cells: show;" bgcolor="#ffffff"
border="0" cellpadding="0" cellspacing="0" width="100%"><tr><td
valign="top" style="text-align: left;" align="left" width="108"><img
src="cid:left@b8c2bfa43541d55928145888195cae14"></td><td
style="vertical-align: top;"><div style="padding: 5px; overflow-x:
auto;"><!-- wmLetter_head_end --><html><body><span
style="font-family:Verdana; color:#000000; font-size:10pt;">Leonid et
al,<br><br> I'm a little confused by this discussion. To make an
analogy, assume that drug company A has a wonderful theory that
drug B will
treat a disease. Theory makes sense by your favorite epistemology
criteria
etc. But of course, being good scientists, we know that theories
must be
verified, so we do an experiment, and the data suggest that the
theory is
wrong. Most of us would criticize as unscientific someone who who
discarded the data (didn't point out flaws in the data, didn't provide
opposing data, simply discounted it) in favor of continuing to
believe the
theory.<br> Why do we not apply the same standards here? Theory says
that models that do not converge (or fail covariance) are "bad". Data
(that so far as I know no one has found to be flawed, nor provided
opposing
data) suggests that, by at least one criteria (same parameter
estimates,
same SD of parameter estimates) there are no important differences. I
don't disagree that failing a covariance step, or failing to converge
provide information about a model. But it doesn't seem to be
informative
about what we probably really care about -does the line go through the
points, how confident are we WRT the precision of the parameters
and is the
model predictive.<br> I'm not sure if the small number of published
examples (of bootstrap with ~500 samples) are a small number of
anecdotes
or a small number of trials with N ~ 500, but I've run 5 or so
myself and
found the same to be consistently the case. That is, a successful
covariance step is not informative WRT the parameter values or their
precision. I suspect others have similar experience. If there are
other
"studies"/anecdotes with different conclusions, someone should publish
them. Otherwise, it seems like we are obligated to abandon this
theory in
favor of the data.<br><br> <br><br><br>Mark Sale MD<br>
Next Level Solutions, LLC<br>
<a href="http://www.NextLevelSolns.com
<http://www.nextlevelsolns.com/>"
mce_href="http://www.NextLevelSolns.com
<http://www.nextlevelsolns.com/>">www.NextLevelSolns.com</a
<http://www.nextlevelsolns.com%3c/a>><br>
919-846-9185<br><br>
<blockquote webmail="1" style="border-left: 2px solid blue;
margin-left:
8px; padding-left: 8px; font-size: 10pt; color: black; font-family:
verdana;">
<div >
-------- Original Message --------<br>
Subject: RE: [NMusers] Models that abort before convergence<br>
From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]><br>
Date: Tue, November 18, 2008 11:13 pm<br>
To: [EMAIL PROTECTED], [email protected],<br>
[EMAIL PROTECTED]<br>
<br>
Dennis,<br>
I do not support extreme views (from places where people walk upside
down<br>
:) ) that Nonmem error messages should be ignored: they serve the
useful<br>
purpose to alert when Nonmem is having some difficulties, and should
always<br>
be part of the picture. If the data looks good, model is simple,
then we<br>
need to look for the reason for the poor convergence. Sometimes it
helps
to<br>
use SIGDIG= 5 or 6 to get 3 significant digits precision. But if
you are<br>
working on the limit of the algorithms (as implemented) abilities:<br>
nonlinear model + stiff differential equations + large range of doses
and<br>
concentrations, etc., then you face the situation when you cannot
force<br>
convergence even if you try hard. On my recent project, none of
the<br>
intermediate model converged even though bootstrap provided pretty
narrow<br>
CI (so it does not look like over-parametrized model), all diagnostic
plots<br>
were good, and the visual predictive check was reasonable. Then
you just<br>
blame the algorithm and move on. You loose the ability to justify
your<br>
covariate selection based on the objective function drop (which is
not a<br>
good idea any way), and may need to provide a little bit more
detailed<br>
investigation to convince reviewers (regulatory and/or journal)
that the<br>
model is adequate for the intended purpose. <br>
Thanks<br>
Leonid<br>
<br>
<br>
Original Message:<br>
-----------------<br>
From: Dennis Fisher [EMAIL PROTECTED]<br>
Date: Tue, 18 Nov 2008 11:21:23 -0800<br>
To: [email protected], [EMAIL PROTECTED]<br>
Subject: [NMusers] Models that abort before convergence<br>
<br>
<br>
Colleagues,<br>
<br>
I am curious as to your thoughts about a particular NONMEM issue.
I <br>
often find myself in a situation where a complex model does not <br>
converge to 3 digits ("no of digits: unreportable") yet the
objective <br>
function is markedly better than a previous model and graphics
suggest <br>
that the model is quite good (and better than the previous one).
Nick <br>
Holford has advocated (and I agree) that NONMEM's SE's have
minimal <br>
utility and the inability to calculate them is not important. <br>
However, I have not seen similar discussion about whether one can
/ <br>
should accept a model that did not converge.<br>
<br>
The particular situation that I dealing with at the moment is that
a <br>
dataset that I am analyzing yielded a series of results that did
not <br>
converge as I added parameters (despite an improving fit and a
marked <br>
decrease in the objective function), then yet a more complicated
model <br>
yielded 3.0 significant digits. In this case, there is no problem
(I <br>
can use this final model for bootstrap, VPC, etc.) but what if
none of <br>
these models had converged.<br>
<br>
Dennis<br>
<br>
Dennis Fisher MD<br>
P < (The "P Less Than" Company)<br>
Phone: 1-866-PLessThan (1-866-753-7784)<br>
Fax: 1-415-564-2220<br>
<a href="http://www.PLessThan.com <http://www.plessthan.com/>"
target="_blank"
mce_href="http://www.PLessThan.com
<http://www.plessthan.com/>">www.PLessThan.com</a
<http://www.plessthan.com%3c/a>><br>
<br>
<br>
--------------------------------------------------------------------<br>
mail2web.com - Microsoft® Exchange solutions from a leading
provider -<br>
<a href="http://link.mail2web.com/Business/Exchange"; target="_blank"
mce_href="http://link.mail2web.com/Business/Exchange";>http://link.mail2web.c
<http://link.mail2web.c/>
om/Business/Exchange</a><br>
<br>
<br>
</div>
</blockquote></span></body></html><!-- wmLetter_tail_start
--></div></td></tr></table><!-- wmLetter_tail_end -->
--------------------------------------------------------------------
mail2web.com - Microsoft® Exchange solutions from a leading provider -
http://link.mail2web.com/Business/Exchange
--
Nick Holford, Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
[EMAIL PROTECTED] tel:+64(9)923-6730 fax:+64(9)373-7090
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
