RE: [NMusers] OMEGA selection

[Mark Sale - Next Level Solutions]

Yaming,   For details, I'd refer you to the abstracts, I've never published this.  But, whenever I do a bootstrap I look at whether the samples that had a successful covariance step are different (in mean or variability), just for my own interest.  They never have been different, I'd guess I've looked at 6 or so.  I have no records of what fraction of samples had a successful covariance step. I'd also refer to any number of good reference on how to decide if a model is "good" (plots, biological plauability, reasonable parameters, various metrics of "goodness". etc.  I'd suggest that if your parameters are poorly defined by the data (e.g., all concentrations near EMAX, unable to define EC50) you'll invariably find that other metrics suggest lack of model goodness.  Whether and how successful covariance or minimization fits into this will have to wait until we have a universally accepted metric of model "goodness". I would list CI (based on bootstrap, not $COV) among my metrics of model goodness, I'd even list a successful covariance step among metrics of model goodness - but pretty far down the list. (everything else being equal, I'd prefer a model that has a successful covariance step - of course everything else is never equal). Mark Sale MD Next Level Solutions, LLC www.NextLevelSolns.com 919-846-9185 -------- Original Message -------- Subject: RE: [NMusers] OMEGA selection From: "Hang, Yaming" <yaming_h...@merck.com> Date: Thu, April 23, 2009 10:35 am To: "Mark Sale - Next Level Solutions" <m...@nextlevelsolns.com> Cc: <nmusers@globomaxnm.com> Hi Mark,   Very interesting point. In general, your logic about why the covariance step doesn't matter in the bootstrapping case makes sense to me. However, I have some questions about why such a conclusion was reached. My questions are: 1. how many data sets are bootstrapped, 2. among them, what's the frequency of failed vs. successful covariance step, 3. are parameter estimates themselves similar across different bootstraps, 4. are there any major difference among the data sets leading to successful and failed covariance step?   I am imagining an example: with an Emax model, I generate two data sets, one with good distribution with regard to the X variable (say concentration) and the other with ill distribution. So that the first data set gives me a successful run including $COV step with reasonable estimates for Emax and EC50, the second data set will lead to a total failure in estimation, even estimates for Emax and EC50 cannot be obtained. I guess I cannot use this as a basis to conclude that even the $ESTIMATE step is not reliable, since both data sets are coming from the same population, right?   I'd love to hear your thoughts on this one.   Thanks, Yaming From: owner-nmus...@globomaxnm.com [mailto:owner-nmus...@globomaxnm.com] On Behalf Of Mark Sale - Next Level Solutions Sent: Wednesday, April 15, 2009 1:00 PM Cc: nmusers@globomaxnm.com Subject: RE: [NMusers] OMEGA selection Nick et al.     At this risk of starting an discussion that probably has little mileage left in it.  First I agree with Nick on covariance - it probably doesn't matter.  But, I'd like to point out what may be an error in our logic.  We content that we have demonstrated that covariance doesn't matter.  Our evidence is that, when bootstrapping, the parameters for the sample that have successful covariance are not different from those that failed.  So, we conclude that the results are the same regardless of covariance outcome across sampled data sets - the independent variable in this test is the data set, the model is fixed. In model selection/building, we have a fixed data set and the independent variable is the model structure.   Whether covariance success is a useful predictor across different models with a fixed data set is a different question than whether covariance is a useful predictor across data sets with a fixed model. But, in the end, I do agree that biological plausibility, diagnostic plots, reasonable parameters and some suggestion of numerical stability/identifiably (such as bootstrap CIs) are more important than a successful covariance step. Mark Mark Sale MD Next Level Solutions, LLC www.NextLevelSolns.com 919-846-9185 -------- Original Message -------- Subject: Re: [NMusers] OMEGA selection From: Nick Holford <n.holf...@auckland.ac.nz> Date: Wed, April 15, 2009 12:17 pm To: nmusers@globomaxnm.com Ethan, Do not pay any attention to whether or not the $COV step runs or even if the run is 'SUCCESSFUL' to conclude anything about your model. Your opinion is not supported experimentally e.g. see http://www.mail-archive.com/nmusers@globomaxnm.com/msg00454.html for discussion and references. NONMEM has no idea if the parameters make sense or not and will happily converge with models that are overparameterised. You cannot rely on a failed $COV step or a MINIMIZATION TERMINATED message to conclude the model is not a good one. You need to use your brains (NONMEM does not have a brain) and your common sense to decide if your model makes sense or is perhaps overparameterised. Nick Ethan Wu wrote: > > Dear all, > > I am fitting a PD response, and the equation goes like this: > > total response = baseline+f(placebo response) +f(drug response) > > first, I tried full omega block, and model was able to converge, but > $COV stop failed. > > To me, this indicates that too many parameters in the model. The > structure model is rather simple one, so I think probably too many Etas. > > I wonder is there a good principle of Eta reduction that I could > implement here. Any good reference? > > -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand n.holf...@auckland.ac.nz tel:+64(9)923-6730 fax:+64(9)373-7090 mobile: +33 64 271-6369 (Apr 6-Jul 17 2009) http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford Notice: This e-mail message, together with any attachments, contains information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station, New Jersey, USA 08889), and/or its affiliates (which may be known outside the United States as Merck Frosst, Merck Sharp & Dohme or MSD and in Japan, as Banyu - direct contact information for affiliates is available at http://www.merck.com/contact/contacts.html) that may be confidential, proprietary copyrighted and/or legally privileged. It is intended solely for the use of the individual or entity named on this message. If you are not the intended recipient, and have received this message in error, please notify us immediately by reply e-mail and then delete it from your system.