Hi Pyry + Martin + NMusers, I wanted to make 2 comments on this discussion. Pyry wrote:
With reference to Martin saying that “A greater problem with these plots is the commonly held expectation that for a "good model" a smooth or regression line should align with the line of unity. Though this seems intuitive it is a flawed assumption”, I would like to defend the assumption that in a relevant number of cases it is reasonable to assume that the plot of observations (y-axis) versus predictions (x-axis) is expected to have a regression line going through unity. Comment 1 I see similarity here between the two positions: DV v IPRED and DV v PRED plots are sometimes misleading - Martin DV v IPRED and DV v PRED plots are often sound - Pyry Both statements are true. There are 4 “errors" we can encounter, as Martin rightly alluded to: 1) The DV v IPRED looks good, but the model is not (slides 19-23 and slide 47). 2) The DV v PRED looks good, but the model is not (slides 19-23 and slide 47). 3) The DV v IPRED looks poor, but the model is right (slide 46). 4) The DV v PRED looks poor, but the model is right. The slide numbers refer to my PAGE 2017 presentation, slides available here (https://www.page-meeting.org/pdf_assets/1203-Maloney%20-%20PAGE%202017%20-%208%20June%2017%20-%20final.pdf) and talk available here (https://www.youtube.com/watch?v=E3T2p6Mv0Xc). The “silly” examples in the presentation simply serve to show how shrinkage and/or model flexibility via random effects can make these graphs misleading, even with linear mixed models. Thus, as always, we need to understand our data, our models, and hence the usefulness of such plots; there is no “golden rule” that they are (or are not) useful. In addition, I personally am “offended" ;-) when an author chooses to present these graphs as “evidence" that their model is sound; either provide/show an extensive set of model diagnostics, PPCs, individual fits, residual analyses, LOO-CV results etc. that truly assess/challenge the model, or do not bother! (rant over!) To comment on DV v PRED, an old adage is that is can support the appropriateness of the “structural” model. This is not true. Consider the following PK data, where the prediction at 2 hr and 12 hr are the same (a flat PK profile!). Dose 2hr 12hr PRED 1 2 0.5 1 10 20 5 10 100 200 50 100 1000 2000 500 1000 Here a plot of DV v PRED would look great (on the log scale), even though the "PK model” is complete nonsense. So again, these basic plots (and the “archaic” value given to them) can be inappropriate. Indeed, simply by adding more and more parameters to our model, we can always “fix” these plots, however foolish these model additions are. Comment 2 I would argue that residuals (like DV - IPRED) versus predicted (or other covariates like time, dose etc.) are much more useful than DV v IPRED, since now our y-axis quantifies the magnitude of the lack of fit; we can much more easily determine the magnitude of the lack of fit for both individual observations and any smooth through the data. A bias may be evident from the smooth, but much more important is the magnitude of that bias. Similarly, having a “flat" smooth is not great if there are many observations being grossly under predicted, but are “balance” by as many observations being grossly over-predicted (...many years ago I recall seeing a “final PK model” WRES v Time plot with a smooth that looked perfectly flat…it looked great, until I saw that the y-axis ran from +20 to -20!). Best wishes, Al Alan Maloney PhD Consultant Pharmacometrician Free Book! - Drug Development For Patients - see www.alanmaloney.com <http://www.alanmaloney.com/> - please read, send comments and support patients! Phone: +46 734 04 38 49 E-mail:al_in_swe...@hotmail.com > On 23 Aug 2023, at 09:35, Pyry Välitalo <pyry.valit...@gmail.com> wrote: > > Dear Martin and NMusers, > > With reference to Martin saying that “A greater problem with these plots is > the commonly held expectation that for a "good model" a smooth or regression > line should align with the line of unity. Though this seems intuitive it is a > flawed assumption”, I would like to defend the assumption that in a relevant > number of cases it is reasonable to assume that the plot of observations > (y-axis) versus predictions (x-axis) is expected to have a regression line > going through unity. > > First, to be clear, I do not disagree with anything said in the classic > Karlsson and Savic 2007 paper. With any model where random effects enter into > the model nonlinearly, the plot of observations (y-axis) versus PRED (x-axis) > can have trends which look like model misspecification, even if the > data-generating model for observations has exactly the same parameter values > as the diagnostics-generating model. This is because the PRED signifies the > prediction for the median individual with random effect values at zero, which > is different from the mean prediction. And the local regression line, as the > name implies, trends around the local mean of the observed data. > > So basically we need a PRED-like data item that reflects the expected mean > population prediction, integrated over the possible individual random effects > values. Lucky for us, this data item exists, and is called EPRED. The data > item was not available at the time the Savic and Karlsson paper was > published. It is available now. The EPRED solves the problems caused by model > nonlinearity and high inter-individual variability by integrating over the > random effects, given the parameter estimates. I do note that it does not > solve the problems of censoring and dose adaptation. > > So, because the local regression line reflects the mean values and the EPRED > data item reflects mean values, I contend that in the absence of censoring > and dose adaptation, the plot of observations (y-axis) versus EPRED (x-axis) > can be expected to have a regression line that mostly agrees with the line of > unity, with some caveats (see below). This expectation holds even if the > observations are not symmetrically distributed over the mean, because the > local regression simply follows the mean. Moreover, if the model accounts for > censoring and dose adaptation, then it would be possible to manually code and > calculate the simulation-predicted population mean values (e.g. simulating > 1000 datasets, and for each observation taking the mean simulated value that > accounts for censoring and dose adaptation), and use those in x-axis. Also to > note, in this NMusers message group I focused on the EPRED data item because > it is NONMEM-specific, but the general concept is software-agnostic: Having > Monte Carlo-generated population mean predictions on the x-axis should result > in the plot of observations (y-axis) versus predictions (x-axis) trending > through the line of unity. > > Caveat 1: Because of random variability, it cannot be expected that the > regression line always goes perfectly through the line of unity. This should > come as no surprise, e.g. it is also not expected that a VPC will have > observed data percentiles always perfectly in the middle of the > simulation-generated confidence intervals for prediction intervals. > > Caveat 2: For small datasets, it is possible that there be additional bias in > the plot of observations (y-axis) versus EPRED (x-axis) if the > data-generating model is exactly the same as the diagnostics-generating > model, because the data-generating model is not necessarily the one that best > agrees with the data. Illustrative example: Suppose we simulate a dataset of > 10 individual concentration-profiles at steady-state with high drug > accumulation, thus the concentrations will be highly dependent on the > clearance parameter. It is entirely possible that the 10 simulated clearance > random effects (eta) values will have a mean that is either above or below > zero to some relevant extent, thus greatly affecting the steady-state > predictions. Thus, as a result there could be an apparent, systematic > disagreement between the simulated data (observations, y-axis) and the EPRED > (x-axis) because of clearance random effects trending above or below zero due > to random variability. This problem could be remedied by fitting a model to > the simulated data, and using that model for generating the diagnostics. At > larger dataset sizes, the problem disappears because it becomes less and less > likely for the mean of the random effects to deviate from zero to a relevant > extent. This same caveat also exists for the VPC diagnostic; if one simulates > a small dataset as observations, and then produces a VPC from the same > simulation model (without fitting the model to the previously simulated > data), then there may be apparent misspecification in the resulting VPC > figure. > > Supplemental remark 1: To illustrate how the loess follows mean even if the > data are not symmetrically distributed, the following R code snippet may be > relevant. It simulates 100 observations from lognormal distribution, and then > compares the smoothing curves from "loess" and "mgcv::gam" functions to the > theoretically expected mean value. There is a close agreement between the > loess curves and the analytically calculated mean value. > library(tidyverse) > with(list(omega=0.6), > map_dfr(1:100,~tibble(x=1:10,y=exp(rnorm(10,0,omega)))) %>% > mutate(theoretical=exp(omega^2/2)) %>% > ggplot(aes(x,y))+geom_point()+ > geom_smooth(method="loess",col=3)+geom_smooth(method=mgcv::gam),col=4)+ > geom_line(aes(y=theoretical),col=2) > > ps. The usual disclaimer, the opinions expressed in this message are mine > alone, and not necessarily those of my employer. > > Best wishes, > Pyry Välitalo > PK Assessor at Finnish Medicines Agency > > On Fri, 18 Aug 2023 at 10:59, Martin Bergstrand > <martin.bergstr...@pharmetheus.com > <mailto:martin.bergstr...@pharmetheus.com>> wrote: >> Dear Joga and all, >> >> Joga makes a valuable point that all pharmacometricians should be aware of. >> Standard methodology for regression assumes that the x-variable is without >> error (loess, linear regression etc.). Note that it is the same for NLME >> models i.e. we generally assume that our independent variables e.g. time, >> covariates etc. are without error. >> >> For DV vs. PRED plots it is common practice, even among those that do not >> know why, to plot PRED on the x-axis and DV on the y-axis. A greater problem >> with these plots is the commonly held expectation that for a "good model" a >> smooth or regression line should align with the line of unity. Though this >> seems intuitive it is a flawed assumption. This issue was clearly pointed >> out by Mats Karlsson and Rada Savic in their 2007 paper titled "Diagnosing >> Model Diagnostics''. For simple well-behaved examples you will see an >> alignment around the line of unity for DV vs. PRED plots. However, there are >> several factors that contribute to an expected deviation from this >> expectation: >> (1) Censoring (e.g. censoring of observations < LLOQ) >> - In this case DVs are capped at LLOQ but PRED values are not. This makes >> it perfectly expected that there will be a deviation from alignment around >> the line of unity in the lower range. >> (2) Strong non-linearities >> - The more nonlinear the modelled system is, the greater the expected >> deviation from the line of unity. Especially in combination with significant >> ETA correlations. >> (3) High variability >> - With higher between/within subject variability (e.g. IIV and RUV) that >> isn't normally distributed (e.g. exponential distributions) will result in >> an expected deviation from the line of unity. Note: this is a form of >> non-linearity so it may fall under the above category. >> (4) Adaptive designs (e.g. TDM dosing) >> - Listed in the original paper by Karlsson & Savic but I have not been able >> to recreate an issue in this case. >> >> I am rather sure that many thousands of hours have been spent on modeling >> trying to correct for perceived model misspecifications that are not really >> there. This is why I recommend relying primarily on simulation-based model >> diagnostics (e.g. VPCs) and as far as possible account for censoring that >> affects the original dataset. As pointed out by Karlsson & Savic a >> simulation/re-estimation based approach can also be used to investigate the >> expected behavior for DV vs. PRED plots for a particular model and dataset >> (e.g. mirror plots in Xpose). Note that to my knowledge there is yet no >> automated way to handle censoring in this context (clearly doable if anyone >> wants to develop a nifty implementation of that). >> >> If we leave the DV vs. PRED plot case, there are many other instances where >> we use scatter plots where it is much less clear what can be considered the >> independent variable and yet other cases where the assumption that the >> x-variable is without error is violated in a way that makes the results hard >> to interpret. One instance of the latter is when exposure-response is >> studied by plotting observed PD response versus observed trough plasma >> concentrations. This is already a way too long email so I will not deep dive >> into that problem as well. >> >> Best regards, >> >> >> Martin Bergstrand, Ph.D. >> >> Principal Consultant >> >> Pharmetheus AB >> >> martin.bergstr...@pharmetheus.com <mailto:martin.bergstr...@pharmetheus.com> >> www.pharmetheus.com <http://www.pharmetheus.com/> >> >> >> >> On Thu, Aug 17, 2023 at 12:44 PM Gobburu, Joga <jgobb...@rx.umaryland.edu >> <mailto:jgobb...@rx.umaryland.edu>> wrote: >>> Dear Friends – Observations versus population predicted is considered a >>> standard diagnostic plot in our field. I used to place observations on the >>> x-axis and predictions on the yaxis. Then I was pointed to a publication >>> from ISOP >>> (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5321813/figure/psp412161-fig-0001/) >>> which recommended plotting predictions on the xaxis and observations on >>> the yaxis. To the best of my knowledge, there was no justification >>> provided. It did question my decades old practice, so I did some thinking >>> and digging. Thought to share it here so others might benefit from it. If >>> this is obvious to you all, then I can say I am caught up! >>> >>> We write our models as observed = predicted + random error; which can be >>> interpreted to be in the form: y = f(x) + random error. It is technically >>> not though. Hence predicted goes on the xaxis, as it is free of random >>> error. It is considered a correlation plot, which makes plotting either way >>> acceptable. This is not so critical as the next one. >>> However, there is a statistical reason why it is important to keep >>> predictions on the xaxis. Invariably we always add a loess trend line for >>> these diagnostic plots. To demonstrate the impact, I took a simple iv bolus >>> single dose dataset and compared both approaches. The results are available >>> at this link: >>> https://github.com/jgobburu/public_didactic/blob/main/iv_sd.html.pdf. I >>> used Pumas software, but the scientific underpinning is agnostic to >>> software. See the two plots on Pages 5 and 6. The interpretation of the >>> bias between the two approaches is different. This is the statistical >>> reason why it matters to plot predictions on the xaxis. >>> >>> Joga Gobburu >>> University of Maryland >> >> This communication is confidential and is only intended for the use of the >> individual or entity to which it is directed. It may contain information >> that is privileged and exempt from disclosure under applicable law. If you >> are not the intended recipient please notify us immediately. Please do not >> copy it or disclose its contents to any other person. >> Any personal data will be processed in accordance with Pharmetheus' privacy >> notice, available here <https://pharmetheus.com/privacy-policy/>.
