This is a great example of the kind of terminology debates that the ASA / ISOP Statistics and Pharmacometrics special interest group (SxP) is trying to tackle.
As Mats and Bill point out, the common usage within our community is to say that the percentiles (5th, 95th) are “prediction intervals” and the interval estimates / uncertainty around these percentiles are “confidence intervals”. But as Ken points out, these terms do not strictly correspond to the statistical definition of each if you take into account what the VPC procedure is actually doing. The VPC is a model diagnostic procedure for the observed data and provides a visual check of whether the model is capturing central tendencies and dispersion in our data. (BTW, I *know* there are debates about the usefulness or otherwise of VPC plots. I’m not going to address that here and I suggest we don’t disappear down *that* rabbit hole.) We are NOT trying to make probabilistic statements about the likelihood of observed percentiles being within the intervals around these. So if the question arises from some reviewer based on our use of statistically woolly terms like “prediction interval” or “confidence interval” we should be ready to put up our hands and admit that the terms we are using do not imply those statistical properties. We could advocate changing the terminology used, but that may not have traction in the community after this length of time. But we *should* be cognizant about what these things are, what they’re for, what the formal, statistical terminology implies and what our use (or maybe misuse) is or isn’t implying. The ASA / ISOP SxP group has had a session accepted at this year’s ACOP meeting where we hope to surface a few of these thorny issues and debate between our use of terminology in pharmacometrics, the statistical interpretation of that terminology and whether it *really* matters. If you’re interested, please come along and be prepared to engage in the discussion! Best regards, Mike (co-chair of ASA / ISOP SxP SIG) From: owner-nmus...@globomaxnm.com<mailto:owner-nmus...@globomaxnm.com> <owner-nmus...@globomaxnm.com<mailto:owner-nmus...@globomaxnm.com>> On Behalf Of Ken Kowalski Sent: 14 March 2019 21:02 To: 'Bill Denney' <wden...@humanpredictions.com<mailto:wden...@humanpredictions.com>>; nmusers@globomaxnm.com<mailto:nmusers@globomaxnm.com>; Soto, Elena <elena.s...@pfizer.com<mailto:elena.s...@pfizer.com>> Subject: [EXTERNAL] RE: [NMusers] VPCs confidence intervals? Hi All, I know there is a lot of confusion about the distinction between a confidence interval and a prediction interval. Here is a layperson’s way of making the distinction. A confidence interval makes inference on a population parameter which is fixed (never changes) regardless of any sample data that is collected to estimate the parameter (if you repeatedly sampled an infinite number of observations to obtain the population value by definition you would get the same population value for each sample with an infinite sample size) . Thus, the confidence interval only reflects the uncertainty in the estimate of that parameter. In contrast, a prediction interval makes inference on a statistic for a future sample set of data. That statistic will vary from sample to sample and hence must also take into account the sampling variation as well as the parameter uncertainty. A prediction interval can be thought of as a confidence interval of the prediction of some statistic from a future sample. That is, both a confidence interval and a prediction interval have a confidence level associated with them. In the case of the confidence interval, the confidence level is the coverage probability that the interval will contain the true value of the population parameter if one were to repeat the experiment an infinite number of times. In the case of the prediction interval, the confidence level is the coverage probability that the interval will contain the future sample mean (of a finite sample size) if one were to repeat the experiment an infinite number of times. There is another type of statistical interval in addition to confidence and prediction intervals and that is a tolerance interval. A tolerance interval can be thought of as a confidence interval that a specified proportion of the individual responses will be contained within the interval. For example, we can calculate a 95% tolerance interval to contain 90% of the observed data (i.e., we are 95% confident that the interval will contain 90% of the individual observations). Tolerance intervals are more common in a manufacturing setting where it is important to produce an item to some specification within some tolerance limits. Nevertheless, there is a certain VPC plot that we often generate that is somewhat akin to a tolerance interval. When we summarize our simulated data for VPCs and summarize the 5th and 95th percentiles of the individual responses this is more akin to a tolerance interval to contain 90% of the observed individual data. In contrast, when we summarize the sample mean or median from say 1000 simulated trials and calculate the 5th and 95th percentiles across the 1000 trials that is more akin to a prediction interval for that statistic (e.g., sample mean or sample median). Note however, the intervals obtained as percentiles of a sample statistic across trials (i.e., prediction interval) or sample observations across individual subjects (i.e., tolerance interval) don’t have valid coverage probabilities for repeated experiments unless they take into account parameter uncertainty. Kind regards, Ken
