A couple of people have mis-interpreted what I wrote, so I think I should clarify.
I was not suggesting that R^2 should be used for formal model selection. But rather, given your model selection suggests that there are several models that are similarly adequate, it's worth asking what you lose by selecting the model that is formally non-optimal. If you don't lose a lot (e.g. R^2 goes down by 0.1%), then from a model fit point of view, it doesn't really matter which model you chose. OTOH, if it changes by 10% it does make a difference, and you would have to decide if it was worth it for the parsimony (most likely not). There are a couple of issues underlying what I wrote. Firstly, AIC is inly optimal in a narrow predictive sense: in the sense of predicting the same data (this has a stability assumption buried in it, i.e. you're assuming that the conditions in which you collected the data will be the same for where you want to predict for). Now, I think it's very rare that we, as ecologists, want to make predictions in this narrow sense: we're not political pollsters. I think we're usually more interested in understanding what our data is telling us. Hence, having a parsimonious model is more important. There is really no point in fitting a model and then finding out we've no idea what it's telling us. The second point is that I come from a statistical background which doesn't blindly run the numbers. We're trying to understand our data, so the OP's question makes sense. The problem is to understand more about the models, and what the consequences are of using a model which isn't optimal. This will involve some subjectivity, but we're human beings so we're going to interpret the results with some subjectivity anyway. The important thing is to understand why the model we chose is the best. Just using AIC encourages a black box mentality, and doesn't remove subjectivity: unless one is doing prediction in a rather narrow sense (i.e. under exactly the same conditions as those used to collect the data), what objective reason is there for thinking that AIC is optimal? Bob Bob O'Hara Tel: +49 69 798 40226 (in Germany) Mobile: +49 1515 888 5440 WWW: http://www.bik-f.de/root/index.php?page_id=219 Blog: http://blogs.nature.com/boboh/ Journal of Negative Results - EEB: www.jnr-eeb.org >>> Hal Caswell 12/02/10 11:38 PM >>> There are a couple of strange things about the description of the scenario. First, the idea of thinking about a model with almost the same AIC (or, better, AICc) but fewer terms, in pursuit of "parsimony" is doing parsimony twice. The AIC already accounts for the relative number of parameters. If the model with fewer parameters has a worse AIC, the result is saying that the better model is better even though it has more parameters. And, advantageously, it is doing it in an objective way rather than some subjective feeling about parsimony. Second, in regard to Bob's reply to the scenario, R^2 is a really weak tool for comparing models. You can always improve R^2 by adding more terms. The value of information criteria (or at least one of the values) is escaping from that bind in a satisfactory way. Finally, in regard to Davis's question about what to report, in at least some parts of the literature, it is standard to report all the models evaluated, ranked by their Delta AIC. That way the reader can judge how much better the best model is than the second, third, etc. best. But in the end, you need a model to use. Model averaging is a really good procedure here, and it seems a little strange to rule it out. Hal Caswell On Dec 2, 2010, at 10:40 AM, Bob ohara wrote: > This might shock some people, bit AIC does not give The Truth. If > you have a model that fits almost as well, but is simpler, then I > don't see a problem with using it. It's worth checking how much less > of the variation is explain (e.g. using R^2), and also how different > the fitted models are. > > AIC has a tendency to give overly complex models (especially with > lots of data), so I often use BIC instead, which tends too far in > the other direction. Or, if the full model isn't too big, I don't > bother with model selection, and report the full model. > > HTH > > Bob > > Bob O'Hara > > Tel: +49 69 798 40226 (in Germany) > Mobile: +49 1515 888 5440 > WWW: http://www.bik-f.de/root/index.php?page_id=219 > Blog: http://blogs.nature.com/boboh/ > Journal of Negative Results - EEB: www.jnr-eeb.org >>>> Lee Davis 12/01/10 23:44 PM >>> > I have what might seem to be a simple question regarding AIC and > parsimony, > and yet the answers I have found on the subject are unsatisfactory. > So, > opinions please. > > Here is the scenario: > > Let's say that one is using AIC for the selection of nested models > to avoid > multiple LRT comparisons. Should you always choose the model with > deltaAIC = > 0 as the best? What if there is a model with deltaAIC <2 that has > fewer > terms? Should it be chosen in the pursuit of parsimony? Or should > you report > some support for both models? If so, what is the proper language in > this > case? > > Let's assume that we are avoiding model averaging. > > Thanks, > > Lee > -- > Lee Davis > Graduate Assistant > State University of New York > College of Environmental Science & Forestry > Department of Environmental & Forest Biology > 452 Illick Hall, 1 Forestry Drive, Syracuse, NY 13210 > --------------------------------- Hal Caswell Senior Scientist Biology Department Woods Hole Oceanographic Institution Woods Hole MA 02543 508-289-2751 [email protected]
