There are a couple of confusing points in your response Bob. 1.) Why would you use R^2 rather than dAIC and wi to see how large the differences between models are? 2.) Doesn't all prediction assume that conditions are "similar enough" that the prediction conditions are valid? 3.) Related to above then is it not more important to try and predict the range of stability rather than just throwing up our hands and saying things are not stable? 4.) Parsimony does not imply nor guarantee an interpretable model. 5.) Actually what we want is the most parsimonious model that adequately explains our data, not just the most parsimonious model.
AIC is merely a metric that has some desirable properties and in a model selection procedure performs better than R^2 or BIC. Interpreting and explaining the model(s) comes down to biology as it should. I do not think anybody is treating AIC as an explanatory black box, but rather as a tool to help us select from a range of models. ><(((º> ><(((º> ><(((º> ><(((º> ><(((º> ><(((º> Jim Novak Biological Sciences Department 1162 life Sciences Annex Eastern Illinois University Charleston, IL 61920 (217) 581-6385 (217) 581-7141 (FAX) [email protected] http://www.ux1.eiu.edu/~jmnovak/ ><(((º> ><(((º> <º)))>< ><(((º> ><(((º> ><(((º> On Dec 3, 2010, at 9:08 AM, Bob ohara wrote: > A couple of people have mis-interpreted what I wrote, so I think I should > clarify. > > > I was not suggesting that R^2 should be used for formal model selection. But > rather, given your model selection suggests that there are several models > that are similarly adequate, it's worth asking what you lose by selecting the > model that is formally non-optimal. If you don't lose a lot (e.g. R^2 goes > down by 0.1%), then from a model fit point of view, it doesn't really matter > which model you chose. OTOH, if it changes by 10% it does make a difference, > and you would have to decide if it was worth it for the parsimony (most > likely not). > > > There are a couple of issues underlying what I wrote. Firstly, AIC is inly > optimal in a narrow predictive sense: in the sense of predicting the same > data (this has a stability assumption buried in it, i.e. you're assuming that > the conditions in which you collected the data will be the same for where you > want to predict for). Now, I think it's very rare that we, as ecologists, > want to make predictions in this narrow sense: we're not political pollsters. > I think we're usually more interested in understanding what our data is > telling us. Hence, having a parsimonious model is more important. There is > really no point in fitting a model and then finding out we've no idea what > it's telling us. > > > The second point is that I come from a statistical background which doesn't > blindly run the numbers. We're trying to understand our data, so the OP's > question makes sense. The problem is to understand more about the models, and > what the consequences are of using a model which isn't optimal. This will > involve some subjectivity, but we're human beings so we're going to interpret > the results with some subjectivity anyway. The important thing is to > understand why the model we chose is the best. Just using AIC encourages a > black box mentality, and doesn't remove subjectivity: unless one is doing > prediction in a rather narrow sense (i.e. under exactly the same conditions > as those used to collect the data), what objective reason is there for > thinking that AIC is optimal? > > > Bob > > Bob O'Hara > > Tel: +49 69 798 40226 (in Germany) > Mobile: +49 1515 888 5440 > WWW: http://www.bik-f.de/root/index.php?page_id=219 > Blog: http://blogs.nature.com/boboh/ > Journal of Negative Results - EEB: www.jnr-eeb.org >>>> Hal Caswell 12/02/10 11:38 PM >>> > There are a couple of strange things about the description of the > scenario. > > First, the idea of thinking about a model with almost the same AIC > (or, better, AICc) but fewer terms, in pursuit of "parsimony" is doing > parsimony twice. The AIC already accounts for the relative number of > parameters. If the model with fewer parameters has a worse AIC, the > result is saying that the better model is better even though it has > more parameters. And, advantageously, it is doing it in an objective > way rather than some subjective feeling about parsimony. > > Second, in regard to Bob's reply to the scenario, R^2 is a really weak > tool for comparing models. You can always improve R^2 by adding more > terms. The value of information criteria (or at least one of the > values) is escaping from that bind in a satisfactory way. > > Finally, in regard to Davis's question about what to report, in at > least some parts of the literature, it is standard to report all the > models evaluated, ranked by their Delta AIC. That way the reader can > judge how much better the best model is than the second, third, etc. > best. But in the end, you need a model to use. Model averaging is a > really good procedure here, and it seems a little strange to rule it > out. > > Hal Caswell > > > On Dec 2, 2010, at 10:40 AM, Bob ohara wrote: > >> This might shock some people, bit AIC does not give The Truth. If >> you have a model that fits almost as well, but is simpler, then I >> don't see a problem with using it. It's worth checking how much less >> of the variation is explain (e.g. using R^2), and also how different >> the fitted models are. >> >> AIC has a tendency to give overly complex models (especially with >> lots of data), so I often use BIC instead, which tends too far in >> the other direction. Or, if the full model isn't too big, I don't >> bother with model selection, and report the full model. >> >> HTH >> >> Bob >> >> Bob O'Hara >> >> Tel: +49 69 798 40226 (in Germany) >> Mobile: +49 1515 888 5440 >> WWW: http://www.bik-f.de/root/index.php?page_id=219 >> Blog: http://blogs.nature.com/boboh/ >> Journal of Negative Results - EEB: www.jnr-eeb.org >>>>> Lee Davis 12/01/10 23:44 PM >>> >> I have what might seem to be a simple question regarding AIC and >> parsimony, >> and yet the answers I have found on the subject are unsatisfactory. >> So, >> opinions please. >> >> Here is the scenario: >> >> Let's say that one is using AIC for the selection of nested models >> to avoid >> multiple LRT comparisons. Should you always choose the model with >> deltaAIC = >> 0 as the best? What if there is a model with deltaAIC <2 that has >> fewer >> terms? Should it be chosen in the pursuit of parsimony? Or should >> you report >> some support for both models? If so, what is the proper language in >> this >> case? >> >> Let's assume that we are avoiding model averaging. >> >> Thanks, >> >> Lee >> -- >> Lee Davis >> Graduate Assistant >> State University of New York >> College of Environmental Science & Forestry >> Department of Environmental & Forest Biology >> 452 Illick Hall, 1 Forestry Drive, Syracuse, NY 13210 >> > > > > > --------------------------------- > Hal Caswell > Senior Scientist > Biology Department > Woods Hole Oceanographic Institution > Woods Hole MA 02543 > 508-289-2751 > [email protected]
