Hello, I think you mean inflated type-1 error due to a lack of controlling for family-wise error propagation. Regardless of whether there's an appropriate AIC analogue for that, the issue of running all-possible combinations (or running a step-wise regression procedure) within an AIC framework goes beyond error propagation.
All-possible/all-subset approaches share the same problems with step-wise in regards to biased parameter estimates and that the scope of inferences that can be drawn from this sort of exploratory approach is necessarily limited. "Data-dredging" isn't at all wrong as long as it is properly recognized that doing so is a form of exploratory analyses and the results should thus be independently validated (or at a minimum retesting the conclusions with a withheld subset of the data). Of course, exploratory analyses have arguably done more to increase our understanding than confirmatory but that's a different topic. In regards to "3", to my recollection there are several individuals with far more knowledge than I who likely disagree with what I'm about to say. Nonetheless admonishing people for presenting model-ranking results side-by-side with "traditional" (or whatever) approaches seems rather dogmatic to me and not strictly necessary. If my two top models are nested then presenting a drop-in-deviance test alongside the AIC differences might provide information that a reader would find useful even if it breaks with scripture. I don't care to enshrine tools and am more interested in what the data say in regards to the questions I'm asking. If presenting both sets of results helps the reader draw their own inferences, even better. Of course I've never really understood how to understand something being a "different and independent statistical paradigm" from a practical standpoint. This is a potentially fun and contentious topic! Ned -- Ned Dochtermann Department of Biology University of Nevada, Reno 775-784-6781 [email protected] www.unr.nevada.edu/~dochterm/ -- -----Original Message----- From: Ecological Society of America: grants, jobs, news [mailto:[email protected]] On Behalf Of Bruce Robertson Sent: Monday, February 08, 2010 10:58 AM To: [email protected] Subject: [ECOLOG-L] AIC, data-dredging, and inappropriate stats Dear Ecologists, I've been using an information-theoretic model-selection approach as a part of my research and have found that the ecological literature appears to be very hypocritical and inconsistent in how these stats are used and interpreted. I've been consulting a statistician and he has verified and clarified some interesting problems, about which I'd love to hear your comments. 1. Data dredging. Starting with 6 independent variables and a single dependent variable. Historically, the recommended approach is to choose a set of a-priori models containing combinations of these variables that make ecological sense, then rank them using AIC scores and weights. Running all possible combinations of these 6 variables has historically been looked down upon because type II error goes through the roof. For example, in hypothesis testing with a crit p-val of .05, 1 out of every 20 models you will run will appear statistically significant just by chance alone. There are not yet any methods to account for the type II error associated with running a bunch of spurious models in the AIC ranking approach. Why do I see soooo many paper in soooo many highly ranked ecological journals (e.g. ecology, ecology letters, ecological applications) that do this (run all possible comibations of variables) anyway? 2. Summing of AIC scores. People who run all combinations of variables in their model selection approach will often sum up all the AIC scores of all models with variable 1, then "," with variable 2, etc. The total of these scores for each variable is supposed to reflect it's importance. The approach seems problematic because it is based upon data dredging (above), but seems common in journals like Ecology, Ecology letters, etc. I actually saw one paper in the Journal of Biogeography in which the author choice to select a set of a-priori models to run, then took this summing approach. Wouldn't this just show that the most important variables were the variables that the analyst thought were important a-priori. 3. Use of other 'fit' statistics along with the model-selection approach. I often see people reporting other statistics (e.g. p-vals, r-squared) in combination with the AIC scores. My statistician friend says that this is totally inappropriate, and uninformative. My general impression is that, while the statistical world has yet to develop more robust techniques (e.g accounting for type II error in model selection), that there are clear recommendations that make some approaches (e.g. data dredging) clearly improper. Please comment on whether ecologists are simply not 'following the rules' (perhaps out of ignorance) or whether there really are different and statistically valid opinions on this topic. Many thanks to all, -- Bruce Robertson Research Associate Kellogg Biological Station Michigan State University 3700 East Gull Lake Drive Hickory Corners, MI 49060 206-71-9172 [email protected] Homepage: www.msu.edu/~roberba1/Index.html/
