Ruben you are mistaken on every single point. But I see it's not worth continuing this discussion. Frank
Rubén Roa wrote > > -----Mensaje original----- > De: r-help-bounces@ [mailto:r-help-bounces@] En nombre de Frank Harrell > Enviado el: viernes, 27 de enero de 2012 14:28 > Para: r-help@ > Asunto: Re: [R] How do I compare 47 GLM models with 1 to 5 interactions > and unique combinations? > > Ruben, I'm not sure you are understanding the ramifications of what Bert > said. In addition you are making several assumptions implicitly: > > -- > Ruben: Frank, I guess we are going nowhere now. > But thanks anyways. See below if you want. > > 1. model selection is needed (vs. fitting the full model and using > shrinkage) > Ruben: Nonlinear mechanistic models that are too complex often just don't > converge, they crash. No shrinkage to apply to a failed convergence model. > > 2. model selection works in the absence of shrinkage > Ruben: I think you are assuming that shrinkage is necessary. > > 3. model selection can find the "right" model and the features selected > would be the same features selected if the data were slightly perturbed or > a new sample taken > Ruben: I don't make this assumption. New data, new model. > > 4. AIC tells you something that P-values don't (unless using structured > multiple degree of freedom tests) > Ruben: It does. > > 5. parsimony is good > Ruben: It is. > > None of these assumptions is true. Model selection without shrinkage > (penalization) seems to offer benefits but this is largely a mirage. > > Ruben: Have a good weekend! > > Ruben > > Rubén Roa wrote >> >> -----Mensaje original----- >> De: Bert Gunter [mailto:gunter.berton@] Enviado el: jueves, 26 de >> enero de 2012 21:20 >> Para: Rubén Roa >> CC: Ben Bolker; Frank Harrell >> Asunto: Re: [R] How do I compare 47 GLM models with 1 to 5 >> interactions and unique combinations? >> >> On Wed, Jan 25, 2012 at 11:39 PM, Rubén Roa <rroa@> wrote: >>> I think we have gone through this before. >>> First, the destruction of all aspects of statistical inference is not >>> at stake, Frank Harrell's post notwithstanding. >>> Second, checking all pairs is a way to see for _all pairs_ which >>> model outcompetes which in terms of predictive ability by -2AIC or >>> more. Just sorting them by the AIC does not give you that if no model >>> is better than the next best by less than 2AIC. >>> Third, I was not implying that AIC differences play the role of >>> significance tests. I agree with you that model selection is better >>> not understood as a proxy or as a relative of significance tests >>> procedures. >>> Incidentally, when comparing many models the AIC is often inconclusive. >>> If one is bent on selecting just _the model_ then I check numerical >>> optimization diagnostics such as size of gradients, KKT criteria, and >>> other issues such as standard errors of parameter estimates and the >>> correlation matrix of parameter estimates. >> >> -- And the mathematical basis for this claim is .... ?? >> >> -- >> Ruben: In my area of work (building/testing/applying mechanistic >> nonlinear models of natural and economic phenomena) the issue of >> numerical optimization is a very serious one. It is often the case >> that a really good looking model does not converge properly (that's >> why ADMB is so popular among us). So if the AIC is inconclusive, but >> one AIC-tied model yields reasonably looking standard errors and low >> pairwise parameter estimates correlations, while the other wasn´t even >> able to produce a positive definite Hessian matrix (though it was able >> to maximize the log-likelihood), I think I have good reasons to select >> the properly converged model. I assume here that the lack of >> convergence is a symptom of model inadequacy to the data, that the AIC >> was not able to detect. >> --- >> Ruben: For some reasons I don't find model averaging appealing. I >> guess deep in my heart I expect more from my model than just the best >> predictive ability. >> >> -- This is a religious, not a scientific statement, and has no place >> in any scientific discussion. >> >> -- >> Ruben: Seriously, there is a wide and open place in scientific >> discussion for mechanistic model-building. I expect the models I built >> to be more than able predictors, I want them to capture some aspect of >> nature, to teach me something about nature, so I refrain from model >> averaging, which is an open admission that you don't care too much >> about what's really going on. >> >> -- The belief that certain data analysis practices -- standard or not >> -- somehow can be trusted to yield reliable scientific results in the >> face of clear theoretical (mathematical )and practical results to the >> contrary, while widespread, impedes and often thwarts the progress of >> science, Evidence-based medicine and clinical trials came about for a >> reason. I would encourage you to reexamine the basis of your >> scientific practice and the role that "magical thinking" plays in it. >> >> Best, >> Bert >> >> -- >> Ruben: All right Bert. I often re-examine the basis of my scientific >> praxis but less often than I did before, I have to confess. I like to >> think it is because I am converging on the right praxis so there are >> less issues to re-examine. But this problem of model selection is a tough >> one. >> Being a likelihoodist in inference naturally leads you to AIC-based >> model selection, Jim Lindsey being influent too. Wanting that your >> models say some something about nature is another strong conditioning >> factor. Put this two together and it becomes hard to do >> model-averaging. And it has nothing to do with religion (yuck!). >> >> ______________________________________________ >> R-help@ mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > > > ----- > Frank Harrell > Department of Biostatistics, Vanderbilt University > -- > View this message in context: > http://r.789695.n4.nabble.com/How-do-I-compare-47-GLM-models-with-1-to-5-interactions-and-unique-combinations-tp4326407p4333464.html > Sent from the R help mailing list archive at Nabble.com. > > ______________________________________________ > R-help@ mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > ______________________________________________ > R-help@ mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > ----- Frank Harrell Department of Biostatistics, Vanderbilt University -- View this message in context: http://r.789695.n4.nabble.com/How-do-I-compare-47-GLM-models-with-1-to-5-interactions-and-unique-combinations-tp4326407p4334353.html Sent from the R help mailing list archive at Nabble.com. ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.