Hi Andrew,
My message was predicated on the poster (Howie) treating his analysis
using ANCOVA in its simplest typical form, a single-factor covariance
model. The "full" model is the one you give as #3.
A start of an answer to your first question would be that the ANOVA
table for this "full" model says nothing about a model where the
intercepts are constrained to be the same and the slopes are
different. Such a model makes sense sometimes, and Howie asked about
"two or more regressions ... if their slopes **and/or** intercepts are
significantly different." A further answer about what hypotheses could
be entertained by model selection that are not addressed by the usual
ANCOVA could go on and on. Any hypothesis that can be specified as a
statistical model could be added to the mix and its support relative
to the other models evaluated. There are more than three models
possible, especially once more than two regressions (a factor with
more than two levels) are entertained. You could of course have a
model where two levels share one slope and the other level has its
own.
A very critical point here is that the ANOVA table alone does not tell
you "which model is best supported." There are a number of reasons for
this, one of which is that model selection uncertainty is not
addressed. A reference worth reading and re-reading on this is:
@ARTICLE{chatfield1995modelselectionuncertainty,
author = {Chatfield, C.},
title = {Model uncertainty, data mining and statistical inference},
journal = {Journal of the Royal Statistical Society, Series A},
year = {1995},
volume = {158, Part 3},
pages = {419-466}
}
For the same reasons that stepwise regression procedures fail,
p-values in F-tables cannot do model selection. Sometimes the answers
are similar ("roughly equivalent" you say -- but who wants "rough"
answers when we can get good ones?), but the approaches are
fundamentally different.
So yes, you're missing a lot about model selection. A quick Google
search will overwhelm with material, but a paper I suspect will give
you a taste of many aspects of this is:
@ARTICLE{hobbshilborn2006,
author = {Hobbs, N. T. and Hilborn, R.},
title = {Alternatives to statistical hypothesis testing in ecology:
a guide to self teaching},
journal = {Ecological Applications},
year = {2006},
volume = {16},
pages = {5-19}
}
-- Dave Hewitt
P.S. -- I'll note that ANCOVA-type analyses are not usually interested
in the importance of the covariate (x in your models). The
relationship with x is assumed to be different from zero (usually by
looking at the data and seeing so) and is included to help evaluate
the effect of the factor, which is usually of primary interest.
On Thu, Apr 1, 2010 Andrew Rominger <[email protected]> wrote:
> I'm novice with respect to AIC model selection so I'm wondering what
> are the
> hypotheses that model selection addresses and ANCOVA does not? To provide
> an example, imagine we've got y (response variable), x (continuous
> predictor), and group (discrete predictor). I could see fitting three
> models:
>
> (1) y as a function of group
> (2) y as a function of group + x
> (3) y as a function of group + x + group*x
>
> And depending on which model is best supported we conclude different
> things:
> model 1 supported = no support for considering x, model 2 = no support for
> considering multiplicative term, model 3 = there is reason to believe all
> terms are worthy of inclusion.
>
> Wouldn't an ANCOVA do a similar thing? We get an ANOVA table that tells
> us
> the sum of squares for each term, and the resultant hypothesis test. So
> depending on the p-values (I know nobody likes to depend on p-values), we
> could reach any conclusion roughly equivalent to those I (perhaps wrongly)
> supposed for the model selection case. Equivalent results would be
> something like: group significant, OR group & x significant, OR group
> & x &
> group*x significant.
>
> Am I missing something with model selection? Thanks for your thoughts and
> insights,
> Andy
