Dear all,
Not a problem that is very specific to R, but I think that it is not only of
interest to me... So I hope that someone finds the time to provide me with some
clues on this probably rather basic issue.
I'm performing an analysis of experimental data with a categorical response
variable. Until now I have been using a classical maximum likelihood approach.
The most basic output measure to evaluate the models is thus minus twice the
log likelihood, that is, the deviance. I am evaluating the statistical
significance of effects by calculating the absolute difference between
deviances (for nested models), which should be approximately chi-square
distributed under the null hypothesis - that is the classical Wilks' test. I
use bootstrapping sometimes where I suspect that the approximation is not good.
But for the rest, the philosophy of the test is the same.I'm quite sure that
the magnitude of effect of the experimental manipulations is highly
subject-dependent. I have, in this specific case, 7 subjects; therefore, I
perform and report 7 separate analyses. The problem afterwards lies in the
model selection. Sometimes, the best-fitting model (either based on
all-possible subset fit!
s or a sequential procedure, with p-values, AIC or BIC based) for subject A is
X1+X2, for subject B X1+X3, for C X1+X1*X2 etc. You can imaginethat this turns
into a mess. I thought that evaluation of the fits of competing models could be
based on a likelihood statistic that is jointly determined for all subjects,
namely by aggregating (adding) deviances across subjects. In theory, if I can
say that within-subject -2 log likelihood ratios to test for effects are
asymptotically chi-square distributed, the sum should also be, with degrees of
freedom added together. So this way I'm actually doing a kind of
meta-analysisby adding deviances.(1) How defendable is this procedure?
Strangely enough (or maybe not...), I didn't find anything written about a
procedure like this. This leads me to believe I must be doing something very
wrong... The problem, I suppose, is that I'm treating effects that are actually
random, as fixed. But what are the consequences of this? Does it mean that !
I am making inferences about the specific sample that I have used in t
he experiment, that are possibly not generalizeable across the population, or
is the problem more serious?(2) In the kind of situation I am facing, I often
see applications of generalized linear mixed model, empirical Bayes, and the
like. Somehow I think that a hierarchical Bayes model would be overkill for
this. Hypothesis tests would turn into confidence regions. It might be
computationally unfeasible anyway because in some of my alternative models,
levels of the experimental variable are treated in a categorical way,
generating a lot of free parameters that are subject to different random
distributions. I'm still getting acquainted with this specific technique, so I
count on your patience in case this is a stupid question: If maximum-likelihood
estimation is as Bayesian inference with a flat prior on the parameters, is the
meta-analysis I am trying out here then equivalent to empirical Bayes with flat
priors for the random effects?
_________________________________________________________________
Hoe herken je de vrienden van je vrienden?
[[alternative HTML version deleted]]
______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
