Hi
From: Wuensch, Karl Louis [mailto:[email protected]]
Yes, meta-analysis is called for here, if you can get the data out of
all those file drawers.
My concern is that even more rigid criteria for publication (larger Ns or
higher significance levels) will increase the file-drawer problem, exacerbating
the challenge of accurate meta-analysis. Reducing the threshold for purely
"empirical" papers or having some way for them to be deposited in some
repository would facilitate meta-analysis and the identification of "real"
effects. Of course the research community would have to have some benefit to
accrue to replication studies; otherwise, people might never undertake them.
There were some other interesting points made in the article.
One was that replication rates were higher in the cognitive than social
journals. They mentioned the use of within-s designs as one possible
explanation. Might it also be the case that dependent variables in the
cognitive area have fewer uncontrolled sources of variability, hence making the
effects "cleaner" (less noisy?) and easier to replicate? I don't recall whether
they distinguished between experimental and non-experimental effects, which
could be another difference between areas.
Another observation was that interaction effects were less likely to replicate
than main effects. The proper analysis of interactions is challenging and
something of an issue. Specifically, the default ANOVA approach is optimal only
for rare cross-over interactions, yet some (e.g., Rosenthal) argue that is the
proper approach to interactions even when it is not appropriate (according to
some people) as when no effect is expected in one condition (e.g., control
group) and only in other conditions. The latter responds better to simple
effects analysis, which according to the ANOVA purist "confounds" main and
interaction effects.
Let's hope these kinds of studies lead to thoughtful and constructive changes
in the discipline and also do not harm the credibility of research in the eyes
of those who already would like to minimize its impact.
Take care
Jim
---
You are currently subscribed to tips as: [email protected].
To unsubscribe click here:
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=46485
or send a blank email to
leave-46485-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
