On Sun, 03 Jan 2016 08:16:18 -0800, Christopher Green wrote:
This article has been getting a lot media play over the past couple
of days (which is interesting in itself, since it was published back
in April).
It should be noted that it became available online on December 9,
2014, as stated on the Wiley website and in a footnote on the first
page.
It strikes me, however, as a classic example of paying way too
much attention to p-values and not enough to effect sizes.
I caution folks not to fail prey to Geoff Cumming's phobia about
p-values and if one is going to focus on effect sizes, one should
be sure that they are using the right one (a point that even Cumming
acknowledges is a problem with effect sizes measures, that is,
knowing which is the right one to use).
Yes, the effects are significant (mostly), but if you look at the full
article, it appears that the R-squares range from .11 downwards.
Not exactly a Eureka! moment.
http://www.ncbi.nlm.nih.gov/pubmed/25491047?utm_content=buffer077fa&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
A few points:
(1) One problem with effect sizes is that people confuse large effect
sizes with importance and smaller effect sizes with lesser importance.
This is an example of an automatic "inference scheme" that Gerd
Gigerenzer has complained about in statistical inference, from his
criticisms of Null Hypothesis Statistical Testing (NHST) to Bayesian
analysis. The old song lyric "it's not the meat, it's the motion" is
relevant because it is not the absolute magnitude of the effect size
that is critical but the theoretical role that the effect size plays.
That is,
a small or even tiny effect size that is statistically significant is
important if one has a theory that predicts that it should NOT occur.
The fact that it does occur serves as falsification of the theory
and it don't make no nevermind what size it is. Of course, if one has
no theory or does not fully understand the phenomenon that they
studying,
one might simply rely upon which results are significant or not, and/or
which effect size is big or small instead of its theoretical
significance.
This is the real problem with NHST -- ignorant practitioners.
For an example of how this approach works(i.e., does a theory predict
or accommodate a significant result or does it predict against it) see
the
following article which showed how to decide which theory of semantic
memory had the greatest evidence in its favor back in the 1980s:
Chang, T. M. (1986). Semantic memory: Facts and models. Psychological
Bulletin, 99(2), 199.
(2) The question that needs answering is "What is the effect size
measure
that one should use in this situation?"
:
(A) Is it Multiple R^2 (under the assumption that we have homogeneity of
regression, that is, the regressions are the same in the three samples;
we know this is not the case because there are zero females in Sample 3
which leaves sex/gender out of the equation for that group and there is
an
interaction between sex/gender and caudate volume in Sample 1 when
unconstrained measures are used), or,
(B) The semi-partial correlations between caudate volume and the three
IQ measures or the full partial correlations?
Technically speaking, ordinary R^2 shouldn't be used since the
regression
equations for the three samples have different numbers of predictors
(Sample 3
does not include sex/gender because it is all male), and an adjusted-R^2
or
shrunken R^2 (= R^2 - [(1 - R^2)/(N - m - 1)] where N is the sample size
for the
sample and m is the number of predictor -- 4 in samples 1 and 2, 3 in
sample 3).
The adjusted R^2 are all reduced for total IQ but only by 1% or .01,
unless my
calculations are off. I leave it as an exercise to the interested
reader to do the
comparable calculation for VIQ and PIQ because I am not going to bother.
(3) It is odd that they are doing the analysis in AMOS because AMOS
(an acronym for "analysis of moment structure") is a structural equation
modeling (SEM) program. Ordinary linear regression, according to the
Gauss-Markov theory assumes that all of the predictors are measured
without error which I'm willing to accept for the sex/gender variable
and
(maybe) age, but I have a feeling that the measure of caudate volume is
not error free. In SEM, we would create a measurement model for
caudate volume (i.e., X(caudate volume) = T(true value/variance) +
E(error value/variance).
This might be a more appropriate analysis to do but I'm still trying to
figure
out the basis for setting caudate volume constant across the three
samples,
especially since the mean volumes are not equal (see Table 1; perhaps
they should have been mean-centered?).
(4) Table 5(V) reports the "partial correlations" (unclear whether these
are
semi-partial or full partials) between the three IQ measures and the
volumes
of the eight brain areas which gives us
3 (samples) x 3 (IQ measures) x 8 (brain areas) = 72 correlations
Using alpha-2 tailed = .05 for each correlation, the Overall Type I
error rate =
Overall alpha = 1 - (1 - alpha per comparison)^k
where k = # of tests done
Overall alpha = 1 - (1-.05)^72 = 1 - (.95)^72 = 1 - 0.0249 = 0.9751 or
there is a 97% probability that one or more statistically significant
results
are Type I errors. The authors report 9 statistically significant
"partial
correlations" using a p< .05 (excludes p=.05). The 9 correlations
involving
caudate volume have only 4 significant partial correlations.(appearing
only in
Samples 1 and 2). What should one conclude from this? How about
that whatever the relationship is between caudate volume and IQ,.
it occurs only when females are present in the sample. Busy, busy,
busy.
I'm not familiar with the Black Magic practiced in the data analysis for
neuroimaging data like this, so I will leave it to those who are to
explain
what is going on in the data analysis and the reasons why. Long story
short, there is a lot more going on here than a focus on p-values.
-Mike Palij
New York University
[email protected]
---
You are currently subscribed to tips as: [email protected].
To unsubscribe click here:
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=47734
or send a blank email to
leave-47734-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
