The usual futile plea: If you reply to this long (24 kB) post please
don't hit the reply button unless you prune copy of this post that
may appear in your reply down to a few relevant lines and then edit
those lines (and the subject heading), otherwise (a) the entire
already archived post may be needlessly resent to subscribers, (b)
you may insert an inappropriate subject heading into your post, and
(c) you may obscure who said what in a maze of marginal angle
brackets ">", ">>",">>>", etc..
************************************************
ABSTRACT: Bryan Roessel (2008) wrote ". . . . my background in
statistical analyses is weak. Can anyone point me to a useful
resource for the sorts of maths that could be helpful to me as a high
school science teacher?" After repeating (and slightly augmenting)
the references given by Sue Ramlo (2008a), I make five comments that
might be useful to those contemplating the use of statistics in
education research.
************************************************
Bryan Roessel (2008), in his PhysLrnR post of 19 Jun 2008, oddly
titled "PHYSLRNR Digest - 17 Jun 2008 to 18 Jun 2008 (#2008-104)" -
probably an artifact of conventional reply button hitting - wrote
[bracketed by lines "BBBBB. . . . ."; my inserts at ". . .
.[[insert]]. . . ."]:
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
Would people mind quoting only the pertinent portions of posts
they're responding to (or quoting nothing at all, where applicable)?
I'm afraid I spend about as much time scrolling through filler as I
do reading messages. . . . .[[Over the decades, Bryan's oft-repeated
plea has fallen on the deaf ears of reply button hitters (RBH's) -
see e.g., "Hitting the Reply Button - Bane of Discussion Lists"
(Hake, 2008a)]]. . . .
I find the recent discussion on statistics to be interesting, but my
background in statistical analyses is weak. Can anyone point me to a
useful resource for the sorts of maths that could be helpful to me as
a high school science teacher?"
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
In response to the second paragraph, Sue Ramlo (2008), in a post
titled "Statistics resources," wrote [bracketed by lines "SSSSS. . .
."; my inserts at ". . . .[[insert]]. . .." ]:
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
There are a number of relatively accessible and good resources for
statistics. Here are a few:
Newman, I. & Newman, C. (1994). Conceptual statistics for beginners
(2nd ed.). Lanham, MD: University Press of America. Note: there is a
more recent edition of this text which now comes with a workbook . .
. . . [[Sue may be referring to Newman, Newman, Brown, & McNeely
(2005)]]. . . . . .; this is an extremely readable book and I
recommend it highly.
McNeil, K.A., Newman, I., Kelly, F.J. (1996). Testing Research
Hypotheses with the General Linear Model. United States of America:
Southern Illinois University Press. . . .[[Amazon.com information at
<http://tinyurl.com/3zsxtu>]]. . . ..
Cohen, J. (1977). Statistical power analysis for the behavioral
sciences (2nd ed.). New York: Academic Press. . . . .[[there was a
"revised edition" published in 1977, but my own copy of the 2nd
edition is dated 1988 - see Cohen (1988) in the REFERENCE list
below]]. . . .
. . . . . . [[Cornell University GRADUATE STUDENTS in Program
Evaluation and Planning, "Research Methods Tutorials," online at]]. .
. . . <http://www.socialresearchmethods.net/tutorial/tutorial.htm>
- online resource that is free and good with a variety of specific
topics addressed.
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Five comments that might be useful to those interested in the use of
statistics in education research:
11111111111111111111111111111111111111111111
1. In my opinion, those contemplating research in education should
consider the possibility that the research DESIGN may as important,
if not more so, than the proper use of statistics. Richard Light et
al. (1990) in "By Design" write on pp. vii-viii:
"We emphasize research design over measurement and analysis. This is
because good design comes first. No matter how precise your
measurement or how sophisticated your analysis, you risk failure if
your research is not well planned. YOU CAN'T FIX BY ANALYSIS WHAT YOU
BUNGLED BY DESIGN."
Some other good books that emphasize research design are - e.g.:
"Research Methods in Education" [Slavin (1992)]; "Estimating Causal
Effects Using Experimental and Observational Designs" [Schneider et
al. (2007)]; "Handbook of Design Research Methods in Education:
Innovations in Teaching" [Kelly, Lesh, & Baek (2008); "Physics &
Philosophy: The Revolution in Modern Sciences" [Heisenberg (1999,
2007) - see the Signature quote below]; and especially "Experimental
and Quasi-Experimental Designs for Generalized Causal Inference"
[Shadish, Cook, & Campbell (2002)] - a goldmine of references on
social-science research.
22222222222222222222222222222222222222222
2. As I indicated in "Re: Another point about effect size" [Hake (2008b)]:
a. I think the recent text "Foundations of Behavioral Statistics: An
Insight-Based Approach," by Bruce Thompson (2006) is well worth
studying. See especially Chapter 12 on "The General Linear Model."
b. On page 191 Thompson (2006) writes:
"A comparison of . . . [the Glass vs Cohen effect sizes]. . . .
affords the opportunity to emphasize the fundamentally important
point that statistics is about thinking, rather than black vs white
decisions or rote memorization of formulas. As Huberty and Morris
(1988, p. 573) argued, 'As in all statistical inference, subjective
judgment cannot be avoided. Neither can reasonableness!' "
Well said Bruce Thompson, Carl Hubberty, and John Morris!
See also Thompson's explanation on pages 177-182 of what the popular
but often misunderstood "p" of Null Hypothesis Statistical
Significance Testing (NHSST) does and does not mean. Thompson writes:
"The p computation presumes that the null hypothesis exactly
describes the population, and then evaluates the probability that the
sample came from this presumed population. The direction of
inference is population to sample, NOT (emphasis in the original)
sample to population. If NHSST really did make an inference from
sample to population, then (and only then) the outcome of the
statistical significance test would bear upon result replicability."
Thompson then quotes from Cohen's (1994) classic "The earth is round
( p < .05)," writing: "In Cohen's immortal words the statistical
significance test 'does not tell us what we want to know, and we so
much want to know what we want to know that, out of desperation, we
nevertheless believe that it does!" (p. 997). ' "
For a collection of references on the widely misunderstood nature of
NHSST and why p values, when they are reported, should be accompanied
by "effect sizes" see "Lessons from the physics education reform
effort" [Hake (2002a)]. Therein I wrote [bracketed by lines "HHHHH. .
. ."]:
HHHHHHHHHHHHHHHHHHHHHHHHHHH
Carver (1993) subjected the Michelson and Morley (1887) data to a
simple analysis of variance (ANOVA) and found *STATISTICAL
significance associated with the direction the light was traveling (p
< 0.001 )!* He writes, "It is interesting to speculate how the course
of history might have changed if Michelson and Morley had been
trained to use this *corrupt form of the scientific method* that is,
testing the null hypothesis first. They might have concluded that
there was evidence of significant differences in the speed of light
associated with its direction and that therefore there was evidence
for the luminiferous ether ... Fortunately Michelson and Morley ...
[first] ... interpreted their data with respect to their research
hypothesis." Consistent with the scientific methodology of physical
scientists such as Michelson and Morley . . . . . Rozeboom (1960)
wrote that " ... the primary aim of a scientific experiment is not to
precipitate decisions, but to make an appropriate adjustment in the
degree to which one accepts, or believes, the hypothesis or
hypotheses being tested."
HHHHHHHHHHHHHHHHHHHHHHHHHHH
For a mildly pro-NHSST article see "Shaping Up the Practice of Null
Hypothesis Significance Testing" [Wainer & Robinson (2003)]. But
even they conclude that "[NHSST] is most often useful as an adjunct
to other results (e.g., effect sizes) rather than as a stand-alone
result."
33333333333333333333333333333333333333333
3. Aaron Titus's (2006) valuable "Assessment Analysis is a
"web-based program (CGI script) that helps teachers analyze test
results perform correlational analysis, and do factor analysis."
Test analysis is a tool used to analyze pre and post test data: t
-test, normalized gain (individual and class), effect size, max, min,
mean, median, standard deviation, Kuder-Richardson reliability
coefficient KR-20, item difficulty, & point biserial coefficient.
4444444444444444444444444444444444444444
4. In "Over Sixty Academic Discussion Lists: List Addresses and
URL's for Archives & Search Engines" [Hake (2008c)] are eight
discussion lists that are often frequented by the statistically
inclined. [The asterisk * indicates the use of LISTSERV software with
its great search engines and archives; "[R]" indicates that access to
the archives is restricted to subscribers, but subscribing usually
takes only a few minutes by clicking on the archive URL and then
clicking on "Join or leave the list. . . " ; the angle brackets
<.......> enclose the list archive URL]:
a. *AERA-D (Measurement and Research Methodology) [The archives are
OPEN but only
AERA members are allowed to post.]
<http://listserv.aera.net/archives/aera-d.html>;
b. *ASSESS (Assessment in Higher Education)
<http://lsv.uky.edu/archives/assess.html>
c. *EdResMeth-L (Educational Research Methodology)
<http://listserv.uconn.edu/edresmeth-l.html> [Recently shifted
to OPEN!];
d. [R] *EDSTAT-L (Teaching and Learning Statistics)
<http://lists.psu.edu/archives/edstat-l.html>;
e. *EVAL-SYS (Systems in Evaluation)
<http://lists.evaluation.wmich.edu/archives/eval-sys.html>
f. [R] *EVALTALK (American Evaluation Association Discussion List) [N = 2230]
<http://bama.ua.edu/archives/evaltalk.html>
g. [R] *Multilevel Modeling
<http://www.jiscmail.ac.uk/lists/multilevel.html>;
h. [R] *SEMNET (Structural Equation Modeling)
<http://bama.ua.edu/archives/semnet.html>.
55555555555555555555555555555555555555555
5. Modesty forbids mention of three posts of mine plus an article
that are replete with references to the psychometric literature:
a. "References for Psychometrically Naive Physicists" [Hake (2002b)];
b. "Re: Floor Effect and ANCOVA" [Hake (2004a,b)];
c. "Possible Palliatives for the Paralyzing Pre/Post Paranoia that
Plagues Some PEP's" [Hake (2006)]. PEP =
Psychology-Education-Psychometric communities.]
Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands.
<[EMAIL PROTECTED]>
<http://www.physics.indiana.edu/~hake/>
<http://www.physics.indiana.edu/~sdi/>
"In the course of coming into contact with empirical material,
physicists have gradually learned how to pose a question properly.
Now proper questioning often means that one is more than half the way
towards solving the problem."
Werner Heisenberg (1999, 2007)
REFERENCES [Tiny URL's courtesy <http://tinyurl.com/create.php>.]
Carver, R.P. 1993. "The case against statistical significance
testing, revisited." Journal of Experimental Education 61(4):
287-292. An abstract is online at
<http://www.questia.com/googleScholar.qst?docId=95855984>.
Cohen, J. 1988. "Statistical power analysis for the behavioral
sciences." Second edition. Lawrence Erlbaum. Amazon.com information
<http://tinyurl.com/57akm8>. Note the "Search Inside" feature.
Hake, R.R. 2002a. "Lessons from the physics education reform effort,"
Ecology and Society 5(2): 28; online at
<http://www.ecologyandsociety.org/vol5/iss2/art28/>. Ecology and
Society (formerly Conservation Ecology) is a free online
"peer-reviewed journal of integrative science and fundamental policy
research" with about 11,000 subscribers in about 108 countries. See
also the more recent Hake (2007).
Hake, R.R. 2002b. "References for Psychometrically Naive Physicists,"
PhysLrnR post of 6 Nov 2002 14:15:05-0800; online at
<http://tinyurl.com/5du8rp>. To access the archives of PhysLnR one
needs to subscribe :-( , but that takes only a few minutes by
clicking on <http://listserv.boisestate.edu/archives/physlrnr.html>
and then clicking on "Join or leave the list (or change settings)."
If you're busy, then subscribe using the "NOMAIL" option under
"Miscellaneous." Then, as a subscriber, you may access the archives
and/or post messages at any time, while receiving NO MAIL from the
list!
Hake, R.R. 2004a. "Re: Floor Effect and ANCOVA"; online at
http://tinyurl.com/5hboc7. Post of 20 Oct 2004 17:22:14-0700 to
AERA-C, AERA-H, AERA-J, AERA-K, ASSESS, EdStat, EvalTalk, PhysLrnR,
POD, & STLHE-L.
Hake, R.R. 2004b. "Re: Floor Effect and ANCOVA"; online at
<http://tinyurl.com/58xhvf>. Post of 24 Oct 2004 21:36:22-0700 to
AERA-C, AERA-H, AERA-J, AERA-K, ASSESS, EdStat, EvalTalk, PhysLrnR,
POD, & STLHE-L.
Hake, R.R. 2006. "Possible Palliatives for the Paralyzing Pre/Post
Paranoia that Plagues Some
PEP's," Journal of MultiDisciplinary Evaluation, Number 6, November; online at
<http://survey.ate.wmich.edu/jmde/index.php/jmde_1/article/view/41/50>.
This even despite the admirable anti-alliteration advice at
psychologist Donald Zimmerman's site
<http://mypage.direct.ca/z/zimmerma/> to "Always assiduously and
attentively avoid awful,
awkward, atrocious, appalling, artificial, affected alliteration."
PEP = Psychology-Education-Psychometric communities. This is a
severely truncated version of Hake (2007).
Hake, R.R. 2007. "Should We Measure Change? Yes!" online at
<http://www.physics.indiana.edu/~hake/MeasChangeS.pdf> (2.5 MB), or
as ref. 43 at
<http://www.physics.indiana.edu/~hake>. To appear as a chapter in
"Evaluation of Teaching and Student Learning in Higher Education, "a
Monograph of the American Evaluation Association
<http://www.eval.org/>.
Hake, R.R. 2008a. "Hitting the Reply Button - Bane of Discussion
Lists," online on the now OPEN archives of EdResMehod at
<http://tinyurl.com/3nyn7j>. Post of 20 Jun 2008 20:29:17-070 to
EdResMeth and PhysLrnR.
Hake, R.R. 2008b. "Re: Another point about effect size" online on the
now OPEN archives of EdResMehod at <http://tinyurl.com/448a4e>. Post
of 20 Jun 2008 15:48:23 -0700to EdResMeth and PhysLrnR.
Hake, R.R. 2008c. "Over Sixty Academic Discussion Lists: List
Addresses and URL's for Archives & Search Engines," online at
<http://www.physics.indiana.edu/~hake/ADL-L.pdf> (640 kB) and as
ref. 49 at <http://www.physics.indiana.edu/~hake>.
Heisenberg, W. 2007. "Physics & Philosophy: The Revolution in Modern
Sciences." Harper Perennial Modern Classics. Originally published in
1999 as "Physik und Philosophie." Amazon.com information at
<http://tinyurl.com/3qn4p3>.
Huberty, C.J. & J.D. Morris. 1988. "A single contrast test procedure,
Educational and Psychological Measurements 48: 567-578; online as a
1.1 MB pdf at <http://epm.sagepub.com/cgi/reprint/48/3/567>.
Kelly, A.E., R.A. Lesh, J.Y. Baek. 2008. "Handbook of Design Research
Methods in Education: Innovations in Teaching." Routledge Education;
publisher's information at <http://tinyurl.com/4eazqs>: "This
'Handbook' presents the latest thinking and current examples of
design research in education. Design-based research involves
introducing innovations into real-world practices (as opposed to
constrained laboratory contexts) and examining the impact of those
designs on the learning process. Designed prototype applications
(e.g., instructional methods, software or materials) and the research
findings are then cycled back into the next iteration of the design
innovation in order to build evidence of the particular theories
being researched, and to positively impact practice and the diffusion
of the innovation. The 'Handbook of Design Research Methods in
Education' -- THE defining book for the field -- fills a need in how
to conduct design research by those doing so right now. The chapters
represent a broad array of interpretations and examples of how
today's design researchers conceptualize this emergent methodology
across areas as diverse as educational leadership, diffusion of
innovations, complexity theory, and curriculum research.
Light, R.J., J.D. Singer, & J.B. Willett. 1990. "By Design: Planning
Research in Higher Education," Harvard University Press, publisher's
information at <http://www.hup.harvard.edu/catalog/LIGBYD.html>.
Michelson, A.A., and E.W. Morley. 1887. "On the relative motion of
earth and luminiferous ether." American Journal of Science 1(34):
333-345.
Newman, I., C. Newman, R. Brown, & S. McNeely. 2005. "Conceptual
Statistics for Beginners," 3rd edition. University Press of America,
publisher's information at <http://tinyurl.com/6pgyfd>.
Roessel, B. 2008. Re: PHYSLRNR Digest - 17 Jun 2008 to 18 Jun 2008
(#2008-104); PhysLrnR post of 19 Jun 2008 09:13:56-0400; online at
<http://tinyurl.com/65fjuy>. To access the archives of PhysLnR one
needs to subscribe :-( , but that takes only a few minutes by
clicking on <http://listserv.boisestate.edu/archives/physlrnr.html>
and then clicking on "Join or leave the list (or change settings)."
If you're busy, then subscribe using the "NOMAIL" option under
"Miscellaneous." Then, as a subscriber, you may access the archives
and/or post messages at any time, while receiving NO MAIL from the
list!
Ramlo, S. 2008. "Statistics resources," PhysLrnR post of 9 Jun 2008
09:49:14-0400; online at <http://tinyurl.com/5sgedw>. To access the
archives of PhysLnR one needs to subscribe :-( , but that takes only
a few minutes by clicking on
<http://listserv.boisestate.edu/archives/physlrnr.html> and then
clicking on "Join or leave the list (or change settings)." If you're
busy, then subscribe using the "NOMAIL" option under "Miscellaneous."
Then, as a subscriber, you may access the archives and/or post
messages at any time, while receiving NO MAIL from the list!
Rozeboom, W. W. 1960. "The fallacy of the null-hypothesis
significance test." Psychological Bulletin 57: 416-428. Available
online at: <http://psychclassics.yorku.ca/Rozeboom/>, thanks to
Christopher Green's "Classics in the History of Psychology"
<http://psychclassics.yorku.ca/>.
Schneider, B. M. Carnoy, J. Kilpatrick, W.H. Schmidt, R.J.
Shavelson. 2007. "Estimating Causal Effects Using Experimental and
Observational Designs." AERA, publishers information and FREE
download at
<http://www.aera.net/publications/Default.aspx?menu_id=46&id=3360>.
Slavin, R.E. 1992. "Research Methods in Education." Allyn & Bacon,
2nd ed. Amazon.com information at <http://tinyurl.com/4k59gm>.
Shadish, W.R., T.D. Cook, & D.T. Campbell. 2002."Experimental and
Quasi-Experimental Designs for Generalized Causal Inference."
Houghton Mifflin. Amazon.com information at
<http://tinyurl.com/6kel78>. Note the"Search Inside" feature.
Thompson, B. 2006. "Foundations of Behavioral Statistics: An
Insight-Based Approach." Guilford Press - information at
<http://tinyurl.com/vftol> and
<http://www.coe.tamu.edu/~bthompson/datasets.htm>.
Titus, A. 2006. "Assessment Analysis"; online at
<http://linus.highpoint.edu/~atitus/assess/> is a "web-based program
(CGI script) that helps teachers analyze test results perform
correlational analysis, and do factor analysis." Test analysis is a
tool used to analyze pre and post test data: t -test, normalized
gain (individual and class), effect size, max, min, mean, median,
standard deviation, KR-20, item difficulty, & point biserial
coefficient.
Wainer, H. & D.H. Robinson. 2003. "Shaping Up the Practice of Null
Hypothesis Significance Testing," Educational Researcher 32(7);
22-30; online as a 148 kB pdf at
<http://www.aera.net/publications/?id=399>.
---
To make changes to your subscription contact:
Bill Southerly ([EMAIL PROTECTED])
