re: [tips] Frequency of Type I errors in published research

[Michael Burman]

Actually, I didn't regret asking at all.  It's crystal clear now - that many 
null hypotheses are actually false prevents us from knowing how many Type 1 
errors are made.  Thanks all!

MIke


---
Michael A Burman Ph.D.
Assistant Professor 
Dept. of Psychology 
K-12 Outreach Coordinator for the Neurosciences
University of New England

328 Decary Hall
207-602-2301
mbur...@une.edu


---
You are currently subscribed to tips as: arch...@jab.org.
To unsubscribe click here: 
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=17105
or send a blank email to 
leave-17105-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu

Re: [tips] Frequency of Type I errors in published research.

[Christopher Green]

Alpha is the proportion of times one would, WHEN THE NULL HYPOTHESIS IS TRUE, 
get data so extreme that one would reject the null hypothesis (falsely). That 
is what p=.05 represents. However, WHEN THE NULL HYPOTHESIS IS FALSE, alpha 
doesn't tell us anything at all. Beta (which we almost never even bother to 
measure, alas!) is the proportion of times we would get data so close to the 
parameter specified by the null that we would (falsely) fail to reject the null 
hypothesis. Power (1-beta) is the proportion of times we correctly reject the 
null (i.e, when the null is false).

So, it is impossible to tell what overall proportion of times the null is 
rejected (as a rather unreliable proxy for number of papers published) are 
cases of the null being *falsely* rejected. The two cells (power + alpha) don't 
"add up to one," so to speak, and since we don't know what (roughly speaking) 
the "base rate" is (i.e., the number of times we test true as opposed to false 
null hypotheses) we can't correctly weight them (power or alpha)  so that we 
can get the number we want.

Chris
---
Christopher D. Green
Department of Psychology
York University
Toronto, ON M3J 1P3
Canada

chri...@yorku.ca
http://www.yorku.ca/christo/
==



On 2012-04-02, at 10:35 AM, Michael Burman wrote:

>  
> 
>  
> 
>  
> 
> I get a sense that I'm going to regret asking this question, but why wouldn't 
> setting the alpha value at .05 result in about 5% false positives in the 
> literature?  Are people suggesting that the true false positive rate would be 
> lower?  I get why it would be higher (statistical tricks, bias, research 
> short cuts, etc), but not why it would be lower.  
> 
> Mike
> 
> 
> ---
> Michael A Burman Ph.D.
> Assistant Professor 
> Dept. of Psychology 
> K-12 Outreach Coordinator for the Neurosciences
> University of New England
> 
> 328 Decary Hall
> 207-602-2301
> mbur...@une.edu
> 
> 
> ---
> 
> You are currently subscribed to tips as: chri...@yorku.ca.
> 
> To unsubscribe click here: 
> http://fsulist.frostburg.edu/u?id=430248.781165b5ef80a3cd2b14721caf62bd92&n=T&l=tips&o=17089
> 
> (It may be necessary to cut and paste the above URL if the line is broken)
> 
> or send a blank email to 
> leave-17089-430248.781165b5ef80a3cd2b14721caf62b...@fsulist.frostburg.edu
> 
> 
>  
> 
>  


---
You are currently subscribed to tips as: arch...@jab.org.
To unsubscribe click here: 
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=17094
or send a blank email to 
leave-17094-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu

RE: [tips] Frequency of Type I errors in published research.

[Wuensch, Karl L]

Should I send him this link:  

http://core.ecu.edu/psyc/wuenschk/StatHelp/Type1.htm  ??

Cheers,

Karl L. Wuensch


-Original Message-
From: Michael Palij [mailto:m...@nyu.edu] 
Sent: Monday, April 02, 2012 8:52 AM
To: Teaching in the Psychological Sciences (TIPS)
Cc: Michael Palij
Subject: re: [tips] Frequency of Type I errors in published research.

On Sun, 01 Apr 2012 15:29:21 -0700, Karl L Wuensch,
>  Referring to the file drawer problem and data falsification, 
>the president of the Association for Psychological Science recently 
>wrote:  "These and related factors should tend to inflate 'false 
>positives' (aka Type I errors), which leads inexorably to the 
>pessimistic conclusion that some unknown (but considerably higher than 
>.05) proportion of our field's published effects are not true effects."

Karl is referring to an article by cognitive psychologist Douglas Medin who is 
currently one of the rotating presidents of APS.  The article is on research 
misconduct and how to deal with it. The article can be accessed here:
http://www.psychologicalscience.org/index.php/publications/observer/obsonline/a-science-we-can-believe-in.html

NOTE:  There is a comment area at the bottom of the webpage where comments, 
questions, and rants can be submitted.

It is also useful to note the Medin's quote above is building upon the research 
presented in the following article:

Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn (2011).
False-Positive Psychology: Undisclosed Flexibility in Data Collection and 
Analysis Allows Presenting Anything as Significant Psychological Science 
October 2011 0956797611417632, first published on October 17, 2011
doi:10.1177/0956797611417632

Amd can be accessed here:
http://pss.sagepub.com/content/22/11/1359

A popular media article on Simmons et al is provided on the Psychology Today 
website; see:
http://www.psychologytoday.com/blog/the-good-life/20/false-positive-psychology

Simmons et al review the little "tricks" that researchers engage in to achieve 
statistical significance in a study but which often failures to replicate.  
This suggests that the false positive rate (i.e., the number of articles that  
represent false rejection of the null
hypothesis) of journals is inflated relative to the 5% rate that might be 
expected if certain conditions for statistical testing are met (e.g., sample 
size, effect size, statistical power, etc.).

Simmons et al can be seen as following in the tradition of John Ioannidis, 
whose most notorious article is:

Ioannidis JP. (2005). Why most published research findings are false.
PLoS Med. 2005 Aug;2(8):e124. Epub 2005 Aug 30.

Which can be accessed for free through this website (see upper right hand 
corner):
http://www.ncbi.nlm.nih.gov/pubmed/16060722/

The magazine "The Atlantic" has an article on Ioannidis and the implications of 
his research which might be worth reading; see:
http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/

Some Tipsters might recognize a couple of the above references from previous 
postings here.

>  Am I misreading this, or does he imply (incorrectly) that use 
>of the
>.05 criterion of statistical significance would be expected to result 
>in 5% of published research findings being Type I errors?

Short answer: Yes

Perhaps one should make a comment on the webpage with Medin's article to point 
out this error.

-Mike Palij
New York University
m...@nyu.edu

---
You are currently subscribed to tips as: wuens...@ecu.edu.
To unsubscribe click here: 
http://fsulist.frostburg.edu/u?id=13060.c78b93d4d09ef6235e9d494b3534420e&n=T&l=tips&o=17083
or send a blank email to 
leave-17083-13060.c78b93d4d09ef6235e9d494b35344...@fsulist.frostburg.edu

---
You are currently subscribed to tips as: arch...@jab.org.
To unsubscribe click here: 
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=17093
or send a blank email to 
leave-17093-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu

RE: [tips] Frequency of Type I errors in published research.

[David Epstein]

On Mon, 2 Apr 2012, Marc Carter went:


A p of .05 does not automatically mean that 5% of the positive
results are false positives.  It just means that on any given test
(if everything else is correctly done) there's a probability of .05
that you're getting a false positive.


No--there's no way to calculate the probability that you're getting a
false positive.

The p value tells you the probability that you WOULD get a false
positive (i.e., your current results) IF THE NULL HYPOTHESIS WERE
TRUE.

A lot of null hypotheses aren't true.  How many?  Not calculable.

But that makes 5% an absolute worst-case scenario (if everything is
done right).

--David Epstein
  da...@neverdave.com

---
You are currently subscribed to tips as: arch...@jab.org.
To unsubscribe click here: 
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=17092
or send a blank email to 
leave-17092-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu

RE: [tips] Frequency of Type I errors in published research.

[Marc Carter]

A p of .05 does not automatically mean that 5% of the positive results are 
false positives.  It just means that on any given test (if everything else is 
correctly done) there's a probability of .05 that you're getting a false 
positive.

Add to that the fact that in many pieces of research the obtained p of getting 
those results is far below .05.

In my mind I think of it a lot like the Gamblers' Fallacy.  They're (or should 
be) independent events.

m

--
Marc Carter, PhD
Associate Professor of Psychology
Chair, Department of Behavioral and Health Sciences
College of Arts & Sciences
Baker University
--

From: Michael Burman [mailto:mbur...@une.edu]
Sent: Monday, April 02, 2012 9:36 AM
To: Teaching in the Psychological Sciences (TIPS)
Subject: re: [tips] Frequency of Type I errors in published research.










I get a sense that I'm going to regret asking this question, but why wouldn't 
setting the alpha value at .05 result in about 5% false positives in the 
literature?  Are people suggesting that the true false positive rate would be 
lower?  I get why it would be higher (statistical tricks, bias, research short 
cuts, etc), but not why it would be lower.

Mike


---
Michael A Burman Ph.D.
Assistant Professor
Dept. of Psychology
K-12 Outreach Coordinator for the Neurosciences
University of New England

328 Decary Hall
207-602-2301
mbur...@une.edu<mailto:mbur...@une.edu>



---

You are currently subscribed to tips as: 
marc.car...@bakeru.edu<mailto:marc.car...@bakeru.edu>.

To unsubscribe click here: 
http://fsulist.frostburg.edu/u?id=13029.76c7c563b32ad9d8d09c72a2d17c90e1&n=T&l=tips&o=17089

(It may be necessary to cut and paste the above URL if the line is broken)

or send a blank email to 
leave-17089-13029.76c7c563b32ad9d8d09c72a2d17c9...@fsulist.frostburg.edu<mailto:leave-17089-13029.76c7c563b32ad9d8d09c72a2d17c9...@fsulist.frostburg.edu>








The information contained in this e-mail and any attachments thereto ("e-mail") 
is sent by Baker University ("BU") and is intended to be confidential and for 
the use of only the individual or entity named above. The information may be 
protected by federal and state privacy and disclosures acts or other legal 
rules. If the reader of this message is not the intended recipient, you are 
notified that retention, dissemination, distribution or copying of this e-mail 
is strictly prohibited. If you have received this e-mail in error please 
immediately notify Baker University by email reply and immediately and 
permanently delete this e-mail message and any attachments thereto. Thank you.

---
You are currently subscribed to tips as: arch...@jab.org.
To unsubscribe click here: 
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=17090
or send a blank email to 
leave-17090-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu

re: [tips] Frequency of Type I errors in published research.

[Michael Burman]

I get a sense that I'm going to regret asking this question, but why wouldn't 
setting the alpha value at .05 result in about 5% false positives in the 
literature?  Are people suggesting that the true false positive rate would be 
lower?  I get why it would be higher (statistical tricks, bias, research short 
cuts, etc), but not why it would be lower.  

Mike


---
Michael A Burman Ph.D.
Assistant Professor 
Dept. of Psychology 
K-12 Outreach Coordinator for the Neurosciences
University of New England

328 Decary Hall
207-602-2301
mbur...@une.edu


---
You are currently subscribed to tips as: arch...@jab.org.
To unsubscribe click here: 
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=17089
or send a blank email to 
leave-17089-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu

RE:[tips] Frequency of Type I errors in published research.

[Marc Carter]

That's how I read it.

And that's how I've read it in several stats books.  Bugs the hell out of me.

m

--
Marc Carter, PhD
Associate Professor of Psychology
Chair, Department of Behavioral and Health Sciences
College of Arts & Sciences
Baker University
--

From: Wuensch, Karl L [mailto:wuens...@ecu.edu]
Sent: Sunday, April 01, 2012 5:29 PM
To: Teaching in the Psychological Sciences (TIPS)
Subject: [tips] Frequency of Type I errors in published research.










  Referring to the file drawer problem and data falsification, the 
president of the Association for Psychological Science recently wrote:  "These 
and related factors should tend to inflate 'false positives' (aka Type I 
errors), which leads inexorably to the pessimistic conclusion that some unknown 
(but considerably higher than .05) proportion of our field's published effects 
are not true effects."

  Am I misreading this, or does he imply (incorrectly) that use of the 
.05 criterion of statistical significance would be expected to result in 5% of 
published research findings being Type I errors?


---

You are currently subscribed to tips as: 
marc.car...@bakeru.edu.

To unsubscribe click here: 
http://fsulist.frostburg.edu/u?id=13029.76c7c563b32ad9d8d09c72a2d17c90e1&n=T&l=tips&o=17069

(It may be necessary to cut and paste the above URL if the line is broken)

or send a blank email to 
leave-17069-13029.76c7c563b32ad9d8d09c72a2d17c9...@fsulist.frostburg.edu








The information contained in this e-mail and any attachments thereto ("e-mail") 
is sent by Baker University ("BU") and is intended to be confidential and for 
the use of only the individual or entity named above. The information may be 
protected by federal and state privacy and disclosures acts or other legal 
rules. If the reader of this message is not the intended recipient, you are 
notified that retention, dissemination, distribution or copying of this e-mail 
is strictly prohibited. If you have received this e-mail in error please 
immediately notify Baker University by email reply and immediately and 
permanently delete this e-mail message and any attachments thereto. Thank you.

---
You are currently subscribed to tips as: arch...@jab.org.
To unsubscribe click here: 
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=17088
or send a blank email to 
leave-17088-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu

re: [tips] Frequency of Type I errors in published research.

[Michael Palij]

On Sun, 01 Apr 2012 15:29:21 -0700, Karl L Wuensch,
>  Referring to the file drawer problem and data falsification, the
>president of the Association for Psychological Science recently wrote:  "These
>and related factors should tend to inflate 'false positives' (aka Type I
>errors), which leads inexorably to the pessimistic conclusion that some unknown
>(but considerably higher than .05) proportion of our field's published effects
>are not true effects."

Karl is referring to an article by cognitive psychologist Douglas Medin
who is currently one of the rotating presidents of APS.  The article is
on research misconduct and how to deal with it. The article can be
accessed here:
http://www.psychologicalscience.org/index.php/publications/observer/obsonline/a-science-we-can-believe-in.html

NOTE:  There is a comment area at the bottom of the webpage where
comments, questions, and rants can be submitted.

It is also useful to note the Medin's quote above is building upon
the research presented in the following article:

Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn (2011).
False-Positive Psychology: Undisclosed Flexibility in Data Collection
and Analysis Allows Presenting Anything as Significant
Psychological Science October 2011 0956797611417632,
first published on October 17, 2011
doi:10.1177/0956797611417632

Amd can be accessed here:
http://pss.sagepub.com/content/22/11/1359

A popular media article on Simmons et al is provided on the Psychology
Today website; see:
http://www.psychologytoday.com/blog/the-good-life/20/false-positive-psychology

Simmons et al review the little "tricks" that researchers engage in
to achieve statistical significance in a study but which often failures
to replicate.  This suggests that the false positive rate (i.e., the
number of articles that  represent false rejection of the null
hypothesis) of journals is inflated relative to the 5% rate that might
be expected if certain conditions for statistical testing are met
(e.g., sample size, effect size, statistical power, etc.).

Simmons et al can be seen as following in the tradition of John Ioannidis,
whose most notorious article is:

Ioannidis JP. (2005). Why most published research findings are false.
PLoS Med. 2005 Aug;2(8):e124. Epub 2005 Aug 30.

Which can be accessed for free through this website (see upper
right hand corner):
http://www.ncbi.nlm.nih.gov/pubmed/16060722/

The magazine "The Atlantic" has an article on Ioannidis and the
implications of his research which might be worth reading; see:
http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/

Some Tipsters might recognize a couple of the above references
from previous postings here.

>  Am I misreading this, or does he imply (incorrectly) that use of the
>.05 criterion of statistical significance would be expected to result in 5% of
>published research findings being Type I errors?

Short answer: Yes

Perhaps one should make a comment on the webpage with Medin's article
to point out this error.

-Mike Palij
New York University
m...@nyu.edu

---
You are currently subscribed to tips as: arch...@jab.org.
To unsubscribe click here: 
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=17083
or send a blank email to 
leave-17083-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu