Wow. You do have a rationale for your answer to 4. I completely
disagree with it but it's a rational.
At 02:42 PM 7/2/2010, David Pennock wrote:
Oops, I meant 1,000,000 for question 5.
I'm not a statistician. I reasoned this way: If I somehow "explore"
lots of hypotheses I need to correct for that, otherwise I may
overfit. In 2, I only checked one randomly chosen hypothesis and
stopped exploring after that.
In 4 and 5, I've in some way explored all 1,000,000 hypotheses at
least implicitly either via my own intuition or via algorithmic
pruning, so I need to correct for that.
Correcting for all 1,000,000 is overkill in some cases and few
people would actually do it, but to be "safe" I was advocating that
you technically should.
I could easily have reasoned incorrectly. :-) Thanks for livening up
this list.
thanks,
Dave
Rich Neapolitan wrote:
Dear David,
Thanks for the reply. Question 5 does not have a yes or no answer.
I find it quite odd that you say yes to 4 and no to 2. Can you
explain? If you were a classic frequentist, you'd say no to both. I
would say yes to 2 and no to 4.
By the way, I just submitted another post with my answers.
Best,
Rich
At 07:45 AM 7/2/2010, David Pennock wrote:
I'll bite.
1. yes
2. no
3. yes
4. yes
5. yes
Rich Neapolitan wrote:
Let's have some fun in this group again instead of just posting
about conferences, post docs, and new books, etc. I offer you
this quiz about the use of the Bonferroni (or any other) correction:
I have 1,000,000 hypotheses that are not mutually exclusive.
1. I test them all. Do I apply the Bonferroni correction?
2. I plan to test them all, but I run out of resources after
testing only one of them. Do I apply the Bonferroni correction?
3. I test one of them, and a year later test the others. Do I
apply the Bonferroni correction to the one tested early? If so, when?
4. I only test the first one because that is the one I suspect.
Do I apply the Bonferroni correction?
5. I run an algorithm that prunes unlikely hypotheses. I end up
testing only 100,000. Do I apply the Bonferroni correction for
100,000 or for 1,000,000 hypotheses.
I have two colleagues who make their living as statisticians, one
is a good Bayesian and the other a good frequentist (oxymoron?).
They both have different answers, and neither agrees with me. It
would be interesting to learn how some of you view this matter.
Best,
Rich
Rich Neapolitan
Professor and Chair of Computer Science
Northeastern Illinois University
5500 N. St. Louis
Chicago, Il 60625
------------------------------------------------------------------------
_______________________________________________
uai mailing list
uai@ENGR.ORST.EDU
https://secure.engr.oregonstate.edu/mailman/listinfo/uai
Rich Neapolitan
Professor and Chair of Computer Science
Northeastern Illinois University
5500 N. St. Louis
Chicago, Il 60625
Rich Neapolitan
Professor and Chair of Computer Science
Northeastern Illinois University
5500 N. St. Louis
Chicago, Il 60625
_______________________________________________
uai mailing list
uai@ENGR.ORST.EDU
https://secure.engr.oregonstate.edu/mailman/listinfo/uai
