I'll say some more about Benjamini-Hochberg's FDR. On Mon, 1 Dec 2003 06:13:38 +0000, "A. G. McDowell" <[EMAIL PROTECTED]> wrote:
> In article <[EMAIL PROTECTED]>, Rich Ulrich > <[EMAIL PROTECTED]> writes > > - I have comments on the question of corrections for multiple > >testing. And I'm asking folks for feedback on Benjamini > >and Hochberg's FDR as an alternative. > > > Not strictly relevant but there is also Holm-Bonferroni: > Given N tests, go through the N tail probabilities smallest first, and > stop at the rth test if its tail probability is > ALPHA/(N-r+1). Reject > all null hypotheses seen before you stop. So the first probability gets > a threshold equivalent to the Bonferroni correction, but the following > ones get treated successively more leniently. ... [ snip, rest] The "following ones" are treated more leniently, but Holm-B. is (it seems to me) a really minor correction -- or variation. The Benjamini-Hochberg FDR seems notably different, and that's why I am wondering about that one. It seems to me that it might change the need, which shows up too often, of switching entirely from the 5% test level. That cutoff is far too lenient if we are doing dozens or hundreds of tests, but the Bonferroni correction can be far too harsh. Astronomers use arbitrary numbers -- arbitrary so far as social scientists are concerned. I think that the whole future of 'data-mining' is going to abandon 5%, if the choice is always the straight, nominal level, or the straight, Bonferroni-corrected version. It seems to me that this FDR has some possibility of giving a standard that can still be shared *somewhat*. - if I have it right - For sequential testing: Starting with 50 variables, 5% level, you perform all the tests and then sort them into order by 'nominal p-level.' Using Holms-Bonferri, you would test the first variable at 5%/50 or .001; the next at 5%/49; the next at 5%/48; and you quit when your ordered test results don't pass the criterion. For 25 tests to pass, number 25 needs to meet the nominal test size of 5%/26, or 0.2%. Using Benjamini-Hochberg, you also test the first one at 5%/50 or .001; but you consider the set from the *other* end, and you can *multiply* by the *fraction* of variables involved. Thus, if you started with 50 tests and a 5% error rate, using FDR, you declare 25 of them 'significant' if 25 (or, one-half) have a nominal p-level less than 2.5% -- one-half of 5%. Let us say that you graph the increasing rank of p-values on the x-axis, and the actual p on the y-axis. The left-most point is 5%/50 for the y-value for both. The right-most point is 5% for both. For Benjamini-Hochberg, you can draw a straight line between the points, and the cutoff for what you get to claim as significant is the *last* place the p-value is smaller than the drawn line, while moving toward larger p-values. For Holm-Bonferroni, the rule says, there is a line that stays lower (1/50, 1/49, 1/48, ... instead of 1/50, 2/50, 3/50,...), and you only get the claim significance until some *first* point fails the criterion. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html "Taxes are the price we pay for civilization." . . ================================================================= Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at: . http://jse.stat.ncsu.edu/ . =================================================================
