At 02:45 PM 6/7/2005, Jesse Mazer wrote: (snip)

Of course in this example Feynman did not anticipate in advance whatlicence plate he'd see, but the kind of "hindsight bias" you are engagingin can be shown with another example. Suppose you pick 100 random wordsout of a dictionary, and then notice that the list contains the words"sun", "also", and "rises"...as it so happens, that particular 3-word"gestalt" is also part of the title of a famous book, "the sun also rises"by Hemingway. Is this evidence that Hemingway was able to anticipate theresults of your word-selection through ESP? Would it be fair to test forESP by calculating the probability that someone would title a book withthe exact 3-word gestalt "sun, also, rises"? No, because this would betailoring the choice of gestalt to Hemingway's book in order to make itseem more unlikely, in fact there are 970,200 possible 3-word gestalts youcould pick out of a list of 100 possible words, so the probability that abook published earlier would contain *any* of these gestalts is a lothigher than the probability it would contain the precise gestalt "sun,also, rises". Selecting a precise target gestalt on the basis of the factthat you already know there's a book/story containing that gestalt is anexample of hindsight bias--in the Heinlein example, you wouldn't havechosen the precise gestalt of Szilard/lens/beryllium/uranium/bomb from along list of words associated with the Manhattan Project if you didn'talready know about Heinlein's story.RM wrote:

In two words: Conclusions first.

`Can you really offer no scientific procedure to evaluate Heinlein's`

`story? At the cookie jar level, can you at least grudgingly admit that the`

`word "Szilard" sure looks like "Silard"? Sounds like it too. Or is that a`

`coincidence as well? What are the odds. Should be calculable--how many`

`stories written in 1939 include the names of Los Alamos scientists in`

`conjunction with the words "bomb" , "uranium. . ."`

You're shaking your head. This, I assume is already a done deal, for you.

`And that, in my view, is the heart of the problem. Rather than swallow`

`hard and look at this in a non-biased fashion, you seem to be glued to the`

`proposition that (1) it's intractable or (2) it's not worth analyzing`

`because the answer is obvious.`

`If your answer is (1), then fine. Let others worry about it. But if your`

`answer is (2), then congratulations---you've likely committed a Type II`

`error. In all of your posts, you seem to present reasons why the Heinlein`

`story should not be investigated because (I'm paraphrasing, of course) it's`

`"obviously" not worthy of investigation. You exclude ALL the`

`evidence---even the Bonferroni doesn't do that. Logically, if you exclude`

`all the evidence, then the probability that you might miss something go to.`

`. .1. One hundred percent.`

`When one chooses to use, say Spearman Correlation Coefficients to evaluate`

`multiple pairs, the usual protocol involves using the Bonferroni`

`correction--in which the alpha (often at 0.05) is divided by some multiple`

`of the number of pairs evaluated--usually simply the number of pairs. A`

`thousand pairs? then, the alpha should be divided by a thousand and the`

`resultant p value accepted as similar to a single p value of 0.05. Problem`

`is, this sort of trick will cost you statistical power. You may not decide`

`something is significant when it is not, but you may also throw out a value`

`that truly is important. As the type I error risk goes down, the Type II`

`error risk goes up. (Reducing alpha increases beta (the probability of`

`making a Type II error.) There are reputable statisticians who suggest not`

`using the Bonferroni at all. In my work, I evaluate cancer rates against`

`radioisotopes in nuclear fallout---but I require a very high Z score for`

`significance.`

`I've yet to see a good protocol defined here to evaluate the Heinlein`

`story, most prefer to fall back onto the soft couch of bias and`

`prejudgment. But in doing so, your beta goes out the roof--and you`

`guarantee yourself that you'll never recognize *anything* as`

`significant. It would seem that it would be far easier and more`

`scientifically sound to just admit that you are aware of no tools that can`

`properly evaluate it.`

`PS: Note I haven't mentioned anything about proof or causation---merely`

`the ability to apply the scientific method--properly free of bias---to a`

`set of circumstances. So far (as with the Thompkins quote)--it looks like`

`"conclusions first, justification later."`

`Hope your drug company doesn't use the same protocol. Because`

`*that* wouldn't be right, would it?;-)`

RM