'bout time. Color me Bayesian. On Thu, Feb 26, 2015 at 2:10 PM, Kenneth Lettow <[email protected]> wrote:
> Just came across this article. It seems pertinent to this discussion. > > > http://www.sciencebasedmedicine.org/psychology-journal-bans-significance-testing/ > > > > On Wed, Feb 25, 2015 at 10:55 AM, Ian Clark <[email protected]> wrote: > > > On Wed, Feb 25, 2015 at 3:31 PM, Devon McCormick <[email protected]> > > wrote: > > > Ian - setting up a framework for this sort of experiment could prove > > > valuable in a wide variety of fields, not just audio testing. > > > > > > However, you may be overshadowing the point of the example from the > book: > > > it's an illustration of the relevance of prior knowledge. As I > remember > > > it, not having the book in front of me, it goes something like this: > > > > > > There are these three examples of evidence supporting a hypothesis: > > > > > > 1) A lady claims to be able to distinguish, by tasting a cup of tea > with > > > milk, whether the tea was added before the milk or the milk before the > > > tea. You test her ten times and she is correct every time. > > > > > > 2) Someone claims to be able to distinguish by ear a score written by > > > Mozart from one not written by Mozart. You test him ten times and he > is > > > correct every time. > > > > > > 3) A drunken friend claims to be able to predict the result of a coin > > > toss. You test him ten times and he is correct every time. > > > > > > Since the empirical evidence in all three cases is identical, why would > > we > > > not believe all three hypotheses to be equally well-proved? > > > > Aha! -- do I detect a Bayesian? Or a disciple of de Finetti? > > > > I recall a Real statistician taking me aside and telling me: "the > > trouble with you people is that you don't think anything's happened in > > Statistics in the last 20 years." -- and that was way back in 1989. > > > > I wouldn't know how to write a program based on a Bayesian analysis to > > replace my classical "H0-rejection" analysis. I suspect it's no use > > even trying. De Finetti, I recall, required you to input your a-priori > > subjective prediction of the outcome as a normalized distribution > > before anything else. > > > > Maybe I'll just stick to a disclaimer… > > "This assessment is based on the assumption that you are not an > > occultist, a superhero, mentally certifiable or under the influence of > > drugs or alcohol." > > …so bang goes all my user audience of musicians and songwriters! > > :-D > > > > > > > > On Wed, Feb 25, 2015 at 3:31 PM, Devon McCormick <[email protected]> > > wrote: > > > Ian - setting up a framework for this sort of experiment could prove > > > valuable in a wide variety of fields, not just audio testing. > > > > > > However, you may be overshadowing the point of the example from the > book: > > > it's an illustration of the relevance of prior knowledge. As I > remember > > > it, not having the book in front of me, it goes something like this: > > > > > > There are these three examples of evidence supporting a hypothesis: > > > > > > 1) A lady claims to be able to distinguish, by tasting a cup of tea > with > > > milk, whether the tea was added before the milk or the milk before the > > > tea. You test her ten times and she is correct every time. > > > > > > 2) Someone claims to be able to distinguish by ear a score written by > > > Mozart from one not written by Mozart. You test him ten times and he > is > > > correct every time. > > > > > > 3) A drunken friend claims to be able to predict the result of a coin > > > toss. You test him ten times and he is correct every time. > > > > > > Since the empirical evidence in all three cases is identical, why would > > we > > > not believe all three hypotheses to be equally well-proved? > > > > > > > > > On Wed, Feb 25, 2015 at 10:19 AM, Ian Clark <[email protected]> > > wrote: > > > > > >> @Chris - just what I've been looking for! > > >> > > >> However did I miss it? -- I guess it must have eluded my choice of > > >> keywords. > > >> Next time I'll use Google with site:jsoftware.com > > >> > > >> On Wed, Feb 25, 2015 at 2:27 PM, chris burke <[email protected]> > > wrote: > > >> >> Has anyone built a standalone Mac app using JQt? > > >> > > > >> > Please see http://www.jsoftware.com/jwiki/Guides/J8%20Standalone > > >> > > > >> > On 25 February 2015 at 06:09, Ian Clark <[email protected]> > > wrote: > > >> > > > >> >> @Henry -- thanks for your comments. Great! > > >> >> > > >> >> IMO this is just the sort of discussion I would like to see aired > in > > >> >> public. Though maybe do the more philosophical stuff in Chat? > > >> >> Ideally I would like a summary of the J community's findings > > >> >> documented on a Jwiki page for wider consumption. > > >> >> > > >> >> Further comments in-line… > > >> >> > > >> >> > > >> >> On Wed, Feb 25, 2015 at 4:27 AM, Henry Rich <[email protected]> > > >> wrote: > > >> >> > We should take this off-group, but I'm replying in public because > > if > > >> I'm > > >> >> > wrong I would like to be corrected (and I'm only an amateur > > >> >> statistician): > > >> >> > > >> >> That's exactly why I'm appealing to the forum too. > > >> >> …To the annoyance of Real Statisticians, no doubt, because this > must > > >> >> be elementary stuff to them. > > >> >> But Wikipedia -- which you'd expect to give simple answers to > simple > > >> >> questions which laypeople want to ask and need to ask -- approaches > > >> >> the whole issue like a cat circling a bowl of hot porridge. > > >> >> …If you're a layperson, just try working out how to score the "Lady > > >> >> Tasting Tea" experiment from these pages… > > >> >> https://en.wikipedia.org/wiki/Binomial_test > > >> >> https://en.wikipedia.org/wiki/Binomial_distribution > > >> >> https://en.wikipedia.org/wiki/Bernoulli_distribution > > >> >> https://en.wikipedia.org/wiki/Bernoulli_trial > > >> >> https://en.wikipedia.org/wiki/Bernoulli_process > > >> >> > > >> >> As a Human Factors *engineer* -- I've been a professional *user* of > > >> >> hypothesis-testing but an amateur Statistician. > > >> >> …Or should that be Probabilist? Or even Epistemologist? > > >> >> > > >> >> Plus… now I'm retired, I'm getting rusty. > > >> >> > > >> >> Plus… I can't find precise enough documentation of JAL verb: > > >> binomialprob. > > >> >> Like… what's the semantics of the 3rd entry of (y) (styled "minimum > > >> >> number of successes (s)") when y has only 3 entries? Can it be > called > > >> >> "minimum" any more? What I've concluded, after a bit of RTFC plus a > > >> >> few idiot tests, is: > > >> >> > > >> >> (binomialprob 0.5,N,s) -: (binomialprob 0.5,N,s,N) > > >> >> > > >> >> Plus… has this doggie got 2 tails or just 1?? > > >> >> > > >> >> > > >> >> > I think you are calling binomialprob correctly but I have some > > >> >> objections to > > >> >> > your use of the result. > > >> >> > > > >> >> > 1. I think your rejectH0 should use 1 - -: CONFIDENCE instead of > > >> >> > 1-CONFIDENCE. > > >> >> > > > >> >> > The question is, "How likely is a result as weird as I am > seeing, > > >> >> assuming > > >> >> > H0?" You should not bias "weird" by assuming that weird results > > will > > >> be > > >> >> > correct guesses - they could just as likely be incorrect guesses. > > To > > >> >> ensure > > >> >> > that you reject 95% of the purely-chance deviations of a certain > > size, > > >> >> that > > >> >> > 95% should be centered around the mean, not loaded toward one > side. > > >> >> > > >> >> The "1-tail-or-2?" question -- or so I thought at first. > > >> >> But it's deeper than that. It's much more serious. Serious enough > to > > >> >> be the key issue for me. > > >> >> Which is precisely why I want to be *sure*. Sure enough to argue my > > >> >> case to a determined layperson. Not merely make an inspired guess, > as > > >> >> most people would in an industrial situation (…knowing no one else > > >> >> knows enough statistics to dare to challenge you!) > > >> >> > > >> >> What I understand @Henry to be saying is: should the 5% area under > > the > > >> >> binomial distribution curve, which sets the pass/fail threshold, be > > >> >> shared equally between both tails? Even if one tail happens to be > in > > >> >> fairyland? > > >> >> > > >> >> What I mean by that last remark is… > > >> >> If The Lady Tasting Tea (TLTT) gets every trial *wrong*, then she's > > >> >> *not* a monkey flipping a fair coin. It's a very biased coin! > > >> >> She is sending a strong signal that she can be depended upon (…with > > X% > > >> >> confidence) to make the wrong decision. > > >> >> But I don't want to credit her this as evidence to support her > claim > > >> >> she can tell the difference (…at least, not tell it correctly). > > >> >> This is what makes TLTT different from detecting a biased coin by > > >> >> repeated tosses. > > >> >> > > >> >> What's to do? > > >> >> > > >> >> > > >> >> > are there really people who think optical might be better than > > USB?? > > >> >> > > >> >> Oh-ho-ho! -- yes, they can still be found. > > >> >> Hi-Fi buffs have not become extinct, and the (undead?) audio > industry > > >> >> still lives off their lifeblood. > > >> >> > > >> >> > > >> >> > This is digital communication, no? 44K samples/sec, 2 channels, > 20 > > >> >> bits/sample, > > >> >> > needs 2Mb/sec max out of 480Mb/sec rated USB speed... how could > > that > > >> not > > >> >> be > > >> >> > enough? > > >> >> > > >> >> My interlocutor claims it's like the group was there, in his front > > >> >> room, playing "just for him". > > >> >> Now this guy is an intelligent chap, a developer of digital musical > > >> >> instruments and a sound engineer as well as being an accomplished > > >> >> musician. He sends me two MP3s (…yes, lossy MP3s!) to support his > > >> >> claim. I drop these into Audacity and inspect the waveform at very > > >> >> fine detail and I cannot for the life of me detect any difference. > > >> >> So I know, as sure as God made little Apples, that I'm not going to > > >> >> *hear* any difference. > > >> >> But I've got lo-fi ears. In fact I'm half-deaf. Most of what I > hear I > > >> >> imagine. Mostly I get it right with people (I think…) But I don't > > know > > >> >> what subliminal cues I'm using to do so. It's the "clever Hans" > > >> >> effect. > > >> >> > > >> >> Maybe there are people who *can* tell the difference? But from my > > >> >> pondering the figures, like you have, plus eyeballing the > waveforms, > > >> >> we're talking about magical superpowers here. > > >> >> > > >> >> > > >> >> > It was ever thus... when I last looked at this sort of thing, 20 > > years > > >> >> back, > > >> >> > the debate was whether big fat expensive cables would make a > > >> difference. > > >> >> > Bob Pease, a respected analog engineer, pointed out that it was > > >> >> impossible, > > >> >> > and James Randi had a bet that no one could discern $7000 cables > > from > > >> >> > ordinary speaker wire, but still the non-EEs have their > > >> superstitions...] > > >> >> > > >> >> That's around the time my son was spending all his pocket-money on > > big > > >> >> fat speaker cables and gold-plated jack-plugs. > > >> >> Now he's teaching a Theory of Knowledge course (…yes, > Epistemology!) > > >> >> at a school in Hong Kong. He is greedy to get his hands on my > little > > >> >> program, and dispel a few lingering superstitions masquerading as > > >> >> received wisdom about science. > > >> >> > > >> >> I want to package it up and send it to him, but I don't want to ask > > >> >> him to install J on his Mac because not only will he grouse like > heck > > >> >> about fairy software but it will discourage him sharing the app > with > > >> >> his colleages, who share his sentiments. > > >> >> > > >> >> I know how to package up a standalone Mac app in J602, but J602 and > > my > > >> >> packaged apps no longer work out-of-the-box on the Mac under > Yosemite > > >> >> (it's to do with 32-bit Java). Has anyone built a standalone Mac > app > > >> >> using JQt? If so I'd dearly love to see a monkey-see monkey-do page > > on > > >> >> Jwiki. I'll write one myself, but it'll be a year before I can get > > >> >> round to it. > > >> >> > > >> >> > > >> >> > 2. Why 95%? I would fear that someone who is thinking about > > optical > > >> >> cable > > >> >> > would rest uneasy with a 5-10% chance that they have not spent > > enough > > >> on > > >> >> > quality audio. Why not simply report, "A monkey with a coin to > > toss > > >> >> would > > >> >> > do as well as you y% of the time. Most researchers accept > results > > as > > >> >> > significant only if the monkey would do as well less than 5% of > the > > >> time. > > >> >> > Take more samples if you want less uncertainty." > > >> >> > > >> >> 95% is just for the sake of argument. 99% is there as an option. > IMO > > >> >> more options are neither necessary nor advisable. > > >> >> The number of trials can be varied too. I'd like to offer 10 or 20 > > >> >> trials. But 20 gets tedious, so I'm offering the option to give up > > >> >> when you're bored and score the number you've done. > > >> >> (This is an app for discretionary users -- we're not paying our > > >> >> subjects $10 a session.) > > >> >> > > >> >> Anything under 7 trials fails to reject H0 however many successes. > > But > > >> >> that's dependent on the value of CONFIDENCE and how it's to be > > >> >> applied. But only to make a difference of 1 or maybe 2 trials. > > >> >> I'm finding in practice that with such a low number of trials as > 10, > > >> >> anything short of 100% correct is statistical hairsplitting when it > > >> >> comes to rejecting H0. With 20 trials there's more leeway: you're > > >> >> allowed to get 3 or 4 wrong before the app rubbishes you. > > >> >> > > >> >> As for your wording: it's theoretically sound, but a trifle > > insulting. > > >> >> Performing musicians have sizeable egos and wouldn't like to be > rated > > >> >> along with performing monkeys. :-) > > >> >> > > >> >> Ian > > >> >> > > ---------------------------------------------------------------------- > > >> >> For information about J forums see > > http://www.jsoftware.com/forums.htm > > >> >> > > >> > > ---------------------------------------------------------------------- > > >> > For information about J forums see > > http://www.jsoftware.com/forums.htm > > >> ---------------------------------------------------------------------- > > >> For information about J forums see > http://www.jsoftware.com/forums.htm > > >> > > > > > > > > > > > > -- > > > Devon McCormick, CFA > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm -- Devon McCormick, CFA ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
