We should take this off-group, but I'm replying in public because if I'm
wrong I would like to be corrected (and I'm only an amateur statistician):
I think you are calling binomialprob correctly but I have some
objections to your use of the result.
1. I think your rejectH0 should use 1 - -: CONFIDENCE instead of
1-CONFIDENCE.
The question is, "How likely is a result as weird as I am seeing,
assuming H0?" You should not bias "weird" by assuming that weird
results will be correct guesses - they could just as likely be incorrect
guesses. To ensure that you reject 95% of the purely-chance deviations
of a certain size, that 95% should be centered around the mean, not
loaded toward one side.
[are there really people who think optical might be better than USB??
This is digital communication, no? 44K samples/sec, 2 channels, 20
bits/sample, needs 2Mb/sec max out of 480Mb/sec rated USB speed... how
could that not be enough?
It was ever thus... when I last looked at this sort of thing, 20 years
back, the debate was whether big fat expensive cables would make a
difference. Bob Pease, a respected analog engineer, pointed out that it
was impossible, and James Randi had a bet that no one could discern
$7000 cables from ordinary speaker wire, but still the non-EEs have
their superstitions...]
2. Why 95%? I would fear that someone who is thinking about optical
cable would rest uneasy with a 5-10% chance that they have not spent
enough on quality audio. Why not simply report, "A monkey with a coin
to toss would do as well as you y% of the time. Most researchers accept
results as significant only if the monkey would do as well less than 5%
of the time. Take more samples if you want less uncertainty."
Henry Rich
On 2/24/2015 9:46 PM, Ian Clark wrote:
Addon 'stats/base/distribution' defines the verb: binomialprob.
Am I using it correctly?
Please cast a beady eye over my train of thought, as I've set it out below...
I've written an app in J to administer a double-blind test which
reruns the classic experiment described by David Salsburg in "The Lady
Tasting Tea". But in place of pre- and post-lactary tea, I play a
snatch of one of two soundfiles in a series of 10 trials to ascertain
if she really can tell the difference.
From her answers I compute 2 numbers:
N=: number of trials (typically 10 or 20)
s=: number of successes.
I have also set up an adjustable parameter:
CONFIDENCE=: 0.95 NB. (the 95% confidence limit)
Instead of using binomialprob directly, I define 2 verbs:
pH0=: 4 : 'binomialprob 0.5,x,y'
rejectH0=: 4 : '(1-CONFIDENCE) > x pH0 y'
(p=. N pH0 s) is the likelihood of s arising at random under the
"null hypothesis" (H0), viz that she's just guessing with
probability=0.5 of success.
(N rejectH0 s) returns 1 iff p is too low, as determined by
CONFIDENCE, implying the null hypothesis (H0) can safely be rejected.
This Boolean value triggers one of 2 messages:
1 --> You can tell the difference.
0 --> You're just guessing.
That's straining the epistemology, I know. But I don't expect a
non-statistician to make much sense of a statistically kosher message,
such as:
0 --> This program has decided that the (null) hypothesis that your
results have arisen by pure guesswork cannot be safely rejected on the
evidence alone of these 10 trials.
There's gratifying interest in the music/audiophile community in such
a sound-test, if it's packaged up and made easy-to-use. Questions
like: "can you hear any improvement if you use an optical cable
instead of a USB one?" come up all the time. And, as I've discovered
for myself, even a strong impression of improvement may not stand up
to this sort of scrutiny. It's the placebo effect.
So not only do I need to assure the soundness of the statistical
theory and its J implementation, plus my use of it, but also publish a
proper write-up (in Jwiki) which makes sense to a non-statistician.
This after all is *the* foundation experiment in the history of
Statistics.
But AFAICS its treatment in Wikipedia leaves much to be desired.
See for instance: https://en.wikipedia.org/wiki/Binomial_test
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
