Exactly. But product managers are not applied psychologists. Especially when they're paying for the subjects, and the experimenter says "sorry -- the smallness of the effect means we need more subjects."
And applied psychologists never work to 5 sigma. 2 is ok. 3 is stellar. And mathematicians never like to fit their fav math to the experimental conditions. They expect it to be done the other way round, like it always has. Small wonder that the best human-factored consumer products on the market today were designed by instinct, by a power freak with a vicious temper and a bad smell. On Thu, Jul 5, 2012 at 8:30 AM, Bo Jacoby <bojac...@yahoo.dk> wrote: > "how many more subjects do we need to test before we can be confident > performance has actually got better?" > > > That depends on the signal-to-noise ratio and on the desired confidence > level. A big improvement is detected sooner than a small improvement which > drown in noise. If you need five sigma confidence you must test Nsubject > computed somewhat like this, I think. > > 'A B C'=:ABC X > > NB.RMS=:Root-Mean-Square > RMS=:(+/%#-2:)&.:*: > > noise=:RMS X-A+B*C<i.#X > ]N=:*:5*noise%B > > > - Bo > > >>________________________________ >> Fra: Ian Clark <earthspo...@gmail.com> >>Til: Programming forum <programming@jsoftware.com> >>Sendt: 12:41 onsdag den 4. juli 2012 >>Emne: Re: [Jprogramming] Kalman filter in J? >> >>That's right. I'm stress-testing my various methods by seeing how soon >>they detect a step. >> >>A perennial cry from the product managers was "how many more subjects >>do we need to test before we can be confident performance has actually >>got better?" (ie there's been a step-up in expected performance). >> >>Ian >> >>On Wed, Jul 4, 2012 at 8:32 AM, Bo Jacoby <bojac...@yahoo.dk> wrote: >>> That explains it! With N=100, A=90, B=20, C=89 you have 90 samples to the >>> left to estimate A~89 but only 10 samples to the right to estimate B~37. >>> This is reasonable. >>> >>> - Bo >>> >>> Til: Programming forum <programming@jsoftware.com> >>>>Sendt: 23:05 tirsdag den 3. juli 2012 >>>>Emne: Re: [Jprogramming] Kalman filter in J? >>>> >>>>@Bo The discrepancy comes about because I altered your original verb: >>>>C for compatibility with my own definition of the Heaviside function: >>>>hv ... >>>> >>>> 10 hv 3 >>>>0 0 0 1 1 1 1 1 1 1 >>>> >>>>I did this simply by increasing its returned value by 1. But I did not >>>>alter the old version of ABC (ABC0) which uses it. (It does need >>>>altering). >>>> >>>>Upon restoring to your original C, I can confirm that ABC0 now returns >>>>the exact correct parameters in the absence of noise (it didn't >>>>before) ... >>>> >>>> ABC0 10 hv 3 >>>>0 1 2 >>>> >>>>and that both versions now give identical results on the few trials >>>>I've made of them, especially with the random variable X which I was >>>>using for the test... >>>> >>>> ABC0 X >>>>98.6542 36.7969 89 >>>> ABC X >>>>98.6542 36.7969 89 >>>> >>>>This version of X (which I've saved and will make available) is a hard >>>>test: when I plot it I can't see the step without knowing where it's >>>>meant to be. The standard deviation of the Gaussian noise is half the >>>>step height, 10. Thus the noise is capable of obliterating the >>>>information as to the exact onset of the step, so I'm impressed that >>>>ABC gets it right here, even if the estimate of B is awry in this >>>>instance. (In the absence of noise it would be 100 20 89.) >>>> >>>>With other X's I've tried, ABC makes far better estimates of B. >>>> >>>>Ian >>>> >>>>On Tue, Jul 3, 2012 at 5:17 PM, Bo Jacoby <bojac...@yahoo.dk> wrote: >>>>> The old ABC program and the new ABC program were supposed to be >>>>> equivalent algorithms, and so they should give exactly the same results >>>>> for the same test data. That is why I am bewildered by the difference. >>>>> Ian did the test and knows about the test data. >>>>> - Bo >>>>>>________________________________ >>>>>> Fra: Raul Miller <rauldmil...@gmail.com> >>>>>>Til: Programming forum <programming@jsoftware.com> >>>>>>Sendt: 16:34 tirsdag den 3. juli 2012 >>>>>>Emne: Re: [Jprogramming] Kalman filter in J? >>>>>> >>>>>>On Tue, Jul 3, 2012 at 6:08 AM, Bo Jacoby <bojac...@yahoo.dk> wrote: >>>>>>> The result changes from C=90 (correct) in the old ABC to C=89 >>>>>>> (incorrect) in the new ABC. Also the old ABC result B=41 and the >>>>>>> new ABC result B=37 should both be B=20. That is bewildering. >>>>>> >>>>>>Which test data are you using here (for X)? >>>>>> >>>>>>(Have you verified that the noise does not shift the values of the >>>>>>"best result"?) >>>>>> >>>>>>-- >>>>>>Raul >>>>>>---------------------------------------------------------------------- >>>>>>For information about J forums see http://www.jsoftware.com/forums.htm >>>>>> >>>>>> >>>>>> >>>>> ---------------------------------------------------------------------- >>>>> For information about J forums see http://www.jsoftware.com/forums.htm >>>>---------------------------------------------------------------------- >>>>For information about J forums see http://www.jsoftware.com/forums.htm >>>> >>>> >>>>Fra: Ian Clark <earthspo...@gmail.com> >>> ---------------------------------------------------------------------- >>> For information about J forums see http://www.jsoftware.com/forums.htm >>---------------------------------------------------------------------- >>For information about J forums see http://www.jsoftware.com/forums.htm >> >> >> > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
