"how many more subjects do we need to test before we can be confident performance has actually got better?"
That depends on the signal-to-noise ratio and on the desired confidence level. A big improvement is detected sooner than a small improvement which drown in noise. If you need five sigma confidence you must test Nsubject computed somewhat like this, I think. 'A B C'=:ABC X NB.RMS=:Root-Mean-Square RMS=:(+/%#-2:)&.:*: noise=:RMS X-A+B*C<i.#X ]N=:*:5*noise%B - Bo >________________________________ > Fra: Ian Clark <earthspo...@gmail.com> >Til: Programming forum <programming@jsoftware.com> >Sendt: 12:41 onsdag den 4. juli 2012 >Emne: Re: [Jprogramming] Kalman filter in J? > >That's right. I'm stress-testing my various methods by seeing how soon >they detect a step. > >A perennial cry from the product managers was "how many more subjects >do we need to test before we can be confident performance has actually >got better?" (ie there's been a step-up in expected performance). > >Ian > >On Wed, Jul 4, 2012 at 8:32 AM, Bo Jacoby <bojac...@yahoo.dk> wrote: >> That explains it! With N=100, A=90, B=20, C=89 you have 90 samples to the >> left to estimate A~89 but only 10 samples to the right to estimate B~37. >> This is reasonable. >> >> - Bo >> >> Til: Programming forum <programming@jsoftware.com> >>>Sendt: 23:05 tirsdag den 3. juli 2012 >>>Emne: Re: [Jprogramming] Kalman filter in J? >>> >>>@Bo The discrepancy comes about because I altered your original verb: >>>C for compatibility with my own definition of the Heaviside function: >>>hv ... >>> >>> 10 hv 3 >>>0 0 0 1 1 1 1 1 1 1 >>> >>>I did this simply by increasing its returned value by 1. But I did not >>>alter the old version of ABC (ABC0) which uses it. (It does need >>>altering). >>> >>>Upon restoring to your original C, I can confirm that ABC0 now returns >>>the exact correct parameters in the absence of noise (it didn't >>>before) ... >>> >>> ABC0 10 hv 3 >>>0 1 2 >>> >>>and that both versions now give identical results on the few trials >>>I've made of them, especially with the random variable X which I was >>>using for the test... >>> >>> ABC0 X >>>98.6542 36.7969 89 >>> ABC X >>>98.6542 36.7969 89 >>> >>>This version of X (which I've saved and will make available) is a hard >>>test: when I plot it I can't see the step without knowing where it's >>>meant to be. The standard deviation of the Gaussian noise is half the >>>step height, 10. Thus the noise is capable of obliterating the >>>information as to the exact onset of the step, so I'm impressed that >>>ABC gets it right here, even if the estimate of B is awry in this >>>instance. (In the absence of noise it would be 100 20 89.) >>> >>>With other X's I've tried, ABC makes far better estimates of B. >>> >>>Ian >>> >>>On Tue, Jul 3, 2012 at 5:17 PM, Bo Jacoby <bojac...@yahoo.dk> wrote: >>>> The old ABC program and the new ABC program were supposed to be equivalent >>>> algorithms, and so they should give exactly the same results for the same >>>> test data. That is why I am bewildered by the difference. Ian did the test >>>> and knows about the test data. >>>> - Bo >>>>>________________________________ >>>>> Fra: Raul Miller <rauldmil...@gmail.com> >>>>>Til: Programming forum <programming@jsoftware.com> >>>>>Sendt: 16:34 tirsdag den 3. juli 2012 >>>>>Emne: Re: [Jprogramming] Kalman filter in J? >>>>> >>>>>On Tue, Jul 3, 2012 at 6:08 AM, Bo Jacoby <bojac...@yahoo.dk> wrote: >>>>>> The result changes from C=90 (correct) in the old ABC to C=89 >>>>>> (incorrect) in the new ABC. Also the old ABC result B=41 and the >>>>>> new ABC result B=37 should both be B=20. That is bewildering. >>>>> >>>>>Which test data are you using here (for X)? >>>>> >>>>>(Have you verified that the noise does not shift the values of the >>>>>"best result"?) >>>>> >>>>>-- >>>>>Raul >>>>>---------------------------------------------------------------------- >>>>>For information about J forums see http://www.jsoftware.com/forums.htm >>>>> >>>>> >>>>> >>>> ---------------------------------------------------------------------- >>>> For information about J forums see http://www.jsoftware.com/forums.htm >>>---------------------------------------------------------------------- >>>For information about J forums see http://www.jsoftware.com/forums.htm >>> >>> >>>Fra: Ian Clark <earthspo...@gmail.com> >> ---------------------------------------------------------------------- >> For information about J forums see http://www.jsoftware.com/forums.htm >---------------------------------------------------------------------- >For information about J forums see http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
