<snip> That is similar to Pardo's WFE (Walk forward efficiency), or a measure of how much curve fitting inflated test results. Pardo suggests taking the concatenated out of sample returns and divide by the result treating the entire combined data set as in sample. Anything below 0.65 will probably not trade well live. The higher, the better.<snip>
There is where the subject becomes very interesting ... it is hard to find an absolute answer i.e. find a firm footing that doesn't slip out from under us e.g. - is 0.65 an empirical number ... where did Pardo get it from ... experience or by calculation (based on observations)? - this is one place where Howard and I disagree .. Howard is emphatic that the metrics from the IS are worthless whereas I think that if the OOS encourages us to continue then the combined IS and OOS provides us with useful stats ... I expect Howard reiterates that point, without compromise, as a teaching device. - examining the principles ... using W/L as the example ... say our IS test returns W/L 65/35 and our PayOffRatio is 1/1 ... this translates into equity curves etc eventually i.e. once we introduce MM. We use the OOS metric to decide if the IS development process was successful but how exactly do we do that? Presumably if the IS metric was good and the OOS is the same +- X% we are happy i.e. if the OOS W/L == 65/35 and the PR == 1/1 i.e. matches the IS, then we can realistically expect to achieve the same eq curves, providing we trade over a stationary market (stationary == the same mean and dispersion as the market used for our IS dataset?). How much X% should we tolerate and does it vary from metric to metric? Comparing it to our benchmark, the null hypothesis (a fair coin toss) ... from my labtests, variance == sample error. So the ratio of the IS metric/OOS metric depends on the size of the sample: IS W/L +- sample error/OOS W/L +- sample error == ratio; The concatenated IS and OOS creates a larger sample and converges on the mean, as N approaches infinity (so this is a valid method afterall?). The IS is always smaller than the combined IS and OOS and sometimes very small (according to user preference). Since W/L always == 1, for a fair coin toss, the ratio of IS/(IS + OOS) metric will always equal the sample error difference. +- sample error *1 == 1 StDev frequency +- sample error *2 == 2 StDev frequency etc So, 100N IS/ infinite N(IS + OOS) == 1 +- 10%/1 +- 0% == 0.9/1;// 66% of the time 0.8;// with 2 StDev significance 0.7;// for 3 StDev significance. etc For biased systems (65/35 W/L) it works in a similar way (give or take a bit)..... it looks like the 0.65 number is an approximation of significant variance. The really interesting part is that I can't (mathematically) tell the difference between market variance and non-stationarity. --- In [email protected], "dloyer123" <dloyer...@...> wrote: > > --- In [email protected], Rajiv Arya <rajivarya87@> wrote: > > > > > > I like to compute a ratio of the out-sample metric and divide it by the > > in-sample metric. > > > > And I like to look for multiple runs of out-sample/in-sample ratio to be > > above 0.5 and with little fluctuation. > > > > That is similar to Pardo's WFE (Walk forward efficiency), or a measure of how > much curve fitting inflated test results. Pardo suggests taking the > concatenated out of sample returns and divide by the result treating the > entire combined data set as in sample. Anything below 0.65 will probably not > trade well live. The higher, the better. >
