Hi Ray -- The t-test is used to test whether two samples come from the same distribution. The example you may be referring to asks whether the expectancy of the out-of-sample results could have come from a random or unprofitable system. The null hypothesis (the one we want to say is very unlikely) is "the OOS results came from a random system. Calculate the mean and standard deviation of the metric you are using -- expectancy, in this case -- for the OOS results. Calculate the mean and standard deviation for the other, random, distribution. The random mean of expectancy is 0.0. In this case, you can assume that the random standard deviation is the same as the OOS standard deviation. Use the count of data points from the OOS results, and assume the random results have the same count. Compute the t-statistic. Look in the precalculated table to see how likely it is that the OOS results came from the same distribution as the random results.
You could make a similar test to see whether the IS and OOS results came from the same distribution. That test would require using all of the trades, IS and OOS, not just the OOS ones. It might be interesting, but that is not the test you probably want. I hope this helps, Thanks, Howard On Sat, Jul 17, 2010 at 7:09 PM, raymondpconnolly < [email protected]> wrote: > > > Hi All, > > I'm looking for input on the calculation of the t-statistic of expectancy > for the OOS periods of the walkforward simulation for the purposes of > validation and control as put forward by Dr. Howard Bandy in his ATAA > presention of October 2009. Many thanks to Howard for making available this > excellent information. > > At first glance I thought this was a pretty straightforward matter ... > until I got down to the nuts and bolts of the calculation. So let me explain > what I've done and the questions I'm having: > > Assumptions: > -t-statistic of expectancy (for the null hypothesis Ho: expectancy <= 0 ) = > ((mean)/(standard deviation))*sqrt(N-1) (1) > - use data for each OOS period generated by Amibroker (no CBT generated > data) > > Calculation Possibilities: > 1.0 calculate the mean and standard deviation using the expectancy values > for each OOS period > 2.0 calculate the weighted mean and weighted standard deviation of > expectancy where the weight for each OOS period is > wt = (number of trades for the OOS period / total trades for all OOS > periods) and where the weighted standard deviation > is defined here: > http://www.itl.nist.gov/div898/software/dataplot/refman2/ch2/weightsd.pdf > 3.0 N = number of OOS periods > 4.0 N = total trades for all OOS periods > > Observations and Questions: > A) What is the correct combination of inputs from the possibilities above > or from possibilities I have failed to recognize that yields the valid > t-statistic ? > B) comparing 1.0 and 2.0 above yeilds slightly different means in my case ( > where the standard deviation in the average number of trades per OOS period > is small). 2.0 above also yeilds the same result as calculating expectancy > by the alternative method: > Expectancy ($) = (TotalProfit - TotalLoss) / NumberOfTrades = NetProfit / > NumberOfTrades over all OOS periods so my suspicion is that 2.0 above is the > way to go > C) with respect to 3.0 and 4.0 above I'm at a loss. I can see using the > number of OOS periods as there are that many observations of expectancy > however I can also see using total trades for all OOS periods since the > results of all trades ultimately generate the expetancies over all OOS > periods. > > Thanks for your input. > > Ray > > >
