Hi Howard, Yes I'm trying to validate my trading system by rejecting Ho: expectancy <= 0 based on the t-statistic. I also want to use the mean and standard deviation to setup my control chart so I really want to make sure I'm using the correct inputs. Based on your input I will proceed as follows in calculating the t-stat: 1) simply average the periodic expectancy values for only OOS periods 2) likewise calculate the stdev of these periodic values 3) N = number of OOS periods = number of OOS observations of expectancy.
Thanks I appreciate your help. Ray --- In [email protected], Howard B <howardba...@...> wrote: > > Hi Ray -- > > The t-test is used to test whether two samples come from the same > distribution. The example you may be referring to asks whether the > expectancy of the out-of-sample results could have come from a random or > unprofitable system. The null hypothesis (the one we want to say is very > unlikely) is "the OOS results came from a random system. Calculate the mean > and standard deviation of the metric you are using -- expectancy, in this > case -- for the OOS results. Calculate the mean and standard deviation for > the other, random, distribution. The random mean of expectancy is 0.0. In > this case, you can assume that the random standard deviation is the same as > the OOS standard deviation. Use the count of data points from the OOS > results, and assume the random results have the same count. Compute the > t-statistic. Look in the precalculated table to see how likely it is that > the OOS results came from the same distribution as the random results. > > You could make a similar test to see whether the IS and OOS results came > from the same distribution. That test would require using all of the > trades, IS and OOS, not just the OOS ones. It might be interesting, but > that is not the test you probably want. > > I hope this helps, > Thanks, > Howard > > On Sat, Jul 17, 2010 at 7:09 PM, raymondpconnolly < > raymondpconno...@...> wrote: > > > > > > > Hi All, > > > > I'm looking for input on the calculation of the t-statistic of expectancy > > for the OOS periods of the walkforward simulation for the purposes of > > validation and control as put forward by Dr. Howard Bandy in his ATAA > > presention of October 2009. Many thanks to Howard for making available this > > excellent information. > > > > At first glance I thought this was a pretty straightforward matter ... > > until I got down to the nuts and bolts of the calculation. So let me explain > > what I've done and the questions I'm having: > > > > Assumptions: > > -t-statistic of expectancy (for the null hypothesis Ho: expectancy <= 0 ) = > > ((mean)/(standard deviation))*sqrt(N-1) (1) > > - use data for each OOS period generated by Amibroker (no CBT generated > > data) > > > > Calculation Possibilities: > > 1.0 calculate the mean and standard deviation using the expectancy values > > for each OOS period > > 2.0 calculate the weighted mean and weighted standard deviation of > > expectancy where the weight for each OOS period is > > wt = (number of trades for the OOS period / total trades for all OOS > > periods) and where the weighted standard deviation > > is defined here: > > http://www.itl.nist.gov/div898/software/dataplot/refman2/ch2/weightsd.pdf > > 3.0 N = number of OOS periods > > 4.0 N = total trades for all OOS periods > > > > Observations and Questions: > > A) What is the correct combination of inputs from the possibilities above > > or from possibilities I have failed to recognize that yields the valid > > t-statistic ? > > B) comparing 1.0 and 2.0 above yeilds slightly different means in my case ( > > where the standard deviation in the average number of trades per OOS period > > is small). 2.0 above also yeilds the same result as calculating expectancy > > by the alternative method: > > Expectancy ($) = (TotalProfit - TotalLoss) / NumberOfTrades = NetProfit / > > NumberOfTrades over all OOS periods so my suspicion is that 2.0 above is the > > way to go > > C) with respect to 3.0 and 4.0 above I'm at a loss. I can see using the > > number of OOS periods as there are that many observations of expectancy > > however I can also see using total trades for all OOS periods since the > > results of all trades ultimately generate the expetancies over all OOS > > periods. > > > > Thanks for your input. > > > > Ray > > > > > > >
