Hi Herman, and all -- The issue determining the length of the in-sample period is not the number of trades, but the number of data points it takes for the model (afl) and the data (OHLC) to become synchronized. Imagine a model that is looking for cyclic behavior in the data. If the cycle completes in one month it takes many fewer data points to sync than if the cycle completes in one year.
As for out-of-sample, the statistics can be run on any number of data points. The data points can be daily equity changes or closed trades -- whatever time granularity the trader will be making position adjustments. Providing the relationship between the mean of the metric being studied, its standard deviation, and the number of data points is reasonable, control charts can used to indicate when the system is operating well and when it is beginning to fail. Assume the metric is expectancy. If the mean expectancy is positive and the ratio of the mean to the standard deviation is high (high would be a ratio of 0.5, extremely high would be 1.0, normal would be 0.2), it takes fewer than 20 trades to have very high confidence that the system is performing well. Since systems go out of sync -- either periodically or forever -- once there are enough OOS data points to give the confidence that the developer requires, there is no benefit in waiting for further confirmation. Trade now, while the system is in sync. Thanks for listening, Howard On Fri, Feb 19, 2010 at 6:51 AM, Herman vandenBergen <[email protected]>wrote: > > > Hello Howard, > > I cannot help but wonder why the OOS periods in optimizations aren't based > on the number of trades, instead of some arbitrary time period. Could it be > that this is just another example of how TA methodology is based on > convenience, i.e., a reluctance to redesign tools and/or a reluctance to > change :-) > > I am no statistician, but I would guess that if one wanted some specific > OOS performance the first thing to do would be to figure out how many trades > would be needed to do the calculations. If 100 trades gives you the > confidence why would you run a OOS period that gives you 200 trades? > > Best regards, > herman > > > Friday, February 19, 2010, 8:18:45 AM, you wrote: > > > > > > Hi SpaceBass -- > > The only way to determine the correct length for the in-sample period is by > running experiments. The length needs to be long enough for the model to > synchronize with the data and learn to recognize the signal. But not so > long that the signal has changed significantly, making it hard to identify. > And not so short that there is not enough signal to learn, resulting in a > system that has synced to the noise. > > In a few words -- the length of the in-sample period should be as short as > is practical and effective. > > Once the length of the in-sample period has been determined, the length of > the out-of-sample period is easy. It is the length of time that the system > remains profitable. > > There is no general relationship between the anything and the length of the > in-sample period. There is no relationship between the length of the > in-sample period and the length of the out-of-sample period. > > There is some controversy in the modeling and simulation field about > whether the in-sample length is a legitimate variable in an optimization. > While it can be dangerous to system validity to run an optimization on > in-sample length, some trials using different lengths are necessary. > > Thanks for listening, > Howard > > > On Thu, Feb 18, 2010 at 7:23 PM, spacebass5000 <[email protected] > > wrote: > > I was wondering if there was a way to optimize the In-Sample and Step time > periods within AB. If not, can someone point me in the direction of a good > resource on this topic? > > > > > >
