Hi Louis, and all -- Select the period of time for the in-sample period that works for the system you are using. Select the period of time for the out-of-sample period and reoptimization period that is sufficient for the system and the market to stay in sync and to give you several walk forward steps. Perform the walk forward analysis. Look at the out-of-sample results from the combined walk forward steps. Decide from there whether to trade or go back to the drawing board.
To make sure I have been clear on this ---- It does not matter At All how many trades or what length of time the in-sample period covers. Results from the in-sample runs have no value in estimating the future performance. Thanks for listening, Howard On Tue, Apr 22, 2008 at 8:22 PM, Louis Préfontaine <[EMAIL PROTECTED]> wrote: > Hi Howard, > > What would you consider to be a sufficiently large sample for IS and then > for OOS? If I develop a system that makes 250 trades a year, then if I > select IS-OOS of 2-3 weeks then it's no more than 10-15 trades. Is this > enough? > > Regards, > > Louis > > 2008/4/22, Howard B <[EMAIL PROTECTED]>: > > > > Hi Simon -- > > > > From your description, the system was developed on a set of data, but > > not tested on any data that was not used during development. The data used > > during development is called the in-sample data. Data used for testing that > > was not used during development is called the out-of-sample data. > > > > The in-sample results always look good -- we do not stop playing with > > the system until they look good. The in-sample results have no value in > > estimating the future out-of-sample results. In order to estimate what the > > likely profitability will be when traded with real money, out-of-sample > > testing is necessary. > > > > I have documented systems that have over 1,300,000 closed trades and > > reasonable looking results for the in-sample period, but were not profitable > > out-of-sample. > > > > There is no substitute for out-of-sample testing. > > > > Thanks for listening, > > Howard > > www.quantitativetradingsystems.com > > > > > > On Thu, Apr 17, 2008 at 2:29 AM, si00si00 <[EMAIL PROTECTED]> wrote: > > > > > Hi all, > > > > > > I have a friend who has developed a trading system. It is an intraday > > > system that makes on average around 5 futures trades per day. We were > > > discussing it the other day and a point of disagreement arose between > > > us. He claims that there is no necessity for him to test the strategy > > > on out of sample data because he has back tested it using over 8 years > > > of historical intraday data, and the patterns the strategy predicts > > > occur 70% of the time or more. > > > > > > My question is, does anyone know if the data-mining bias can be > > > considered irrelvant when the sample size is so large? (in this case, > > > the sample size is roughly 8400 trades). Put another way, with so many > > > observations, how many different rules would have to be back tested in > > > order for data-mining bias to creep in? > > > > > > Thanks in advance for any thoughts you might have! > > > > > > Simon > > > > > > > > > >
