Hi Howard, What would you consider to be a sufficiently large sample for IS and then for OOS? If I develop a system that makes 250 trades a year, then if I select IS-OOS of 2-3 weeks then it's no more than 10-15 trades. Is this enough?
Regards, Louis 2008/4/22, Howard B <[EMAIL PROTECTED]>: > > Hi Simon -- > > From your description, the system was developed on a set of data, but not > tested on any data that was not used during development. The data used > during development is called the in-sample data. Data used for testing that > was not used during development is called the out-of-sample data. > > The in-sample results always look good -- we do not stop playing with the > system until they look good. The in-sample results have no value in > estimating the future out-of-sample results. In order to estimate what the > likely profitability will be when traded with real money, out-of-sample > testing is necessary. > > I have documented systems that have over 1,300,000 closed trades and > reasonable looking results for the in-sample period, but were not profitable > out-of-sample. > > There is no substitute for out-of-sample testing. > > Thanks for listening, > Howard > www.quantitativetradingsystems.com > > > On Thu, Apr 17, 2008 at 2:29 AM, si00si00 <[EMAIL PROTECTED]> wrote: > > > Hi all, > > > > I have a friend who has developed a trading system. It is an intraday > > system that makes on average around 5 futures trades per day. We were > > discussing it the other day and a point of disagreement arose between > > us. He claims that there is no necessity for him to test the strategy > > on out of sample data because he has back tested it using over 8 years > > of historical intraday data, and the patterns the strategy predicts > > occur 70% of the time or more. > > > > My question is, does anyone know if the data-mining bias can be > > considered irrelvant when the sample size is so large? (in this case, > > the sample size is roughly 8400 trades). Put another way, with so many > > observations, how many different rules would have to be back tested in > > order for data-mining bias to creep in? > > > > Thanks in advance for any thoughts you might have! > > > > Simon > > > > > >
