Excellent questions Mike. That's exactly what I was wondering about. Louis
2008/4/24, Mike <[EMAIL PROTECTED]>: > > > To make sure I have been clear on this ---- > > It does not matter At All how many trades or what length of time the > > in-sample period covers. Results from the in-sample runs have no > value in > > estimating the future performance. > > Howard, > > In an earlier post you stated that the number of IS observations will > impact the error bands of the backtested statistics. > > Given that these same statistics are then used in the calculation of > the objective function, which in turn will dictate the parameter > values to use in the next OOS period. Wouldn't it be a logical > extension that an IS period should have sufficient observations to > allow the error bands to stabilize? Not for "estimating the future > performance". But rather for estimating the best parameter values. > > Or, are you satisfied that performant OOS walk forward periods is all > that counts? How many OOS periods do you like to have before making > your final judgement? > > As always, thanks for sharing. > > Mike > > --- In [email protected] <amibroker%40yahoogroups.com>, "Howard B" > <[EMAIL PROTECTED]> wrote: > > > > Hi Louis, and all -- > > > > Select the period of time for the in-sample period that works for > the system > > you are using. > > Select the period of time for the out-of-sample period and > reoptimization > > period that is sufficient for the system and the market to stay in > sync and > > to give you several walk forward steps. > > Perform the walk forward analysis. > > Look at the out-of-sample results from the combined walk forward > steps. > > Decide from there whether to trade or go back to the drawing board. > > > > To make sure I have been clear on this ---- > > It does not matter At All how many trades or what length of time the > > in-sample period covers. Results from the in-sample runs have no > value in > > estimating the future performance. > > > > Thanks for listening, > > Howard > > > > > > On Tue, Apr 22, 2008 at 8:22 PM, Louis Préfontaine <[EMAIL PROTECTED]> > > wrote: > > > > > Hi Howard, > > > > > > What would you consider to be a sufficiently large sample for IS > and then > > > for OOS? If I develop a system that makes 250 trades a year, > then if I > > > select IS-OOS of 2-3 weeks then it's no more than 10-15 trades. > Is this > > > enough? > > > > > > Regards, > > > > > > Louis > > > > > > 2008/4/22, Howard B <[EMAIL PROTECTED]>: > > > > > > > > > Hi Simon -- > > > > > > > > From your description, the system was developed on a set of > data, but > > > > not tested on any data that was not used during development. > The data used > > > > during development is called the in-sample data. Data used for > testing that > > > > was not used during development is called the out-of-sample > data. > > > > > > > > The in-sample results always look good -- we do not stop > playing with > > > > the system until they look good. The in-sample results have no > value in > > > > estimating the future out-of-sample results. In order to > estimate what the > > > > likely profitability will be when traded with real money, out- > of-sample > > > > testing is necessary. > > > > > > > > I have documented systems that have over 1,300,000 closed > trades and > > > > reasonable looking results for the in-sample period, but were > not profitable > > > > out-of-sample. > > > > > > > > There is no substitute for out-of-sample testing. > > > > > > > > Thanks for listening, > > > > Howard > > > > www.quantitativetradingsystems.com > > > > > > > > > > > > On Thu, Apr 17, 2008 at 2:29 AM, si00si00 <[EMAIL PROTECTED]> wrote: > > > > > > > > > Hi all, > > > > > > > > > > I have a friend who has developed a trading system. It is an > intraday > > > > > system that makes on average around 5 futures trades per day. > We were > > > > > discussing it the other day and a point of disagreement arose > between > > > > > us. He claims that there is no necessity for him to test the > strategy > > > > > on out of sample data because he has back tested it using > over 8 years > > > > > of historical intraday data, and the patterns the strategy > predicts > > > > > occur 70% of the time or more. > > > > > > > > > > My question is, does anyone know if the data-mining bias can > be > > > > > considered irrelvant when the sample size is so large? (in > this case, > > > > > the sample size is roughly 8400 trades). Put another way, > with so many > > > > > observations, how many different rules would have to be back > tested in > > > > > order for data-mining bias to creep in? > > > > > > > > > > Thanks in advance for any thoughts you might have! > > > > > > > > > > Simon > > > > > > > > > > > > > > > > > > > > > > > > >
