Hi Louis -- The walk forward process solves those problems.
Thanks, Howard On Wed, Apr 23, 2008 at 6:47 AM, Louis Préfontaine <[EMAIL PROTECTED]> wrote: > Hi Howard, > > The problem is this: the market is ever changing, as you say in your > book. Let's say my system reacts a lot to what is doing the market as a > whole, then I sure would need a shorter time-frame! 1 month would probably > be too much, if we look at what happened in the first 2 weeks of > January... You say it does not matter how many trades; but how to judge > the value of an OOS result with only 10-15 trades? This could be luck! > > I agree that it could be tested and re-tested, but testing until I get a > correct time-frame seems to me like another original way of doing > curve-fitting, don't you think? > > That's the whole problm I see with walk-forward; it is good to know if the > system is ready but it does not help that much to make the system better, > because I only get the best result of each optimization and with limited > number of trades the absolute best can always be best because of luck... > > Louis > > > > 2008/4/23, Howard B <[EMAIL PROTECTED]>: > > > Hi Louis, and all -- > > > > Select the period of time for the in-sample period that works for the > > system you are using. > > Select the period of time for the out-of-sample period and > > reoptimization period that is sufficient for the system and the market to > > stay in sync and to give you several walk forward steps. > > Perform the walk forward analysis. > > Look at the out-of-sample results from the combined walk forward steps. > > Decide from there whether to trade or go back to the drawing board. > > > > To make sure I have been clear on this ---- > > It does not matter At All how many trades or what length of time the > > in-sample period covers. Results from the in-sample runs have no value in > > estimating the future performance. > > > > Thanks for listening, > > Howard > > > > > > > > On Tue, Apr 22, 2008 at 8:22 PM, Louis Préfontaine <[EMAIL PROTECTED]> > > wrote: > > > > > Hi Howard, > > > > > > What would you consider to be a sufficiently large sample for IS and > > > then for OOS? If I develop a system that makes 250 trades a year, then > > > if I > > > select IS-OOS of 2-3 weeks then it's no more than 10-15 trades. Is this > > > enough? > > > > > > Regards, > > > > > > Louis > > > > > > 2008/4/22, Howard B <[EMAIL PROTECTED]>: > > > > > > > > Hi Simon -- > > > > > > > > From your description, the system was developed on a set of data, > > > > but not tested on any data that was not used during development. The > > > > data > > > > used during development is called the in-sample data. Data used for > > > > testing > > > > that was not used during development is called the out-of-sample data. > > > > > > > > The in-sample results always look good -- we do not stop playing > > > > with the system until they look good. The in-sample results have no > > > > value > > > > in estimating the future out-of-sample results. In order to estimate > > > > what > > > > the likely profitability will be when traded with real money, > > > > out-of-sample > > > > testing is necessary. > > > > > > > > I have documented systems that have over 1,300,000 closed trades and > > > > reasonable looking results for the in-sample period, but were not > > > > profitable > > > > out-of-sample. > > > > > > > > There is no substitute for out-of-sample testing. > > > > > > > > Thanks for listening, > > > > Howard > > > > www.quantitativetradingsystems.com > > > > > > > > > > > > On Thu, Apr 17, 2008 at 2:29 AM, si00si00 <[EMAIL PROTECTED]> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I have a friend who has developed a trading system. It is an > > > > > intraday > > > > > system that makes on average around 5 futures trades per day. We > > > > > were > > > > > discussing it the other day and a point of disagreement arose > > > > > between > > > > > us. He claims that there is no necessity for him to test the > > > > > strategy > > > > > on out of sample data because he has back tested it using over 8 > > > > > years > > > > > of historical intraday data, and the patterns the strategy > > > > > predicts > > > > > occur 70% of the time or more. > > > > > > > > > > My question is, does anyone know if the data-mining bias can be > > > > > considered irrelvant when the sample size is so large? (in this > > > > > case, > > > > > the sample size is roughly 8400 trades). Put another way, with so > > > > > many > > > > > observations, how many different rules would have to be back > > > > > tested in > > > > > order for data-mining bias to creep in? > > > > > > > > > > Thanks in advance for any thoughts you might have! > > > > > > > > > > Simon > > > > > > > > > > > > > > > > > > > > >
