Hi Thomas, Do you use Walk-forward as a random optimizer like Monte Carlo Simulation or do you use it as a parameter optimization (let's say you want to know the best numbers for a MA crossover). Or maybe both?
I ask this because my feeling is that if I use it only as a parameter optimization, each parameter would be tested only one time in each IS or OOS, hence this could not be significative. I tried to add a random simulation 5 times to get the best out of 5 results, but I was wondering if this was correct or simply a waste of time. Thanks, Louis 2008/4/16, Thomas Ludwig <[EMAIL PROTECTED]>: > > > > Thank you very much Thomas. So in fact the walk-forward measures the > > data-mining bias in some way? I will read what you say I should > > read, and I will look at chapter 20 in Howard's book... > > > Yes, it's explained there in detail. It's great that Amibroker now > automates this process (that wasn't the case when Howard's book was > published). > > > > > > But still, so far I get the impression that if I backtested let's say > > cross (ma,ma...) for 2000 to 2008 and I take this best result and it > > is 100% CAR, then eben if OOS is 50% CAR I guess there can still be > > place for data-ming bias (or curve-fitting) because the optimization > > was done with the best result. > > > If your IS period is 2000-2008 I'm afraid that there is no time left for > the OOS period ;-) Both periods must not overlap! In the book (and you > can configure Amibroker accordingly) the IS period is chosen to be 2 > years and OOS is one year. If you start the process in 2000, the first > IS period would be 2000-2001 and the first OOS period would be 2002. > The second IS period is 2001-2002 and OOS is 2003 - and so forth. In my > understanding this approach simulates the fact that you normally don't > trade a sytem with the same parameters unchanged for many years (unless > it's a really long-term system). Rather, you would re-optimize the > system every 1-2 years to adjust it to changing market conditions. > That's what the walk-forward test is actually doing. Every new > optimization is compared with the results of an OOS period. If the OOS > results are considerably worse than the IS results this is a strong > hint that the sytem will not work in real trading. > > > > > Anyway, if I understand correctly what you say, the OOS will ALWAYS > > be less than IS because the IS is optimized - that is, it will take > > the best-of-100 (or 200, etc.) result, and compare it with a more > > random result that would occure in real life. Am I wrong? > > > The OOS results are not necessarily worse - but most often they are. > I've analyzed a couple of systems (which I don't trade) that had > beautiful IS equity curves - they all failed the walk-forward test > spectacularly. So it seems they were all overfitted and were not able > to generalize. > > Greetings, > > > Thomas > > > > > Thanks, > > > > Louis > > > > 2008/4/16, Thomas Ludwig <[EMAIL PROTECTED]>: > > > Louis, > > > > > > in the IS period your system is optimized, then the best values > > > from the optimization are used to perform a test over the IS and > > > OOS periods. > > > > > > If the OOS results are worse than the IS results, this means that > > > the system doesn't generalize well enough. BTW: This topic is very > > > well explained in chapter 20 of Howard's book. I also suggest to > > > look at > > > > > > http://www.amibroker.com/kb/2008/02/12/getting-started-with-automat > > >ic-walk-forward-optimization/. > > > > > > I must say, that walk forward testing has completely changed my way > > > of thinking. It's much easier to see now if a trading system is > > > worth a second look. > > > > > > Greetings, > > > > > > Thomas > > > > > > > Hi, > > > > > > > > I've been experimenting with walking-forward, and I have some > > > > questions regarding how it works. > > > > > > > > I ran a complete random optimization or buying/selling using the > > > > variables I set (a MCS in fact), and systematically OOS results > > > > were worst than IS. I don't understand how it works, because > > > > whatever if the sampling is IS or OOS it is always the same > > > > variables that are in place. > > > > > > > > Anyone could explain how this work? > > > > > > > > Thanks, > > > > > > > > Louis > > > > > > ------------------------------------ > > > > > > Please note that this group is for discussion between users only. > > > > > > To get support from AmiBroker please send an e-mail directly to > > > SUPPORT {at} amibroker.com > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > http://www.amibroker.com/devlog/ > > > > > > For other support material please check also: > > > http://www.amibroker.com/support.html > > > Yahoo! Groups Links > > > > ------------------------------------ > > Please note that this group is for discussion between users only. > > To get support from AmiBroker please send an e-mail directly to > SUPPORT {at} amibroker.com > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > http://www.amibroker.com/devlog/ > > For other support material please check also: > http://www.amibroker.com/support.html > Yahoo! Groups Links > > > >
