The problem with that is the following: let's say my signal is a MA crossover, and I optimized each MA. I apply a walk-forward of 3 months, and each time the MA Crossover is different. So, in the end, if the OOS is worse than IS, I don't know much more because each time the walk-forward was acting on different MA Crossover. Do you understand what I mean? And because the walk-forward only shows the BEST result of ONE optimized portfolio result (RAR, RRR, CAR, etc.) I just can't make it better because I can't see what is result number 2, or 3. In that way, optimization seems superior, but maybe I am not using walk-forward correctly. That is my main concern!
Louis 2008/4/17, Louis Préfontaine <[EMAIL PROTECTED]>: > > Hi Thomas, > > Do you use Walk-forward as a random optimizer like Monte Carlo Simulation > or do you use it as a parameter optimization (let's say you want to know the > best numbers for a MA crossover). Or maybe both? > > I ask this because my feeling is that if I use it only as a parameter > optimization, each parameter would be tested only one time in each IS or > OOS, hence this could not be significative. I tried to add a random > simulation 5 times to get the best out of 5 results, but I was wondering if > this was correct or simply a waste of time. > > Thanks, > > Louis > > 2008/4/16, Thomas Ludwig <[EMAIL PROTECTED]>: > > > > > > > Thank you very much Thomas. So in fact the walk-forward measures the > > > data-mining bias in some way? I will read what you say I should > > > read, and I will look at chapter 20 in Howard's book... > > > > > > Yes, it's explained there in detail. It's great that Amibroker now > > automates this process (that wasn't the case when Howard's book was > > published). > > > > > > > > > > But still, so far I get the impression that if I backtested let's say > > > cross (ma,ma...) for 2000 to 2008 and I take this best result and it > > > is 100% CAR, then eben if OOS is 50% CAR I guess there can still be > > > place for data-ming bias (or curve-fitting) because the optimization > > > was done with the best result. > > > > > > If your IS period is 2000-2008 I'm afraid that there is no time left for > > the OOS period ;-) Both periods must not overlap! In the book (and you > > can configure Amibroker accordingly) the IS period is chosen to be 2 > > years and OOS is one year. If you start the process in 2000, the first > > IS period would be 2000-2001 and the first OOS period would be 2002. > > The second IS period is 2001-2002 and OOS is 2003 - and so forth. In my > > understanding this approach simulates the fact that you normally don't > > trade a sytem with the same parameters unchanged for many years (unless > > it's a really long-term system). Rather, you would re-optimize the > > system every 1-2 years to adjust it to changing market conditions. > > That's what the walk-forward test is actually doing. Every new > > optimization is compared with the results of an OOS period. If the OOS > > results are considerably worse than the IS results this is a strong > > hint that the sytem will not work in real trading. > > > > > > > > Anyway, if I understand correctly what you say, the OOS will ALWAYS > > > be less than IS because the IS is optimized - that is, it will take > > > the best-of-100 (or 200, etc.) result, and compare it with a more > > > random result that would occure in real life. Am I wrong? > > > > > > The OOS results are not necessarily worse - but most often they are. > > I've analyzed a couple of systems (which I don't trade) that had > > beautiful IS equity curves - they all failed the walk-forward test > > spectacularly. So it seems they were all overfitted and were not able > > to generalize. > > > > Greetings, > > > > > > Thomas > > > > > > > > Thanks, > > > > > > Louis > > > > > > 2008/4/16, Thomas Ludwig <[EMAIL PROTECTED]>: > > > > Louis, > > > > > > > > in the IS period your system is optimized, then the best values > > > > from the optimization are used to perform a test over the IS and > > > > OOS periods. > > > > > > > > If the OOS results are worse than the IS results, this means that > > > > the system doesn't generalize well enough. BTW: This topic is very > > > > well explained in chapter 20 of Howard's book. I also suggest to > > > > look at > > > > > > > > http://www.amibroker.com/kb/2008/02/12/getting-started-with-automat > > > >ic-walk-forward-optimization/. > > > > > > > > I must say, that walk forward testing has completely changed my way > > > > of thinking. It's much easier to see now if a trading system is > > > > worth a second look. > > > > > > > > Greetings, > > > > > > > > Thomas > > > > > > > > > Hi, > > > > > > > > > > I've been experimenting with walking-forward, and I have some > > > > > questions regarding how it works. > > > > > > > > > > I ran a complete random optimization or buying/selling using the > > > > > variables I set (a MCS in fact), and systematically OOS results > > > > > were worst than IS. I don't understand how it works, because > > > > > whatever if the sampling is IS or OOS it is always the same > > > > > variables that are in place. > > > > > > > > > > Anyone could explain how this work? > > > > > > > > > > Thanks, > > > > > > > > > > Louis > > > > > > > > ------------------------------------ > > > > > > > > Please note that this group is for discussion between users only. > > > > > > > > To get support from AmiBroker please send an e-mail directly to > > > > SUPPORT {at} amibroker.com > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > > http://www.amibroker.com/devlog/ > > > > > > > > For other support material please check also: > > > > http://www.amibroker.com/support.html > > > > Yahoo! Groups Links > > > > > > > > ------------------------------------ > > > > Please note that this group is for discussion between users only. > > > > To get support from AmiBroker please send an e-mail directly to > > SUPPORT {at} amibroker.com > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > http://www.amibroker.com/devlog/ > > > > For other support material please check also: > > http://www.amibroker.com/support.html > > Yahoo! Groups Links > > > > > > > > >
