Hi Thomas, I understand what you mean and I agree with you on what you say. My concern is more about what to do when each IS and the following OOS uses some parameters and then each new IS-OOS uses another parameter. Moreover, all those parameters are (as far as I know) the best-of-many and they are not resampled, so I see a potential problem and I have trouble identifying where my system fails because if I use a walk-forward of 2 weeks each 2 weeks has new parameters...
Do you understand what I mean? Sorry if I am confusing... English is not my first language. Louis 2008/4/18, Thomas Ludwig <[EMAIL PROTECTED]>: > > > The problem with that is the following: let's say my signal is a MA > > crossover, and I optimized each MA. I apply a walk-forward of 3 > > months, and each time the MA Crossover is different. So, in the end, > > if the OOS is worse than IS, I don't know much more because each time > > the walk-forward was acting on different MA Crossover. > > > I disagree. Look at it this way: You perform an optimization ovet your > IS period, and then you usually select the best parameter combination > based on the metric you chose (like K-ratio). The basic *assumtion* is > that this "best" parameter combination gives you a trading system that > can generalize - i.e., it will be able to be profitable even if market > conditions change somewhat. But you can't be sure before you apply this > optimized system to NEW data different from the IS data - that's the > OOS period. That's exactly what walk-forward is doing. It just adds one > more assumption: That you normally wouldn't stick to the same > parameters over several years. That's why it walks forward though the > time comparing the optimized result of the respective IS period with > the NEW data of the subsequent OOS period. Now, if you compare your IS > results with the OOS results and you find the latter ones considerably > worse that shows that your system is not robust. Think about the 3D > optimization graph available in Amibroker: Such a poor system would > probably produce a 3D surface plot with big changes (spikes, drops) > instead of stable plateaus - i.e. it is *very* dependent on the > parameters you chose. A small change in the parameters for the tested > period would lead to poor results as the system has only "learned" the > market structure in that period and will fail if that changes over > time. > > > > Do you > > understand what I mean? And because the walk-forward only shows the > > BEST result of ONE optimized portfolio result (RAR, RRR, CAR, etc.) I > > just can't make it better because I can't see what is result number > > 2, or 3. In that way, optimization seems superior, but maybe I am > > not using walk-forward correctly. That is my main concern! > > > > Louis > > > > 2008/4/17, Louis Préfontaine <[EMAIL PROTECTED]>: > > > Hi Thomas, > > > > > > Do you use Walk-forward as a random optimizer like Monte Carlo > > > Simulation or do you use it as a parameter optimization (let's say > > > you want to know the best numbers for a MA crossover). Or maybe > > > both? > > > > > > I ask this because my feeling is that if I use it only as a > > > parameter optimization, each parameter would be tested only one > > > time in each IS or OOS, hence this could not be significative. I > > > tried to add a random simulation 5 times to get the best out of 5 > > > results, but I was wondering if this was correct or simply a waste > > > of time. > > > > > > Thanks, > > > > > > Louis > > > > > > 2008/4/16, Thomas Ludwig <[EMAIL PROTECTED]>: > > > > > Thank you very much Thomas. So in fact the walk-forward > > > > > measures the data-mining bias in some way? I will read what > > > > > you say I should read, and I will look at chapter 20 in > > > > > Howard's book... > > > > > > > > Yes, it's explained there in detail. It's great that Amibroker > > > > now automates this process (that wasn't the case when Howard's > > > > book was published). > > > > > > > > > But still, so far I get the impression that if I backtested > > > > > let's say cross (ma,ma...) for 2000 to 2008 and I take this > > > > > best result and it is 100% CAR, then eben if OOS is 50% CAR I > > > > > guess there can still be place for data-ming bias (or > > > > > curve-fitting) because the optimization was done with the best > > > > > result. > > > > > > > > If your IS period is 2000-2008 I'm afraid that there is no time > > > > left for the OOS period ;-) Both periods must not overlap! In > > > > the book (and you can configure Amibroker accordingly) the IS > > > > period is chosen to be 2 years and OOS is one year. If you start > > > > the process in 2000, the first IS period would be 2000-2001 and > > > > the first OOS period would be 2002. The second IS period is > > > > 2001-2002 and OOS is 2003 - and so forth. In my understanding > > > > this approach simulates the fact that you normally don't trade a > > > > sytem with the same parameters unchanged for many years (unless > > > > it's a really long-term system). Rather, you would re-optimize > > > > the system every 1-2 years to adjust it to changing market > > > > conditions. That's what the walk-forward test is actually doing. > > > > Every new optimization is compared with the results of an OOS > > > > period. If the OOS results are considerably worse than the IS > > > > results this is a strong hint that the sytem will not work in > > > > real trading. > > > > > > > > > Anyway, if I understand correctly what you say, the OOS will > > > > > ALWAYS be less than IS because the IS is optimized - that is, > > > > > it will take the best-of-100 (or 200, etc.) result, and compare > > > > > it with a more random result that would occure in real life. > > > > > Am I wrong? > > > > > > > > The OOS results are not necessarily worse - but most often they > > > > are. I've analyzed a couple of systems (which I don't trade) that > > > > had beautiful IS equity curves - they all failed the walk-forward > > > > test spectacularly. So it seems they were all overfitted and were > > > > not able to generalize. > > > > > > > > Greetings, > > > > > > > > > > > > Thomas > > > > > > > > > Thanks, > > > > > > > > > > Louis > > > > > > > > > > 2008/4/16, Thomas Ludwig <[EMAIL PROTECTED]>: > > > > > > Louis, > > > > > > > > > > > > in the IS period your system is optimized, then the best > > > > > > values from the optimization are used to perform a test over > > > > > > the IS and OOS periods. > > > > > > > > > > > > If the OOS results are worse than the IS results, this means > > > > > > that the system doesn't generalize well enough. BTW: This > > > > > > topic is very well explained in chapter 20 of Howard's book. > > > > > > I also suggest to look at > > > > > > > > > > > > http://www.amibroker.com/kb/2008/02/12/getting-started-with-a > > > > > > >utomat ic-walk-forward-optimization/. > > > > > > > > > > > > > I must say, that walk forward testing has completely changed > > > > > > my way of thinking. It's much easier to see now if a trading > > > > > > system is worth a second look. > > > > > > > > > > > > Greetings, > > > > > > > > > > > > Thomas > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I've been experimenting with walking-forward, and I have > > > > > > > some questions regarding how it works. > > > > > > > > > > > > > > I ran a complete random optimization or buying/selling > > > > > > > using the variables I set (a MCS in fact), and > > > > > > > systematically OOS results were worst than IS. I don't > > > > > > > understand how it works, because whatever if the sampling > > > > > > > is IS or OOS it is always the same variables that are in > > > > > > > place. > > > > > > > > > > > > > > Anyone could explain how this work? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Louis > > > > > > > > > > > > ------------------------------------ > > > > > > > > > > > > Please note that this group is for discussion between users > > > > > > only. > > > > > > > > > > > > To get support from AmiBroker please send an e-mail directly > > > > > > to SUPPORT {at} amibroker.com > > > > > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check > > > > > > DEVLOG: http://www.amibroker.com/devlog/ > > > > > > > > > > > > For other support material please check also: > > > > > > http://www.amibroker.com/support.html > > > > > > Yahoo! Groups Links > > > > > > > > ------------------------------------ > > > > > > > > Please note that this group is for discussion between users only. > > > > > > > > To get support from AmiBroker please send an e-mail directly to > > > > SUPPORT {at} amibroker.com > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > > http://www.amibroker.com/devlog/ > > > > > > > > For other support material please check also: > > > > http://www.amibroker.com/support.html > > > > Yahoo! Groups Links > > > > ------------------------------------ > > Please note that this group is for discussion between users only. > > To get support from AmiBroker please send an e-mail directly to > SUPPORT {at} amibroker.com > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > http://www.amibroker.com/devlog/ > > For other support material please check also: > http://www.amibroker.com/support.html > Yahoo! Groups Links > > > >
